Description

Design and implement end-to-end Azure Databricks platform architectures from the ground up.

Responsibilities

• Design and build ingestion, data processing, logging, monitoring, and data sharing frameworks.
• Develop data models across different layers within Databricks.
• Orchestrate data pipelines in Azure and Databricks using metadata-driven frameworks.
• Define strategies for configuration management and CI/CD deployment.
• Drive technical discussions with clients and partners to provide architectural thought leadership.

Required Skills

• 10+ years of experience implementing large-scale data and analytics platforms.
• 5+ years of experience architecting and building data platforms on Azure.
• 5+ years of experience building Databricks platforms from scratch.
• Expertise in Databricks engineering services including Unity Catalog, Spark Jobs, Delta Live Tables (DLT), DLT Meta, Databricks Workflow, and Auto Loader.
• Hands-on proficiency with Databricks SQL and PySpark.
• Strong understanding of data modeling, data mappings, and architectural best practices.
• Experience deploying applications via CI/CD pipelines.
• Proficiency in Agile methodologies and Scrum.

Preferred Skills

• Conduct Proof of Concepts (POCs) and write technical user stories within Agile/Scrum environments.
• Any Graduate degree.

Key Skills

Education

Any Graduate