Description

You will design and implement end-to-end data engineering solutions using Databricks and the Azure ecosystem.

Responsibilities

• Execute native Spark migrations to Databricks and implement Medallion architecture.
• Build orchestration layers using Databricks and Azure Data Factory, including control flow logic and complex workflows.
• Develop CI/CD pipelines for Databricks deployments across Dev, QA, and Prod environments using Azure DevOps and GitHub.
• Optimize Databricks performance and cost through Auto Compaction, Z-Ordering, and vacuuming.
• Process near real-time data using Auto Loader and Delta Live Tables (DLT) pipelines.

Required Skills

• 5+ years of experience in data engineering roles.
• Expertise in PySpark, Apache Spark, and Spark SQL.
• Proficiency in Python libraries including Pandas, NumPy, and pymysql.
• Hands-on experience with Azure Data Factory (ADF), Azure Synapse Analytics, and Azure Storage.
• Experience building data governance solutions using Unity Catalog or Azure Purview.
• Strong knowledge of Azure DevOps for CI/CD pipeline design.
• Ability to implement security layers in Delta Lake and manage secrets via Azure Key Vault.
• Experience extracting data via APIs and through SQL stored procedures using Python.
• Competency in writing dynamic SQL and complex stored procedures.

Preferred Skills

• Experience with Azure Functions, Logic Apps, and SQL Pools.

Key Skills

Education

Any Graduate