Description

You will design, implement, and optimize large-scale data pipelines on Azure, owning the end-to-end ETL/ELT lifecycle.

This role is hybrid.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks and Azure Data Factory.
• Integrate data from multiple sources including Azure Blob Storage, ADLS, and relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency and performance.
• Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
• Implement and manage data security and governance standards using Unity Catalog.

Required Skills

• 12+ years of professional experience in data engineering.
• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture).
• Proficiency in PySpark and Spark for big data processing.
• Solid understanding of SQL and relational databases.
• Expertise in Azure Cloud Services.
• Experience leveraging Databricks Asset Bundles for deployment.
• Knowledge of Git/GitLab for version control.

Preferred Skills

• Experience with non-relational data systems integration.

Key Skills

Education

Any Graduate