Description

Build and optimize scalable data pipelines on Azure Databricks using Delta Lake, Unity Catalog, and Lakehouse Architecture.

You will own the end-to-end development of ETL/ELT pipelines, ensuring data governance, security, and performance optimization across large-scale datasets.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from Azure Blob Storage, ADLS, and relational/non-relational sources.
• Implement data governance and access controls using Unity Catalog for compliance.
• Optimize Databricks workloads for cost efficiency and reliability.
• Deploy jobs and configurations using Databricks Asset Bundles and GitLab.

Required Skills

• 5+ years of experience in Azure Cloud Services and data engineering.
• Strong expertise in Databricks: Delta Lake, Unity Catalog, Lakehouse Architecture, Delta Live Pipelines, and Databricks Runtime.
• Proficiency in Spark and PySpark for big data processing.
• Experience with relational databases and SQL.
• Familiarity with Databricks Asset Bundles and GitLab for deployment and version control.
• Ability to collaborate with cross-functional teams to deliver tailored data solutions for BI tools.

Preferred Skills

• Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification.
• Knowledge of streaming frameworks like Spark Streaming and experience with real-time data solutions.

Key Skills

Education

Any Graduate