Description

Build and optimize scalable data pipelines on Azure Databricks, focusing on Delta Lake, Unity Catalog, and Lakehouse Architecture.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Databricks, PySpark, and SQL to transform large datasets.
• Integrate data from Azure Blob Storage, ADLS, and relational/non-relational sources.
• Implement data governance, security, and access controls using Unity Catalog to ensure compliance.
• Optimize Databricks workloads for cost efficiency and performance, monitoring pipelines for reliability.
• Deploy Databricks jobs and configurations using Databricks Asset Bundles and GitLab version control.

Required Skills

• 5+ years of experience in Azure Cloud Services and data engineering.
• Strong expertise in Databricks: Delta Lake, Unity Catalog, Lakehouse Architecture, Delta Live Pipelines, and Databricks Runtime.
• Proficiency in Spark and PySpark for big data processing.
• Experience with relational databases and SQL.
• Familiarity with Databricks Asset Bundles and GitLab for deployment and version control.
• Ability to collaborate with cross-functional teams to deliver tailored data solutions for BI tools.

Preferred Skills

• Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification.
• Knowledge of streaming frameworks like Spark Streaming and experience with real-time data solutions.

Key Skills

Education

Any Graduate