Description

Design, build, and maintain scalable data pipelines and platforms using Databricks, Apache Spark, and Delta Lake on Azure.

This role is hybrid.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Databricks, Apache Spark, and PySpark.
• Build and manage Delta Lake tables implementing Medallion Architecture (Bronze, Silver, Gold layers).
• Ingest data from structured and unstructured sources, including databases, APIs, and cloud storage.
• Implement data governance, security, and lineage using Unity Catalog.
• Optimize Spark workloads, clusters, and SQL queries for performance and cost efficiency.

Required Skills

• Strong hands-on experience with Databricks and Apache Spark.
• Advanced SQL and data transformation skills.
• Experience with Delta Lake and Lakehouse architecture.
• Experience designing and implementing ETL/ELT pipelines.
• Strong understanding of data warehousing, dimensional modeling, and data lake concepts.
• Experience working with Microsoft Azure and ADLS.
• Familiarity with Git and CI/CD practices.

Preferred Skills

• Experience with Databricks Auto Loader, Structured Streaming, or Delta Live Tables.
• Knowledge of orchestration tools such as Azure Data Factory or Airflow.
• Experience with Terraform or other Infrastructure-as-Code technologies.

Key Skills

Education

Any Graduate