Description

Design and optimize large-scale data pipelines on Azure Databricks for scalability and performance.

This role is on-site.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, Delta Live Pipelines, and PySpark.
• Integrate data from Azure Blob Storage, ADLS, and relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency and performance.
• Implement data security, access controls, and governance standards using Unity Catalog.
• Deploy Databricks jobs and configurations using Databricks Asset Bundles and GitLab.

Required Skills

• 7+ years of experience in data engineering.
• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture).
• Proficiency in SQL and Python.
• Solid understanding of Spark and PySpark for big data processing.
• Experience with Azure Cloud Services and Data Warehouse concepts.
• Knowledge of Databricks Asset Bundles and GitLab for version control and deployment.
• Experience with relational databases.

Key Skills

Education

Any Graduate