Description

Design and optimize large-scale data pipelines on Azure Databricks to ensure scalability, reliability, and performance.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, Delta Live Pipelines, and PySpark.
• Integrate data from Azure Blob Storage, ADLS, and relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency and performance.
• Implement data security, access controls, and governance standards using Unity Catalog.
• Deploy Databricks jobs and configurations using Databricks Asset Bundles and GitLab.

Required Skills

• 7+ years of experience in data engineering.
• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture).
• Proficiency in SQL and Python.
• Solid understanding of Spark and PySpark for big data processing.
• Experience with Azure Cloud Services and Data Warehouse concepts.
• Knowledge of Databricks Asset Bundles and GitLab for version control and deployment.
• Experience with relational databases.

Preferred Skills

• Familiarity with Databricks Runtimes and advanced configurations.
• Knowledge of streaming frameworks like Spark Streaming.

Key Skills

Education

Any Graduate