Description

Design, implement, and optimize large-scale data pipelines using Databricks on Azure.

This role is hybrid.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Integrate data from Azure Blob Storage, ADLS, and other relational/non-relational systems.
• Optimize Databricks workloads for cost efficiency and performance.
• Monitor and troubleshoot data pipelines to ensure accuracy and reliability.
• Implement data security, access controls, and governance standards using Unity Catalog.

Required Skills

• 10+ years of experience in data engineering.
• Strong expertise in Databricks, including Delta Lake and Lakehouse Architecture.
• Proficiency in Azure Cloud Services.
• Solid understanding of Spark and PySpark for big data processing.
• Experience with relational databases.
• Knowledge of Databricks Asset Bundles and GitLab.
• Ability to prepare data for dashboard and BI Tools.

Key Skills

Education

Any Graduate