Description

You will design, develop, and maintain data pipelines using Azure Databricks and Azure Data Factory. You ingest and transform data from multiple source systems into the Azure Data Lake and manage Delta Tables to ensure efficient processing. You also implement data governance and access management using Databricks Unity Catalog and perform unit and end-to-end testing for all pipelines.

Responsibilities

• Design and maintain data pipelines using Azure Databricks and Azure Data Factory.
• Ingest and transform data from source systems into the Azure Data Lake.
• Manage Delta Tables within Databricks for reliable data processing.
• Implement data governance and access management using Databricks Unity Catalog.
• Perform unit testing and end-to-end system testing for data pipelines.

Required Skills

• 3+ years of hands-on Databricks experience with certification.
• 3+ years of experience building and supporting pipelines using Azure Data Factory.
• 5+ years of strong SQL development experience.
• Hands-on experience with Databricks Delta Tables and Unity Catalog.
• Understanding of Agile development processes.

Key Skills

Education

Any Graduate