Description

Design, develop, and support data pipelines using Azure Databricks and Azure Data Factory. Ingest, transform, and load data from multiple source systems into the Azure Data Lake. Leverage Databricks-managed Delta Tables to ensure scalable, efficient, and reliable data processing. Work with Databricks Unity Catalog to support data governance, access control, and metadata management. Monitor and troubleshoot data pipeline execution, data quality, and performance issues.

Responsibilities

• Design and build data pipelines using Azure Data Factory and Azure Databricks.
• Implement data governance and access control using Databricks Unity Catalog.
• Monitor pipeline execution and resolve data quality and performance issues.
• Conduct unit testing and system testing for data pipeline components.
• Collaborate within Agile development processes to deliver data solutions.

Required Skills

• 3+ years of hands-on experience with Databricks (certification required).
• 3+ years of experience building and supporting data pipelines using Azure Data Factory.
• 5+ years of strong SQL development experience.
• Hands-on experience with unit testing and system testing of data pipelines.
• Proficiency with Databricks-managed Delta Tables.
• Understanding of data governance, metadata management, and access control.
• Strong verbal communication skills.

Preferred Skills

• Experience with Azure Data Lake storage structures.
• Knowledge of Agile development methodologies.

Key Skills

Education

Any Graduate