Description

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using Azure Databricks.

Build ETL/ELT workflows using PySpark, Python, and Spark SQL.

Develop and optimize complex SQL queries and data transformations.

Work with Azure Data Factory (ADF) for orchestration and workflow automation.

Implement data Lakehouse architecture using Delta Lake.

Work with Azure Data Lake Storage Gen2 (ADLS).

Perform data modelling, data integration, and data migration.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Implement data quality checks, monitoring, and error handling.

Collaborate with data architects, analysts, and business stakeholders.

Implement CI/CD pipelines and deployment processes.

Required Technical Skills:

Strong hands-on experience with Azure Databricks.

Excellent programming skills in Python and PySpark.

Strong knowledge of Apache Spark and Spark SQL.

Experience with Azure Data Factory (ADF).

Hands-on experience with Azure Data Lake Storage Gen2.

Strong understanding of Delta Lake and Lakehouse architecture.

Expertise in SQL and data warehousing concepts.

Experience with Azure DevOps and CI/CD.

Knowledge of data pipeline optimization and performance tuning.

Good to Have:

Experience with Unity Catalog and Databricks Workflows.

Knowledge of Databricks Asset Bundles.

Experience with Azure Synapse Analytics.

Knowledge of Terraform and infrastructure automation.

Experience with streaming technologies such as Kafka or Event Hubs.

Understanding of data governance and security

Key Skills

Education

Bachelor's degree