Description

You will design, build, and maintain scalable data pipelines using Azure Databricks, Python, and Azure Data Factory.

This role is on-site.

Responsibilities

• Design and orchestrate ETL/ELT pipelines using Azure Data Factory and Azure Databricks to ingest data from on-premises and cloud sources.
• Perform complex data transformations, cleaning, and validation using Python (PySpark) and SQL within Databricks notebooks.
• Manage data in Azure Data Lake Storage (ADLS Gen2), implementing Delta Lake best practices for ACID transactions and time travel.
• Optimize pipeline performance, addressing latency, throughput, and cost for large-scale datasets.
• Implement automated testing, data quality checks, and monitoring to ensure data integrity.

Required Skills

• 5+ years of proven experience in Data Engineering.
• Advanced proficiency in Python coding and scripting.
• Strong hands-on experience with Azure Databricks, including cluster management, job scheduling, and performance tuning.
• Deep understanding of Apache Spark architecture and PySpark.
• Extensive experience creating pipelines, linked services, and datasets in Azure Data Factory.
• Proficiency with Azure Data Lake Storage (ADLS Gen2) and Blob Storage.
• Strong Cassandra and PostgreSQL skills for querying and analyzing data.
• Ability to troubleshoot complex data issues and optimize slow-running queries or jobs.

Preferred Skills

• Experience with Unity Catalog for data governance.
• Understanding of Event Hubs or Kafka for real-time data streaming.

Key Skills

Education

Any Graduate