Description

You will engineer data processing pipelines and transformations using Azure Data Factory and PySpark with Databricks.

This role is on-site.

Responsibilities

• Create data flows and complex transformations using Azure Data Factory (ADF) and PySpark.
• Provision and configure Databricks clusters for batch and continuous streaming data processing.
• Ingest data from disparate sources into Azure Data Lake Gen2 via Azure Data Factory V2.
• Build reusable pipelines to extract, transform, and load data into Azure SQL DB and SQL Data Warehouse.

Required Skills

• 5+ years of relevant experience in data engineering.
• Proficiency in Azure Data Factory (ADF).
• Strong experience with PySpark and Databricks.
• Ability to develop streaming data pipelines using Apache Spark with Python.
• Proficiency in SQL.
• Experience handling both batch and continuous streaming data processing.

Preferred Skills

• Experience with Azure SQL Data Warehouse.

Key Skills

Education

Any Graduate