Description

You will lead the development and management of ETL workflows and big data processing pipelines within the Azure ecosystem.

Responsibilities

• Design and implement data pipelines using PySpark on Azure Databricks.
• Develop and manage ETL workflows using Azure Data Factory (ADF).
• Optimize Spark jobs for performance and scalability.
• Automate data workflows and implement error handling and monitoring in ADF.
• Debug and troubleshoot PySpark scripts and ADF pipeline failures.

Required Skills

• 6+ years of professional experience.
• Extensive experience with Azure Data Factory (ADF) for ETL development.
• Hands-on expertise with Azure Databricks and PySpark for big data processing.
• Proficiency in Python and PySpark for data transformation.
• Strong expertise in SQL for data extraction and transformations.
• Experience implementing CI/CD pipelines for ADF and Databricks notebooks.
• Knowledge of Azure services including Azure Synapse Analytics, Azure Data Lake, and Azure Blob Storage.
• Understanding of Spark performance tuning and Azure cost optimization.
• Ability to implement data governance, security, and compliance practices.

Key Skills

Education

Any Graduate