Description

Design and maintain data pipelines on the Azure cloud platform, focusing on ingestion, transformation, and warehousing.

This role is on-site.

Responsibilities

• Build and optimize ETL/ELT workflows using Azure Data Factory and Databricks.
• Develop PySpark and Spark SQL scripts within Databricks notebooks for data processing.
• Manage data storage and access using Azure Data Lake Storage (ADLS) and Azure SQL.
• Implement CI/CD pipelines and version control using Azure DevOps and Git.
• Troubleshoot pipeline failures, cluster issues, and performance bottlenecks.

Required Skills

• 8+ years of experience in data engineering.
• Hands-on expertise with Azure Data Factory, Databricks, ADLS, and Azure SQL.
• Strong proficiency in PySpark, Spark SQL, and Delta Lake.
• Deep understanding of ETL/ELT concepts, data ingestion patterns, and data warehousing.
• Experience with Azure DevOps, Git, and CI/CD pipeline implementation.
• Familiarity with monitoring tools including Azure Monitor, Log Analytics, and Databricks logs.
• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree