Description

You will design, build, and optimize data pipelines on Microsoft Azure, focusing on Azure Databricks and Delta Lake. You own the end-to-end data engineering lifecycle, from ingestion to transformation and deployment.

This role is on-site.

Responsibilities

• Build and maintain scalable data pipelines using Azure Data Bricks (ADB) and PySpark.
• Implement CI/CD processes in Azure DevOps for stories, bugs, and issue management.
• Transform and cleanse data to ensure accuracy, consistency, and compliance with security standards.
• Troubleshoot data issues, monitor pipeline performance, and optimize for seamless functionality.
• Collaborate with cross-functional teams and data architects to align data models with project objectives.

Required Skills

• 9+ years of experience in data engineering roles.
• Strong proficiency in PySpark for large-scale data processing.
• Hands-on experience with Azure Databricks and Delta Lake architectures.
• Expertise in Azure DevOps, including CI/CD implementation and repository management.
• Experience working in Agile development environments with sprint deliverables.
• Bachelor's degree in Computer Science or a related technical field.

Key Skills

Education

Bachelor's degree