Description

You will lead the design and development of large-scale batch and streaming data pipelines on Azure Cloud.

This role is hybrid.

Responsibilities

• Design and build real-time and batch data pipelines using Azure Databricks and Structured Streaming.
• Write efficient ETL scripts and automate workflows using Python.
• Integrate data across diverse sources including DB2 and MongoDB.
• Tune Spark jobs to optimize performance and reduce compute costs.
• Implement CI/CD for Databricks pipelines and notebooks using GitHub and Azure DevOps.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in Python and Apache Spark.
• Hands-on experience with Databricks Notebooks and Structured Streaming.
• Experience with Kafka for real-time data integration.
• Proficiency with relational databases (DB2) and NoSQL systems (MongoDB).
• Working knowledge of GitHub and Azure DevOps for CI/CD implementation.
• Ability to translate data requirements into production-ready pipelines.

Key Skills

Education

Any Gradute