Description

You will lead the development of large-scale batch and streaming data pipelines on Azure Cloud.

Responsibilities

• Design and develop real-time and batch data pipelines using Azure Databricks, Apache Spark, and Structured Streaming.
• Write efficient ETL scripts and automate workflows using Python.
• Integrate data across various sources and destinations including DB2 and MongoDB.
• Tune Spark jobs to optimize performance and compute cost-effectiveness on Azure.
• Implement CI/CD for Databricks pipelines and notebooks using GitHub and Azure DevOps.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in Python and Apache Spark.
• Hands-on experience with Databricks Notebooks and Structured Streaming.
• Experience with Kafka for real-time integration.
• Proficiency with relational databases like DB2 and NoSQL systems like MongoDB.
• Working knowledge of GitHub and Azure DevOps for CI/CD implementation.
• Ability to translate data requirements into production-ready pipelines.
• Any Graduate degree.

Key Skills

Education

Any Gradute