Description

You will design and implement observability strategies for scalable, real-time data pipelines in a cloud environment. This role requires building robust instrumentation and monitoring systems across Azure infrastructure to ensure data reliability and performance.

This role is on-site.

Responsibilities

• Design and implement observability strategies for data pipelines and real-time streaming systems.
• Build instrumentation and monitoring solutions using Databricks, Spark, Python, and SQL.
• Troubleshoot and maintain streaming systems across Azure cloud infrastructure.
• Drive best practices for data system observability, telemetry, and performance optimization.
• Collaborate on CI/CD pipelines and system reliability engineering.

Required Skills

• 6+ years of hands-on experience in Azure Data Engineering.
• Proficiency in Azure Databricks and Apache Spark.
• Strong command of Python and SQL for data engineering tasks.
• Experience with streaming technologies: Kafka, Event Hubs, and Structured Streaming.
• Solid understanding of distributed data pipelines and cloud systems architecture.
• Background in building highly observable and reliable data systems.

Preferred Skills

• Familiarity with telemetry and metrics platforms such as Grafana, Prometheus, OpenTelemetry, Dynatrace, or Splunk.
• Exposure to DevOps practices and CI/CD pipeline implementation.
• Background in system performance or reliability engineering.

Key Skills

Education

Any Graduate