Description

You will design and own real-time and batch data pipelines in Azure Databricks, leveraging modern orchestration and CI/CD tools.

Responsibilities

• Build and maintain scalable data pipelines using Python, Spark, and Databricks on Azure.
• Implement automated data quality frameworks and validation to ensure integrity and reliability.
• Drive engineering excellence by enforcing code review standards, unit testing, and integration testing.
• Design analytics-ready datasets and observability systems to monitor data drift and system health.
• Partner with Data Scientists and ML Engineers to define use cases and deliver robust data products.

Required Skills

• 6+ years of experience building data pipelines, distributed systems, or large-scale data platforms on major clouds (Azure preferred).
• 4+ years of hands-on experience using Python for maintainable, testable data engineering.
• Strong software engineering fundamentals, including design patterns, testing strategies, and system design.
• Proficiency with workflow orchestration tools (Airflow) and streaming technologies (Kafka).
• Experience with CI/CD practices and building cloud-native data architectures.

Preferred Skills

• Experience with Delta Lake, Unity Catalog, Docker, Kubernetes, and Terraform.
• Familiarity with Snowflake, Cassandra, or ML infrastructure.

Key Skills

Education

Any Graduate