Description

You will design, build, and support end-to-end data pipelines including ingestion, transformation, validation, and publishing.

This role is on-site.

Responsibilities

• Develop and optimize SQL and PySpark transformations in Databricks for large datasets.
• Build production-grade Python components with modular code, logging, error handling, and testing.
• Create and maintain Azure Data Factory pipelines with triggers, parameterization, and failure handling.
• Provision and manage Azure resources using Pulumi Infrastructure as Code.
• Participate in code reviews, documentation, and operational triage.

Required Skills

• 5+ years of experience in ETL/ELT pipeline development.
• Advanced SQL skills including CTEs, views, joins, and performance tuning.
• Production-quality Python development with CI/CD, unit/integration testing, and code coverage.
• PySpark expertise in distributed transformations, performance optimization, and debugging.
• Azure Data Factory experience with Databricks/ADLS integration.
• Databricks proficiency with Delta Lake patterns and Unity Catalog.
• Azure fundamentals including ADLS Gen2, Azure SQL, and Resource Groups.
• Hands-on experience with Pulumi for infrastructure provisioning.

Preferred Skills

• TypeScript for Pulumi pipelines or Java/.NET for existing service integration.

Key Skills

Education

Any Gradute