Description

Design and build production-grade data pipelines spanning source ingestion, transformation, and delivery within the squad's domain. Own pipeline reliability, monitoring, and incident response for assigned data products.

This role is on-site.

Responsibilities

• Implement data ingestion patterns for batch, API, and event-driven streaming (Kafka, Azure Event Hub) using medallion architecture.
• Execute field-level transformations, business rule logic, and data quality checks in collaboration with System Analysts.
• Support migration and deprecation of legacy platforms (SCOpsBI, SAP) following extract, validate, rebuild, cutover, and decommission patterns.
• Maintain comprehensive pipeline documentation including data lineage, transformation logic, SLA definitions, and dependency maps.
• Mentor junior engineers and contribute to CI/CD setup, testing frameworks, and incident runbooks.

Required Skills

• 7-10+ years of data engineering experience delivering production pipelines in large enterprise environments.
• Expert proficiency in PySpark or Spark SQL for large-scale data transformation on distributed compute.
• Hands-on experience with Databricks, including Delta Lake, Unity Catalog, and workflow orchestration.
• Experience with Microsoft Fabric or Azure Synapse Analytics; familiarity with Fabric IQ and OneLake.
• Proficiency with orchestration tools such as Azure Data Factory, Databricks Workflows, or Apache Airflow.
• Solid SQL skills for data validation, transformation logic, and ad-hoc source system analysis.
• Experience building ingestion pipelines from enterprise operational systems (ERP, WMS, TMS).
• Strong understanding of data quality frameworks, anomaly alerting, and SLA-compliant health monitoring.
• Experience with DevOps practices for data pipelines, including Git, CI/CD, and automated testing.

Preferred Skills

• Experience with real-time streaming (Kafka, Azure Event Hub, Delta Live Tables, Spark Structured Streaming).
• Familiarity with Oracle, ERP ingestion patterns, or large-scale ERP migration programs.

Key Skills

Education

Bachelor's degree