Description

Design and build production-grade data pipelines for ingestion, transformation, and delivery within domain scope. Own pipeline reliability, monitoring, and incident response for assigned data products.

This role is on-site.

Responsibilities

• Implement medallion architecture (bronze, silver, gold) and Fabric IQ standards across all data product builds.
• Build ingestion patterns for batch files, APIs, and event-driven streaming using Kafka and Azure Event Hub.
• Collaborate with System Analysts to execute field-level transformations, business rules, and data quality checks.
• Support migration and deprecation of legacy platforms (SCOpsBI SQL Server, SAP) using extract, validate, rebuild, cutover, and decommission patterns.
• Mentor junior and mid-level engineers while maintaining comprehensive documentation for lineage, logic, and SLAs.

Required Skills

• 7-10 years of data engineering experience delivering production pipelines in enterprise environments.
• Expert proficiency in PySpark or Spark SQL for large-scale data transformation.
• Hands-on experience with Databricks, including Delta Lake, Unity Catalog, and workflow orchestration.
• Experience with Microsoft Fabric or Azure Synapse Analytics; familiarity with Fabric IQ and OneLake preferred.
• Proficiency with orchestration tools such as Azure Data Factory, Databricks Workflows, or Apache Airflow.
• Solid SQL skills for validation, transformation logic, and source system analysis.
• Experience building ingestion pipelines from ERP, WMS, or TMS systems.
• Strong understanding of data quality frameworks, anomaly alerting, and SLA-compliant health monitoring.
• DevOps practices for data pipelines including Git, CI/CD, and automated testing.

Preferred Skills

• Experience with real-time streaming (Kafka, Azure Event Hub, Delta Live Tables, Spark Structured Streaming).
• Familiarity with Oracle, ERP ingestion patterns, or large-scale ERP migration programs.

Key Skills

Education

Bachelor's degree