Description

You will build and maintain data ingestion pipelines and analytics infrastructure on Databricks and Azure. You own data quality, governance, and cost optimization for high-volume batch and event-driven workloads.

This role is hybrid.

Responsibilities

• Build and enhance ingestion pipelines for large batch and event-driven data from third-party vendors, CAPI integrations, and rewards systems.
• Implement data validation, idempotency, replay strategies, and deduplication to ensure quality and reliability.
• Monitor pipeline performance, troubleshoot Spark failures, and optimize for shuffle, skew, and partitioning efficiency.
• Apply governance standards including Unity Catalog, lineage, and PII access controls across all datasets.
• Design pipelines with cost awareness, balancing compute/storage usage against SLA and data quality requirements.

Required Skills

• Strong coding skills in PySpark and SQL with hands-on experience in Databricks notebooks and jobs.
• Deep understanding of Spark fundamentals, including partitioning, skew/shuffle optimization, and log analysis.
• Experience with Snowflake for data modeling and analytics workloads.
• Proficiency in Azure Data Factory (ADF) for orchestration and Azure-native integrations.
• Knowledge of data engineering reliability patterns: validation, idempotency, replay/backfills, and auditability.
• Familiarity with data governance concepts, specifically Unity Catalog, lineage, and access control patterns.
• 5+ years of experience in data engineering.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Delta Live Tables (DLT) and event-driven/streaming ingestion.
• Familiarity with identity resolution concepts or CAPI integrations.

Key Skills

Education

Bachelor's degree