Description

You build and maintain data ingestion pipelines and analytics infrastructure using PySpark, SQL, Databricks, and Azure services.

This role is hybrid.

Responsibilities

• Build and enhance ingestion pipelines for large batch and event-driven paths, integrating third-party enrichment vendors and digital platforms.
• Implement data quality, reliability, and operational readiness including validation, idempotency, replay strategies, and deduplication.
• Own monitoring, alerting, and dashboarding; troubleshoot Spark failures, performance issues, and root causes beyond simple reruns.
• Apply governance standards such as Unity Catalog, lineage, and access controls while managing PII and non-PII data separation.
• Optimize pipeline cost and performance through cluster sizing, workload tuning, and efficient compute/storage usage decisions.

Required Skills

• 5+ years of hands-on experience with PySpark and SQL, including medallion patterns and notebook/job management.
• Strong understanding of Spark fundamentals: partitioning, skew/shuffle optimization, and log-based failure diagnosis.
• Experience with Databricks, including performance tuning and Delta Lake concepts.
• Proficiency in Snowflake for data modeling and analytics workloads.
• Hands-on experience with Azure Data Factory (ADF) for orchestration and Azure-native integrations.
• Knowledge of ETL reliability patterns: validation, idempotency, replay/backfills, deduplication, and auditability.
• Familiarity with data governance, lineage, access control patterns, and PII handling.
• Ability to execute independently with an ownership mindset in a fast-moving environment.

Preferred Skills

• Experience with event-driven/streaming ingestion or Delta Live Tables (DLT).
• Familiarity with Conversion API (CAPI) integrations or identity resolution concepts.

Key Skills

Education

Any Gradute