Description

Architect and own automated testing frameworks for data pipelines across Raw, Curated, and Mart layers using PySpark and Databricks.

You will own the end-to-end testing lifecycle from QA through Production.

Responsibilities

• Design and implement data quality validation frameworks to ensure accuracy, completeness, and consistency across transformation layers.
• Create metadata-driven tests integrated with CI/CD pipelines, covering all ingestion and transformation stages.
• Lead QA user stories and acceptance criteria, precisely defining test scenarios for data ingestion and consumption.
• Perform complex data reconciliation testing across 10+ source systems, troubleshooting pipeline failures with Data Engineers.
• Plan and facilitate User Acceptance Testing (UAT) for Tableau on Databricks, driving defect triage and resolution.

Required Skills

• 5+ years of experience in Data Engineering with proven expertise testing and validating data pipelines.
• Proficient in Databricks notebooks, PySpark, Python, and SQL for data quality testing.
• Experience with medallion architecture (Raw, Curated, Mart layers) and data validation frameworks.
• Strong knowledge of CI/CD pipelines, specifically Azure DevOps for automated test execution.
• Familiarity with SAFe Agile framework, including PI planning and sprint ceremonies.
• Expertise in testing AI/BI models, ensuring data quality from feature engineering through model scoring.
• Strong understanding of data governance, including data lineage, audit trails, and compliance testing.

Preferred Skills

• Experience with Azure Purview and Profisee MDM.
• Ability to create advanced data quality KPIs and integrate them into automated dashboards.

Key Skills

Education

Bachelor's degree