Description

You will own quality assurance for data ingestion and ETL pipelines, building reusable test frameworks for unit, integration, and end-to-end validation.

Responsibilities

• Design and implement reusable test frameworks for unit, integration, and end-to-end testing of data ingestion and ETL pipelines.
• Write and maintain unit tests for transformation logic, connector behavior, and schema validation using Python.
• Develop integration tests to validate end-to-end data flow from source ingestion through raw, curated, and consumption layers.
• Build automated test suites that execute within CI/CD pipelines, gating production deployments based on results.
• Validate data quality checks, ensuring DQ rules for completeness, schema conformance, and freshness correctly trigger alerts.

Required Skills

• 8-10 years of experience in QA/test engineering with a focus on data pipelines, ETL processes, or data platforms.
• Strong proficiency in Python for test automation and data validation scripting.
• Hands-on experience testing data pipelines on AWS services including Glue, Spark, S3, Athena, and Redshift.
• Experience with data testing frameworks such as Great Expectations, dbt tests, pytest, or custom frameworks.
• Ability to write SQL queries for data validation and reconciliation.
• Experience integrating automated tests into CI/CD pipelines using GitHub Actions, CodePipeline, or Jenkins.
• Understanding of data quality dimensions: completeness, accuracy, consistency, freshness, and schema conformance.

Preferred Skills

• Experience with performance or load testing for data pipelines.
• Familiarity with contract testing for APIs and schema registries.
• Knowledge of data observability tools like Monte Carlo or Datafold.

Key Skills

Education

Bachelor's degree