Description

You ensure data quality and validate pipelines across Azure-based cloud platforms.

Responsibilities

• Validate datasets, transformations, and business rules for completeness, accuracy, and schema integrity.
• Test ETL/ELT workflows in Databricks, ADF, and dbt, including SQL logic and Python/PySpark transformations.
• Develop automated test scripts and reusable validation frameworks integrated into CI/CD workflows.
• Investigate data anomalies and partner with Data Engineering to resolve root causes.
• Prepare test plans, case summaries, and defect reports while collaborating with Product and BI teams.

Required Skills

• 3+ years of experience in data analysis, testing, or quality assurance.
• Strong SQL skills for profiling, querying, and validating large datasets.
• Proficiency in Python, with experience in PySpark for data transformations.
• Hands-on experience testing ETL/ELT pipelines using Databricks, Azure Data Factory (ADF), and dbt.
• Understanding of data engineering concepts including schemas, partitions, metadata, and storage formats like Parquet and Delta.
• Familiarity with Git version control and CI/CD-driven testing workflows.
• Ability to translate business rules into measurable data validation steps.

Preferred Skills

• Experience with data observability tools such as Great Expectations or Soda.
• Familiarity with dashboard testing in Power BI or Tableau.

Key Skills

Education

Bachelor's degree