Description

You will ensure data integrity and pipeline reliability across Azure and Databricks environments.

Responsibilities

• Develop and implement test strategies, test cases, and automation scripts to validate Azure and Databricks data pipelines.
• Perform data validation, reconciliation, and comparative analysis between source and target systems, including ETL/ELT pipelines built with ADF.
• Collaborate with Data Engineers and Product Owners to validate Source-to-Target Mapping (STM) and transformation logic.
• Monitor data quality across Delta tables and Data Warehouses, identifying anomalies and driving defect resolution.
• Integrate data testing into CI/CD workflows and support test data management and regression testing.

Required Skills

• 7+ years of experience in data QA and validation within cloud-based platforms.
• Proficiency in SQL and Python scripting.
• Hands-on experience with Azure Data Factory (ADF) and Databricks.
• Strong knowledge of data profiling, reconciliation, and schema validation.
• Understanding of SCD Type 2 and complex data transformation logic.
• Experience with Azure DevOps or GitHub Actions for CI/CD integration.
• Ability to work with large datasets, performance testing, and data lineage tools.
• Experience working with Data Warehouses.

Key Skills

Education

Not specified