Description

You will validate data pipelines and ensure data integrity across Azure and Databricks environments.

Responsibilities

• Develop and implement test strategies, test cases, and automation scripts for Azure and Databricks data pipelines.
• Perform data validation, reconciliation, and comparative analysis between source and target systems.
• Validate ETL/ELT pipelines built using ADF and Databricks.
• Collaborate with Data Engineers and Product Owners to verify transformation logic against Source-to-Target Mapping (STM).
• Identify data anomalies, document defects, and work with the engineering team to resolve them.
• Integrate data testing into DevOps workflows to support CI/CD pipelines.

Required Skills

• 5+ years of experience in data QA and validation within cloud-based data platforms.
• Proficiency in SQL and Python scripting.
• Strong knowledge of Azure Data Factory and Databricks.
• Hands-on experience with data profiling, reconciliation, and schema validation.
• Understanding of SCD Type 2 and data transformation logic.
• Experience with Azure DevOps or GitHub Actions for CI/CD integration.
• Experience working with large datasets, performance testing, and data lineage tools.
• Ability to monitor and validate data quality across Delta tables and Data Warehouses.

Key Skills

Education

Any Graduate