Description

You will own the design and execution of automated testing frameworks for ETL/ELT pipelines on Azure Databricks.

This role is remote.

Responsibilities

• Design and develop automated testing frameworks for ETL/ELT pipelines.
• Perform end-to-end validation of data pipelines running on Azure Databricks.
• Execute source-to-target data reconciliation and validation activities.
• Develop SQL and Python-based automation scripts for data quality checks.
• Validate data transformations built using PySpark and Delta Lake.

Required Skills

• 5-7 years of experience in ETL/ELT Testing and Data Validation.
• Strong hands-on experience with Azure Databricks, notebooks, and jobs.
• Expertise in SQL/Spark SQL for data validation and reconciliation.
• Proficiency in Python for test automation and data quality testing.
• Experience testing PySpark and Delta Lake-based data pipelines.
• Strong understanding of source-to-target validation and data migration testing.
• Experience with Agile methodologies and test management practices.

Preferred Skills

• Knowledge of data quality frameworks and validation techniques.

Key Skills

Education

Any Graduate