Description

Key Responsibilities:

• Support ETL/ELT pipelines using Azure Data Factory and Azure Databricks.

• Optimize data ingestion frameworks and Data Lake processing pipelines.

• Implement data quality checks, data reconciliation, and validation frameworks.

• Monitor and troubleshoot data pipeline failures, performance bottlenecks, and production issues.

• Perform end-to-end ETL testing across source and target systems.

• Execute source-to-target validation, reconciliation, schema validation, and data integrity checks.

• Create and execute SQL-based validation scripts for large datasets.

• Validate transformation logic, business rules, and incremental data loads.

• Identify, track, and support closure of ETL and data quality defects.

• Support SIT, UAT, Regression, and Production validation activities.

• Automate regression and reconciliation activities to reduce manual effort.

• Integrate automated validation scripts into CI/CD pipelines.

• Drive continuous improvement through automation and AI-assisted testing practices.

Required Technical Skills:

• Azure Databricks

• Azure Data Factory (ADF)

• Python

• PySpark

• SQL / Azure SQL

• ETL Testing

• Bitbucket / Git

Key Skills

Education

Bachelor's degree