Description

Education Start date (dd-mmm-yy) Duration of assignment (in Months)
6+ years of experience in ETL/Data Testing.
3+ years of experience testing Databricks or Spark-based platforms.
Hands-on experience in AI/ML Model Testing or MLOps validation.
Strong coding experience in Python and SQL.

Must Have Technical/Functional Skills

ETL & Data Testing
· Validate end-to-end ETL/ELT pipelines across enterprise data platforms.
· Perform Source-to-Target validation and reconciliation testing.
· Validate data transformations, aggregations, business rules, and lineage.
· Execute data quality checks on large-scale structured and semi-structured datasets.
· Perform schema, metadata, and Delta Lake validations.
· Write complex SQL queries for backend validation and data integrity testing.
· Validate CDC (Change Data Capture) and incremental data loading processes.
· Test batch and streaming data pipelines.

Databricks Testing
· Validate Databricks notebooks, Workflows, Jobs, Delta Live Tables, and Unity Catalog implementations.
· Test Bronze, Silver, and Gold layer data transformations.
· Verify workflow orchestration and dependency handling across Databricks jobs.
· Perform performance validation of Spark and PySpark transformations.
· Validate data quality frameworks implemented in Databricks.

AI/ML Testing
· Validate AI/ML model integrations within enterprise applications.
· Verify model scoring results and prediction outputs.
· Perform data drift, schema drift, and feature validation testing.
· Validate feature engineering pipelines and training datasets.
· Execute model input/output verification against expected business outcomes.
· Validate explainability, traceability, and auditability requirements.
· Test model deployment workflows integrated through MLflow, Databricks ML, Azure ML, or similar platforms.
· Validate GenAI/RAG workflows, prompt-response accuracy, and grounding mechanisms where applicable.

Integration Testing
· Perform end-to-end integration testing across:
o Databricks
o Data warehouses
o APIs
o AI/ML services
o Cloud platforms
o Downstream reporting systems
· Validate data flow between source systems, data lakes, ML models, and consuming applications.
· Perform contract testing and API validation.

Test Automation
· Design and develop automation frameworks for:
o ETL testing
o API testing
o Data validation
o ML model validation
· Build reusable test utilities using Python.
· Integrate automated tests into CI/CD pipelines.
· Develop automated quality gates for deployment validation.
· Generate automated execution reports and test metrics

Key Skills

Education

Bachelor's degree