Description

You will own the quality assurance for data pipelines and services within the Azure/Databricks ecosystem.

Responsibilities

• Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks & Azure.
• Develop automated tests for Spark transformations (PySpark, Spark SQL) and Delta Lake features.
• Validate data flows across Azure Data Factory, Synapse, ADLS, and Azure SQL, verifying IAM/ACLs.
• Integrate data validation and smoke tests into CI/CD pipelines using Azure DevOps or GitHub Actions.
• Instrument tests with logging and metrics, defining SLIs/SLOs for data quality and pipeline reliability.

Required Skills

• 7+ years in QA/Test Engineering, including 2+ years in data platform or Spark environments.
• Hands-on experience with Databricks: Notebooks, Jobs, Delta Lake, and cluster policies.
• Strong proficiency in Python for test utilities and solid SQL for data validation.
• Experience testing ETL/ELT patterns, idempotency, and data partitioning.
• Familiarity with Azure Data Factory, Synapse Pipelines, ADLS Gen2, and Key Vault.
• Proven experience with CI/CD for data using Azure DevOps or GitHub Actions.
• Knowledge of schema validation and API testing (REST/Databricks Jobs API).
• Understanding of security concepts like RBAC/ABAAC and managed identities.

Key Skills

Education

Any Graduate