Description

You will focus on automation testing for data engineering, ETL, and large-scale data pipelines.

Responsibilities

• Automate testing for ETL processes and data engineering workflows.
• Validate large-scale data pipelines within Azure Databricks or Snowflake environments.
• Perform data validation and complex querying using SQL.
• Verify streaming data pipelines using Azure Event Hub or Kafka.
• Monitor pipeline performance and analyze test results using Azure Monitor and Log Analytics.

Required Skills

• 5+ years of experience in QA automation for ETL or data engineering domains.
• Strong proficiency in Python for test automation using PyTest or unittest.
• Expertise in SQL for data validation and querying.
• Hands-on experience with Azure Data Cloud services including ADF, Synapse, ADLS Gen 2, and Cosmos DB.
• Experience testing data pipelines in Azure Databricks or Snowflake.
• Knowledge of data Lakehouse formats such as Delta table and Parquet.
• Understanding of data modeling concepts including Star Schema, Data Vault, and Medallion Architecture.
• Familiarity with Azure Blob Storage and Azure Data Lake for validation tasks.
• Familiarity with data security and governance practices for Azure Databricks.

Preferred Skills

• Direct experience with Azure Databricks.

Key Skills

Education

Any Graduate