Description

This role involves building and validating data pipelines using Python and PySpark within an Azure Cloud environment.

Responsibilities

• Develop, maintain, and execute automated test scripts for data pipelines and engineering workflows using Python and PySpark.
• Validate data transformations, data quality, and pipeline performance on Azure Cloud environments.
• Collaborate with data engineers and developers to understand data workflows and test scenarios.
• Build and optimize test frameworks that integrate with CI/CD pipelines.
• Perform regression, performance, and API testing for data applications.

Required Skills

• Strong programming skills in Python and PySpark.
• 5+ years of professional experience.
• Experience in automated testing for data pipelines and big data platforms.
• Working knowledge of Azure Cloud services, including Azure Data Factory.
• Familiarity with CI/CD pipelines and test automation frameworks.
• Understanding of data validation, ETL testing, and SQL query optimization.
• Proficiency in SQL.
• Excellent problem-solving and communication skills.

Preferred Skills

• Hands-on experience with Azure Databricks and related Azure data technologies.
• Background in software development and test engineering for data-intensive applications.

Key Skills

Education

Any Graduate