Description

You will build and maintain automated data quality checks for large-scale data pipelines.

This role is on-site.

Responsibilities

• Design, develop, and maintain automated data quality checks for petabyte-scale data pipelines.
• Implement scalable data validation frameworks using PySpark, PyTest, Python, SQL, and Hive.
• Integrate automated tests into CI/CD workflows with Data Engineers and DevOps teams.
• Analyze test results to identify data anomalies and provide actionable insights for resolution.
• Develop monitoring and alerting solutions for data quality in production environments.

Required Skills

• 5+ years of professional experience in test engineering or data engineering.
• Proficiency in PySpark and Python for data validation.
• Strong command of SQL for data querying and validation.
• Hands-on experience with Data Quality principles and ETL processes.
• Familiarity with Big Data technologies.
• Experience in Test Engineering and Automation practices.
• Working knowledge of DevOps principles.

Preferred Skills

• Experience with data quality frameworks like Deequ and Great Expectations.
• Familiarity with workflow orchestration tools such as Airflow.

Key Skills

Education

Any Graduate