Description

You will build and maintain automated data quality checks for large-scale data pipelines.

Responsibilities

• Design, develop, and maintain automated data quality checks for complex data pipelines handling petabyte scale datasets.
• Implement scalable data validation frameworks using PySpark, PyTest, Python, SQL, and Hive, ensuring comprehensive test coverage.
• Collaborate with Data Engineers and DevOps teams to integrate automated tests into CI CD workflows.
• Analyze test results, identify data anomalies, and provide actionable insights to resolve data quality issues.
• Develop monitoring and alerting solutions for data quality in production environments.

Required Skills

• 5+ years of professional experience in test engineering or data engineering.
• Proficiency in PySpark and Python for data validation.
• Strong command of SQL for data querying and validation.
• Hands-on experience with Data Quality principles and ETL processes.
• Familiarity with Big Data technologies.
• Experience in Test Engineering and Automation practices.
• Working knowledge of DevOps principles.

Preferred Skills

• Experience with data quality frameworks like Deequ and Great Expectations.
• Familiarity with workflow orchestration tools such as Airflow.

Key Skills

Education

Any Graduate