Description

You will ensure data integrity and model accuracy through rigorous automated testing and validation within big data environments.

Responsibilities

• Collaborate with Data Scientists and Engineers to develop QA test suites for data models and ETL processes.
• Select and integrate data testing tools and frameworks to meet technical requirements.
• Design and implement automated tests to increase coverage and efficiency across the data pipeline.
• Monitor system performance and advise on necessary infrastructure or cluster adjustments.
• Troubleshoot application defects and manage the full defect life cycle in an Agile environment.

Required Skills

• 5+ years of experience in data testing or quality engineering.
• Advanced Python programming for creating comprehensive test suites.
• Expertise in writing complex SQL queries for data extraction and validation.
• Proficiency in Hive QL and Spark QL.
• Hands-on experience with data testing tools such as DBT, iCEDQ, QuerySurge, Denodo, or Informatica.
• Strong understanding of Big Data concepts and NoSQL databases.
• Experience with Linux or Unix environments.
• Working knowledge of messaging systems like Kafka or RabbitMQ.
• Ability to conduct User Acceptance Testing (UAT) with business users.

Preferred Skills

• Experience with BI tools including PowerBI or Tableau.
• Knowledge of Machine Learning and Statistical Analysis.

Key Skills

Education

Bachelor's degree in Computer Science