Description

You will design and implement test automation solutions for LLM and RAG systems.

This role is remote.

Responsibilities

• Design and implement test automation frameworks for LLM and RAG applications.
• Test for model bias, drift, and fairness using precision, recall, and F1 metrics.
• Develop and maintain CI/CD pipelines using Azure DevOps.
• Implement AI/ML testing practices, including test generation and anomaly detection.
• Perform API and database testing alongside distributed data validation.

Required Skills

• 5+ years of experience as an SDET.
• Proficiency in Python and Pytest for framework development.
• Hands-on experience with Azure DevOps, including YAML templates and agent pools.
• Direct experience with RAGAS and LLMs for evaluation and testing.
• Familiarity with Azure Cloud services and Databricks.
• Experience with API and database testing.
• Knowledge of performance testing tools like JMeter or LoadRunner.
• Understanding of Agile/Scrum methodology and JIRA.

Key Skills

Education

Any Graduate