Description
Key Responsibilities
1. AI Quality Engineering & Architecture
- Architect and implement enterprise-wide AI Quality Engineering frameworks and testing strategies.
- Establish testing standards for AI/ML, Generative AI, LLMs and Agentic AI solutions.
- Define quality assurance frameworks covering functional, API, integration, performance, security and AI/ML validation.
- Establish governance for LLM drift detection, prompt quality, agent monitoring, observability, data lineage and model versioning.
- Define AI model evaluation KPIs, including accuracy, bias, toxicity and hallucination rates.
2. Automation Strategy & Implementation
- Drive automation strategy and implementation using tools such as Python, PyTest, Selenium and Playwright.
- Independently design, develop, execute and maintain automated test scripts and regression test suites.
- Build scalable automation frameworks and reusable testing components for enterprise applications.
- Develop automated validation pipelines and integrate testing frameworks with CI/CD pipelines.
- Perform API testing and validation of RESTful services.
- Drive automation improvements to increase test coverage, delivery efficiency and overall product quality.
- Exposure to performance and load testing tools such as K6 and JMeter is preferred.
3. Generative AI & Agentic AI Quality Engineering
- Design, develop, validate and deploy Generative AI and AI-powered solutions.
- Establish testing methodologies for LLMs, prompt engineering and AI safety evaluation.
- Validate AI agents, multi-agent workflows and autonomous decision-making systems.
- Work with frameworks such as LangChain, LangGraph, AutoGen, CrewAI and Hugging Face.
- Implement Model Context Protocol (MCP) for real-time automation, autonomous workflows and CI/CD integrations.
- Validate RAG architectures, embeddings, vector databases and semantic search implementations.
- Establish continuous monitoring and observability for AI/ML applications.
4. AI Governance, Risk & Compliance
- Establish enterprise AI/ML quality standards, testing guidelines and risk control frameworks.
- Define governance practices for data security, privacy, model lifecycle and versioning.
- Ensure AI solutions meet regulatory, compliance and audit requirements.
- Implement continuous monitoring mechanisms for model performance, drift detection and AI reliability.
- Collaborate with internal risk, compliance and audit teams to ensure governance readiness.
- Support regulatory alignment, particularly in Energy, Utilities and Nuclear environments.
5. Program & Delivery Management
- Lead program and delivery management activities, ensuring projects are delivered on time, within scope, budget and quality expectations.
- Engage with stakeholders to understand business needs, gather project requirements, define scope and align delivery objectives.
- Manage project planning, sprint execution, resource allocation, dependencies and delivery milestones.
- Identify and mitigate project risks, issues and delivery bottlenecks.
- Establish effective stakeholder communication and reporting mechanisms.
- Manage client relationships and ensure alignment between technical teams and business stakeholders.
- Track delivery progress, quality metrics and project outcomes.
6. Cloud, DevOps & CI/CD
- Support the deployment of AI and automation solutions on AWS.
- Implement CI/CD pipelines for ML and LLM models.
- Establish automated evaluation gates, model versioning and lifecycle management processes.
- Support model retraining workflows and infrastructure-as-code practices.
- Implement observability frameworks for AI/ML systems and automated testing environments.
7. Collaboration & Technical Leadership
- Work closely with developers, product managers, data scientists, AI architects and business stakeholders.
- Collaborate with cross-functional teams to improve delivery efficiency, test coverage, product quality and process effectiveness.
- Provide technical guidance and leadership on QE architecture, automation and AI governance.
- Drive defect triage, risk-based testing and quality strategy initiatives.
- Communicate test strategy, project risks, coverage and governance readiness to technical and business leadership.
Required Technical Skills & Qualifications
Core Experience
- 10–14 years of experience as a Technology Architect, SDET, QE Leader, Automation Architect or equivalent technical leadership role.
- Strong programming skills in Python, TypeScript or Java.
- Proven experience designing and implementing enterprise-scale automation frameworks.
- Strong hands-on experience in automation framework development and test script execution.
- Experience managing technical delivery, project execution and stakeholder relationships.
- Strong understanding of SDLC, Agile methodologies and QE practices.
AI / ML Technologies
Practical experience with:
- LangChain, Hugging Face and GPT models.
- LLMs, prompt engineering and AI safety evaluation.
- Vector databases and RAG architectures.
- Agentic AI frameworks such as LangGraph, AutoGen and CrewAI.
- Model Context Protocol (MCP).
- Working knowledge of Scikit-learn, TensorFlow, Keras, PyTorch, Hugging Face Transformers, OpenCV, NLTK and BART.
Automation & Quality Engineering
- Python, PyTest, Selenium, Playwright and Requests.
- Functional, API, integration, regression, performance and security testing.
- LLM testing, hallucination detection and AI model validation.
- REST API testing and automation.
- CI/CD pipeline integration and automated quality gates.
- K6 or JMeter exposure is preferred.
Cloud & DevOps
- AWS cloud platform knowledge.
- CI/CD implementation for AI/ML and automation solutions.
- Model versioning, lifecycle management and retraining workflows.
- Infrastructure-as-Code familiarity.
- AI/ML observability and monitoring