AI Quality Engineer, Safety and RAG - Assistant Vice President

AI Quality Engineer, Safety and RAG - Assistant Vice President Location(s): Chennai, Tamil Nadu, India, Pune, Maharashtra, India

Design and executeend-to-end test strategies for Agentic AI pipelines, including single-agent and multi-agent workflows. Validateagent reasoning, planning, and decision-making chains (e.g., ReAct, Chain-of-Thought, Plan-and-Execute, Reflexion). Testtool-use correctness — ensuring agents invoke the right tools, with correct parameters, at the right time. Evaluateagent memory systems (short-term, long-term, episodic) for accuracy and context retention across sessions. Validateagent handoff and delegation logic in multi-agent orchestration frameworks (e.g., AutoGen, CrewAI, LangGraph). Testtermination conditions, loop detection, andinfinite loop prevention in autonomous agent loops. RAG (Retrieval-Augmented Generation) Testing Design comprehensive test strategies forend-to-end RAG pipelines — covering ingestion, chunking, embedding, retrieval, reranking, and generation stages. Validateretrieval accuracy and relevance — ensuring the correct context chunks are retrieved for a given query. Testembedding model quality and vector similarity thresholds across different document corpora. Evaluatefaithfulness, groundedness, and answer relevance of generated responses using frameworks likeRAGAS, TruLens, DeepEval. Testchunking strategies (fixed, semantic, hierarchical) for their impact on retrieval quality. Validatecontext window management — ensuring retrieved context does not exceed token limits or degrade generation quality. Conductend-to-end regression testing when the underlying knowledge base, embedding model, or LLM changes. Testmulti-turn conversational RAG for context coherence and citation accuracy across turns. Test Automation Build and maintainautomated test harnesses for Agentic and RAG systems, including agent trajectory replay, tool mock injection, and prompt simulation. Developautomated evaluation pipelines integrated into CI/CD workflows for continuous model and agent validation. Createdata validation and data quality frameworks (using Great Expectations, Deequ, or custom tooling) for training, retrieval, and inference data. Buildprompt regression suites to detect behavioral drift across LLM versions or prompt changes. Implementdeterminism and reproducibility tests for stochastic LLM-based decisions. Automatevector database validation — index integrity, embedding drift, and retrieval consistency checks. AI Safety & Security Testing Conductred-teaming and adversarial testing to uncover jailbreaks, prompt injection vulnerabilities, and goal misalignment in LLM-based systems. Testoutput guardrails and content filters for unsafe, biased, toxic, or out-of-scope model behavior. Validateprivilege escalation controls — ensuring agents do not exceed permitted actions or access unauthorized resources. Performdata poisoning and backdoor attack simulations to assess model robustness. Evaluate models forbias, fairness, and discrimination using frameworks such as AI Fairness 360 and Aequitas. TestPII leakage and data privacy controls in RAG and agent pipelines in accordance with GDPR, CCPA, and internal data governance policies. Conduct security testing aligned with theOWASP Top 10 for LLM Applications, including: Prompt Injection (Direct & Indirect) Insecure Output Handling Training Data Poisoning Validateconstitutional AI constraints, RLHF-aligned behavior boundaries, and system prompt integrity. Collaborate with cybersecurity teams onAI-specific threat modeling and vulnerability management. Maintainsafety testing playbooks and document red-team findings with severity ratings and remediation recommendations. Performance & Reliability Testing Define and executeload, stress, soak, and spike testing for AI-powered APIs, inference endpoints, and agent orchestration services. Measure and optimizeend-to-end latency across RAG and agentic pipelines — from query to final response. BenchmarkLLM inference throughput (tokens/second) and identify bottlenecks across model serving infrastructure. Testauto-scaling behavior of AI services under variable load conditions. Validatecircuit breaker, retry, and fallback mechanisms in agentic and RAG systems for graceful degradation. Testvector database performance — query latency, index build time, and retrieval accuracy under high concurrency. Conductcost efficiency analysis — measuring token consumption, API call costs, and infrastructure spend per agent task. EstablishSLOs (Service Level Objectives) andSLAs for AI system availability, latency percentiles (P50, P95, P99), and error rates. Collaborate with MLOps teams to set upobservability dashboards, monitoring alerts, and automated anomaly detection for production AI systems. Performchaos engineering experiments to validate agent and RAG system resilience under infrastructure failures. Domain Knowledge Deep understanding ofRAG architecture patterns — naive RAG, advanced RAG, modular RAG. Solid grasp ofagent design patterns: ReAct, Plan-and-Execute, Reflexion, MRKL, Mixture-of-Agents. Familiarity withAI safety and alignment principles (RLHF, Constitutional AI, guardrail layers). Knowledge oftoken economics, context management, and LLM cost optimization. Proficiency inperformance engineering methodologies for distributed AI systems.