Engineering Product Manager

Meet The Team: The Cisco AI Software & Platform Group incubates and delivers mission-critical Generative AI and platform infrastructure across Cisco's global product portfolio. Our group builds foundational evaluation, benchmarking, and observability platforms that empower engineering teams to deploy autonomous AI agents and LLM-powered systems safely, reliably, and with verified accuracy at enterprise scale.

Your Impact: As an Engineering Product Manager you will lead the strategy, architecture, and organizational adoption of Cisco's end-to-end evaluation platform: Define AI Evaluation Platform Strategy: Architect the product vision, staged roadmap, and platform architecture for Cisco's unified offline benchmarking suites, automated LLM judges, and continuous production evaluation systems. Operationalize Quality & Safety Standards: Convert ambiguous quality standards (e.g., factual accuracy, hallucination mitigation, safety policies, agentic tool-use fidelity) into objective, repeatable test suites and structured human-in-the-loop review pipelines. Establish Release Gating & Quality Bars: Establish pre-deployment quality bars, risk-based release criteria, and automated regression-detection loops to govern AI model and application rollouts. Bridge Offline & Production Telemetry: Connect offline model evaluations directly with live telemetry, product analytics (Amplitude), and real-time user feedback loops to rapidly detect and remediate performance regressions. Drive Multi-Team Platform Adoption: Partner deeply with research scientists, engineering leaders, and product managers across business units to drive organization-wide adoption of standardized evaluation frameworks.

Minimum Qualifications: Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field with 12+ years of product management experience, OR Master's degree with 8+ years, OR PhD with 5+ years of experience. Prior experience leading product management for AI/ML platforms, model evaluation frameworks, Large Language Model (LLM) infrastructure, or data-intensive software products. Applied background designing and operationalizing offline benchmarking systems, automated evaluation judges, or real-time production quality monitoring pipelines. Previous experience defining risk-based launch criteria, release-gating frameworks, and regression mitigation workflows for software or AI systems. Proven experience driving multi-team platform adoption across matrixed engineering and product organizations. Preferred

Qualifications: Direct experience evaluating Generative AI, Retrieval-Augmented Generation (RAG) pipelines, and autonomous multi-agent systems. Familiarity with modern AI evaluation toolkits (e.g., DeepEval, Ragas, LangSmith, TruLens) and enterprise telemetry architectures. Prior experience operating across enterprise SaaS, cloud infrastructure, or hyperscaler environments. Demonstrated background in statistical evaluation methodologies, automated LLM-as-a-judge calibration, and human-in-the-loop annotation workflows. Practical experience with online evals for agent systems: observing production traces and trajectories to diagnose failure patterns and decide which techniques to apply to make the agent work better.