Managing AI Agent Performance Degradation in Production | Kumaran Ponnambalam, Cisco

Kumaran Ponnambalam, Principal AI Engineer at Cisco, presents a session on ‘Agent Drift’ at the MLOps World | GenAI Summit 2025. He defines agent drift as the gradual performance degradation of AI agents in production due to evolving data, user behavior, tools, or model parameters. He contrasts it with classical ML drift, highlighting additional variables specific to agents: goal drift (changes in user input characteristics), context drift (changes in external data sources), reasoning drift (model performance changes), and collaboration drift (issues with tool/agent interactions). He describes two measurement techniques: distribution comparison using statistical tests (chi-square, KS) and slope analysis of aggregated metrics. He presents results from a renewals agent, showing that fine-grained drift metrics can explain overall drift. Root causes include mismatches between evaluation and production data, silent changes to backend systems, model upgrades, instruction changes, and frequent data source changes. Best practices include rigorous evaluation, continuous calibration, versioned APIs, strict input/output controls, context engineering, threshold experimentation, user training, and investment in tracing tools. The talk concludes with a Q&A on sample size requirements and seasonal variations.