Ep.6: How Real-World Messiness Impacts AI Agent Success

Ep.6: How Real-World Messiness Impacts AI Agent Success

In this episode of the Cisco AI Insights Podcast, hosts Rafael Herrera and Sónia Marques are joined by Cisco ML Engineer Paul Mutawe to explore the fascinating paper, "Measuring AI Ability to Complete Long Software Tasks," which introduces a novel time horizon metric to evaluate how autonomously AI agents can execute complex, multi-hour engineering projects. The discussion looks at the rapid evolution of these agents, highlighting the key finding that the fifty percent success time horizon is doubling every two hundred and seven days, while detailing how unbiased benchmarking environments like the Modular Public harness are used to evaluate frontier models alongside real-world complexities like the sixteen-item messiness factor, which significantly reduces agent success rates, and the critical need for human-in-the-loop oversight to combat context rot. A special thank you to the research team at Model Evaluation and Threat Research, who developed this paper. If you are interested in reading the paper yourself, please visit the link: https://arxiv.org/pdf/2503.14499

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(500)

The evolution of good technology | Tech Unscripted

The evolution of good technology | Tech Unscripted

In this episode, industry leaders discuss how to redefine “good” technology today — weighing speed, agility and AI against security, ethics and human impact. Panelists share real examples, from digita...

14 Aug 33min

Scaling AI: The Playbook for IT Services Delivery

Scaling AI: The Playbook for IT Services Delivery

Cisco Technical Assistance Center (TAC) manages 1.4 million SRs annually, and today, 97% are touched by AI. As we shift from task-based automation to Agentic AI, some of these interactions are becomin...

14 Aug 29min

404 Script Not Found: Part 2! AI Deep Dive with Annie Hardy

404 Script Not Found: Part 2! AI Deep Dive with Annie Hardy

Back for part two of our split episode with the founder of Split Futures, Annie Hardy and jump right in where we left off...getting practical about choosing the right AI model—because apparently not e...

13 Aug 24min

404 Script Not Found: AI Deep Dive with Annie Hardy

404 Script Not Found: AI Deep Dive with Annie Hardy

Your company bought all the AI tools and now nobody knows what to do with them. Sound familiar? Kat and Ian sit down with Annie Hardy, founder and Chief AI Architect at Split Futures, to unpack why l...

11 Aug 19min

Cisco Tech Stories Season 4 Kickoff: Top Episodes & Wild Tech Tales

Cisco Tech Stories Season 4 Kickoff: Top Episodes & Wild Tech Tales

Episode 37 kicks off Cisco Tech Stories' fourth season with a highlights reel of the best episodes from the past three years. Hosts Nico and Tristan revisit standout interviews and mini‑series, sharin...

2 Aug 43min

Shaping Cisco’s Portfolio Strategy with Jeetu Patel and Jeff Schultz

Shaping Cisco’s Portfolio Strategy with Jeetu Patel and Jeff Schultz

Jeff Schultz joins Jeetu Patel to delve into the evolution of the Cisco Portfolio and how this focus on critical business outcomes for the AI era is delivering real results for the ecosystem.

30 Juli 35min

Building Future-Proofed Workplaces with Jeetu Patel and Anurag Dhingra

Building Future-Proofed Workplaces with Jeetu Patel and Anurag Dhingra

Jeetu and Anurag Dhingra explore the transformation of physical infrastructure and intelligent software as the foundation for the workplace. As the nature of work continues to evolve, the demand for e...

30 Juli 28min

Scaling AI: Enterprise Requirements of Agentic AI

Scaling AI: Enterprise Requirements of Agentic AI

95% of enterprise IT leaders plan to launch agentic AI projects — but 69% aren't ready. So, what's standing in the way?  In this episode of Scaling AI in the Enterprise, we bring together an expert p...

30 Juli 33min

Populärt inom Teknik

uppgang-och-fall
market-makers
skogsforum-podcast
rss-uppgang-och-fall
rss-laddstationen-med-elbilen-i-sverige
rss-elektrikerpodden
rss-en-ai-till-kaffet
natets-morka-sida
 och-bilen-gar-bra
bli-saker-podden
rss-veckans-ai
under-femton
hej-bruksbil
elbilsveckan
developers-mer-an-bara-kod
bosse-bildoktorn-och-hasse-p
rss-fabriken-2
garagehang
klocksnack-tillsammans-med-nymans-ur-1851
bilar-med-sladd