Ep.6: How Real-World Messiness Impacts AI Agent Success

Ep.6: How Real-World Messiness Impacts AI Agent Success

In this episode of the Cisco AI Insights Podcast, hosts Rafael Herrera and Sónia Marques are joined by Cisco ML Engineer Paul Mutawe to explore the fascinating paper, "Measuring AI Ability to Complete Long Software Tasks," which introduces a novel time horizon metric to evaluate how autonomously AI agents can execute complex, multi-hour engineering projects. The discussion looks at the rapid evolution of these agents, highlighting the key finding that the fifty percent success time horizon is doubling every two hundred and seven days, while detailing how unbiased benchmarking environments like the Modular Public harness are used to evaluate frontier models alongside real-world complexities like the sixteen-item messiness factor, which significantly reduces agent success rates, and the critical need for human-in-the-loop oversight to combat context rot. A special thank you to the research team at Model Evaluation and Threat Research, who developed this paper. If you are interested in reading the paper yourself, please visit the link: https://arxiv.org/pdf/2503.14499

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(500)

Why Resilience Needs the Human Element | Tech Unscripted

Why Resilience Needs the Human Element | Tech Unscripted

Host and guest Evan Mintzer (Director of Production Infrastructure, Customers Bank) discuss how banks build digital resilience through network design, redundancy, data visibility and governance. The e...

25 Sep 27min

Cisco Optics Podcast Ep66: Confronting the Physics of 400G per Lane - John Calvin, Part 3 of 8

Cisco Optics Podcast Ep66: Confronting the Physics of 400G per Lane - John Calvin, Part 3 of 8

In the pluggable optics industry every next generation is led by countless presentations and proposals at standards bodies meetings along with lots of lab development and testing behind the scenes. Ri...

25 Sep 17min

404 Script Not Found: Cisco Kool-Aid

404 Script Not Found: Cisco Kool-Aid

Ian's got two bone-to-pick this week: a guy lifting in slides, and a CVS receipt that could legitimately double as a scarf. Kat's here to talk him down from both grievances, and once she does, they pi...

24 Sep 15min

404 Script Not Found: Social Selling with a Purpose

404 Script Not Found: Social Selling with a Purpose

Ian is flying solo BUT is joined by LinkedIn expert Darren McKee this week—founder of 531 Social and Darren McKee Co. Darren's been posting on LinkedIn almost every single day since March 2020, and ...

18 Sep 28min

Efficient IT Management with AI | Tech Unscripted

Efficient IT Management with AI | Tech Unscripted

This episode explores how AI agents, network sensors, and cloud-managed Meraki solutions are transforming workplaces across industries—from stadiums and events to critical infrastructure. Leaders sha...

14 Sep 35min

Cisco Optics Podcast Ep65: The Tradeoffs Shaping the Race to 3.2T - John Calvin, Part 2 of 8

Cisco Optics Podcast Ep65: The Tradeoffs Shaping the Race to 3.2T - John Calvin, Part 2 of 8

In the pluggable optics industry every next generation is led by countless presentations and proposals at standards bodies meetings along with lots of lab development and testing behind the scenes. Ri...

11 Sep 15min

404 Script Not Found: The Learning Curve

404 Script Not Found: The Learning Curve

This week, Kat is speaking Spanish, Ian is preparing to run an ungodly number of miles, and they’re both making fun of men with guitars—so naturally, it’s time to talk about onboarding. With Ian’s new...

10 Sep 17min

404 Script Not Found: Career Changes

404 Script Not Found: Career Changes

Our best friend (and career coach) Sammy Brenner is back with a new role, a new language to learn, and approximately six months before she’s ready to teach any of it to us (her words). Sammy and Kat...

3 Sep 25min

Populært innen Teknologi

teknisk-sett
tomprat-med-gunnar-tjomlid
lydartikler-fra-aftenposten
energi-og-klima
elektropodden
rss-ki-praten
hans-petter-og-co
nasjonal-sikkerhetsmyndighet-nsm
shifter
smart-forklart
rss-alt-som-gar-pa-strom
rss-ai-forklart
teknologi-og-mennesker
rss-snakk-om-sikkerhet
rss-kunstig-intelligens-med-elisabeth-maren-og-morten
fornybaren
rss-teknologioptimistene-en-podkast-om-teknologi-og-mennesker
pedagogisk-intelligens
rss-alt-vi-kan
rss-heis