Build self-improving AI agents: Turn failing traces into higher pass@k

Improving your agent from its traces

This is the heart of the series — and the step most teams skip. Kwasi Ankomah shows how to read an agent’s failing runs, find the patterns behind them, and automatically improve the agent without any fine-tuning.

About this webinar

Once you can measure an agent, the next question is how to make it better. Most teams stop at evaluation and fix problems by hand, one prompt tweak at a time.

In Part 4, we built an evaluation harness with pass@k, trajectory checks, and full tracing. In Part 5, Kwasi Ankomah puts those traces to work. He analyzes the agent’s failing runs, diagnoses the recurring failure modes, and then updates the agent’s instructions, memory, and tools automatically — using off-the-shelf libraries rather than a custom pipeline. Because the test suite gives a hard signal, the improved agent is A/B tested against the baseline so you can watch pass@k climb.

What you will learn

• How to read and analyze an agent’s failing traces to find what’s really going wrong
• How to group individual failures into recurring failure modes
• How to automatically update an agent’s instructions, memory, and tools based on those failures
• How to do all of this with off-the-shelf libraries instead of building a bespoke pipeline
• How to A/B test the improved agent against the baseline and measure the gain in pass@k — no fine-tuning required

Why improving from traces matters

Every failed run contains information about how the agent could do better. Teams that ignore their traces keep fixing the same problems by hand. Teams that learn from them build agents that steadily improve. Because this approach changes the agent’s instructions, memory, and tools rather than the model’s weights, it’s faster, cheaper, and easier to roll back than fine-tuning.

Who should attend

AI and ML engineers running agents in development or production, data scientists working on LLM evaluation and optimization, and technical leads looking for a practical, repeatable way to improve agent performance. Following Part 4 is helpful but not required.

Kwasi Ankomah is the Lead AI Architect at SambaNova Systems, where he builds production agentic AI systems on purpose-built RDU inference infrastructure. With 15 years of experience spanning financial services, consulting, government, and tech startups, Kwasi bridges deep technical expertise with a hands-on understanding of real business challenges. He specializes in deep agent architectures, multi-agent orchestration, and context engineering. Connect with Kwasi on LinkedIn →