ASAPP · 2025
In-context bot testing
A real-time preview for testing AI agent behavior across draft, sandbox, and production, right inside the configuration workspace.
- Role
- Staff Product Designer, led UX and UI
- Timeline
- 2025
- Team
- Product and engineering
- Scope
- Internal tooling and customer evaluations
Overview
Bot testing lived in a separate internal tool that only worked in production. To check a change, people had to leave the configuration, deploy, or dig through backend logs, and there was no safe place to try drafts. I designed and led a real-time preview that brought testing into the configuration workflow, across draft, sandbox, and production.
The product
Test where you configure
Side-by-side testing
The preview slides in beside the configuration, so changes can be tested without losing context.
Environments and testing tools
Switch between draft, sandbox, and production, and test with mock users, authentication, and edited or regenerated responses.
Debugging with bot thoughts
Show the agent’s reasoning: the task it enters, the APIs it calls, and the knowledge it uses.
Part of the AI agent tooling platform →
Test without leaving
Configuration and preview sit side by side, so changes show up immediately and focus stays on the task.
Safe environments
Separate draft, sandbox, and production make testing safer and keep stable states apart from experiments.
Make reasoning visible
Bot thoughts expose tasks, API calls, and knowledge base articles, so people can debug on their own.
Outcome
From internal tool to sales asset
The preview replaced a production-only tool with real-time, in-context testing and became a base for debugging, QA, and onboarding. It also became one of the strongest assets in live demos, letting prospects experience the agent before signing.