Agents: Alert the owner when thumbs-down, tool errors or step limits spike
Fin Monitors evaluate live conversations against standards you define and alert when a threshold is crossed (https://fin.ai/testing, https://fin.ai/pricing). We record ratings in agent_feedback, tool errors in agent_tool_calls and the step_limit status in runAgent (lib/agents.ts), but only handoffs produce a notification (lib/agent-tools.ts). A small team will not watch a dashboard daily; a push when something breaks (for example a Cal event type deactivated making every booking fail) is what keeps an agent safe to leave running. First scope: hourly cron check per agent for tool error rate, thumbs-down count and step_limit count over the last 24h vs threshold, sending one in-app and email notification with a link to the failing conversations. Priority: medium. Effort: S. Compared with Intercom Fin, Zapier Agents, Lindy (plus OpenAI AgentKit as reference for evals/versioning).
Comments 0
No comments yet.
Sign in to comment