Why Containment Rate by Intent Is the KPI That Predicts Chatbot Success

Stop Chasing Deflection. Measure Containment by Intent.

Deflection, the count of chats that don’t reach agents, misses the point. It obscures what truly matters. Containment rate by intent cuts through the noise, showing exactly where your chatbot resolves user issues. This metric reveals which use cases actually work, which fall short, and where your next improvements should go. If you want a KPI that reliably predicts chatbot success, start here.

Containment by Intent, Defined

Containment means your bot fully handles the user’s issue without involving a human agent. Measuring it by intent means you analyze how well the bot performs for each specific purpose, such as order status or refund policy. This targeted approach provides real insights, moving past misleading overall averages to reveal true performance by use case.

Containment Rate for an Intent = number of conversations for that intent resolved entirely by the bot ÷ total number of conversations for that intent.

Two chatbots might have the same overall containment rate, but that doesn’t mean they function identically. One may be highly effective at password resets while missing the mark on refunds. Without intent-level detail, you risk overlooking these critical gaps and making misguided improvements.

Why This KPI Predicts Success

• It’s a leading indicator. While CSAT lags, containment by intent shifts immediately as flows or prompts improve.
• It isolates ROI. See clearly which intents reduce handle time or ticket volume, and forecast savings with confidence.
• It guides your roadmap. Investment is typically directed toward the areas with the most significant differences or changes. With this data, you’ll know exactly which intents need content, connectors, or better handoff logic.
• It exposes risk. A dip in containment for a high-volume intent provides an early warning before support queues spike.

How to Calculate It Without Drama

Keep your measurement process clear and consistent. Track with machine-readable events using this simple formula:

CR_intent = bot_resolved_intent / total_intent_conversations

For every conversation, capture three key details:

1. Intent_assigned at the first confident bot turn.
2. Outcome, whether the bot resolved the query, handed off to a human, or the user abandoned the session.
3. Reason for any handoff, such as authentication failure or missing data.

Example breakdown:

• Order status: 1,200 conversations; Bot resolved 972. CR = 81%.
• Password reset: 600 conversations; Bot resolved 540. CR = 90%.
• Refund request: 400 conversations; Bot resolved 168. CR = 42%.

This reveals where your focus is needed, for instance, refund requests require better flows or policy logic, while other intents are performing well.

Targets and Guardrails That Actually Help

There are no universally accepted target rates for containment by intent. It’s best to set your own benchmarks based on the complexity and risk associated with each intent:

• Transactional, low risk (status lookups, FAQs): aim for 80% to 90%.
• Transactional, medium risk (password reset, address change): 70% to 85%, with strong guardrails.
• Policy or judgment (refunds, cancellations): 35% to 60%, with seamless handoff.

Consider these tiers as starting points for your own strategy. It’s essential to work with your legal and quality assurance (QA) teams to ensure compliance and performance, adjusting for your specific call volumes and seasonal variations.

Instrumentation Checklist You Can Ship This Week

• Define a clean intent taxonomy. Focus on the top 20 to 40 intents. Group the long tail under Other for now.
• Log the intent at conversation start. Update if confidence shifts. Keep a revision history.
• Mark the explicit outcome. Don’t assume silence means resolution, use a clear done state.
• Track handoff cause codes. Keep these concise and relevant.
• Store bot actions. Note which tool integrations succeed or fail.
• Separate channels. Web chat may perform differently from WhatsApp or email.

{  conversation_id: abc123,  channel: web,  intent: refund_request,  intent_confidence: 0.86,  outcome: human_handoff,  handoff_reason: policy_edge_case,  tools: [order_lookup:success, refund_api:denied]}

What the Trend Line Tells You

Monitoring containment rate by intent highlights underlying causes quickly:

• Flat containment with rising volume: Scale up infrastructure or optimize guardrails for faster checks.
• Drop after a policy change: Update your knowledge base or revise prompts.
• Low containment, high CSAT: The bot empathizes well but hands off often, expand capabilities.
• High containment, low CSAT: The bot resolves queries but frustrates users, improve flow and language.

Fix the Common Failure Modes

• Mislabeled intents. Periodically sample and re-tag conversations. Retrain using difficult examples.
• Silent transfers. Log every human join with a reason code, these count as not contained.
• Multi-intent threads. Score based on the primary intent, recording secondary intents separately.
• Policy drift. Tie prompts to a single source of truth and set alerts for policy changes.
• Tool reliability. If APIs are unreliable, add retries and circuit breakers to prevent failure.

For a detailed troubleshooting guide, see this resource on diagnosing and fixing chatbot failures.

How to Act on Low Containment

1. Improve the Journey, Not Just the Prompt

Analyze the user’s path: Are data requests too late in the flow? Does authentication fail often? Address blockers early and add clear confirmation steps.

2. Add the Missing Tools

Bots stall when they can’t complete tasks. Connect relevant APIs, manage permissions, and log all failures for review.

3. Teach with Real Phrasing

Use failed-case transcripts as training data, keep examples concise and label them by intent.

4. Design the Handoff

Ensure agents receive full conversation context, including intent, tool results, and the end user’s goal. This helps agents resolve issues without forcing customers to repeat themselves.

Reporting That Teams Actually Use

• Leaderboard by intent. Rank by conversation volume, then by containment rate.
• Weekly delta. Highlight changes in containment for your top five intents.
• Top handoff reasons. Rank reasons both by count and impact on handle time.
• Agent save rate. Track how often agents successfully resolve escalated issues.

Distribute these reports within your CRM or help desk tools, keep them accessible and relevant to daily operations, rather than buried in dashboards.

Tooling Landscape in Short

There are various platforms capable of supporting intent analytics and containment tracking. When choosing, consider whether each solution offers essential features such as intent logging, outcome tracking, and tool call logging. If a vendor cannot provide intent-level containment reporting, you won’t be able to accurately manage performance.

Forecasting with Intent-Level Data

Intent-level containment data makes forecasting straightforward. Multiply your expected conversation volume for each intent by your target containment rate. For the remaining unresolved cases, multiply by average agent handle time to calculate staffing needs. As containment improves on a specific intent, you’ll see precisely how this impacts support hours and cost.

Governance and Quality

Containment should not be the only priority. Pair it with robust guardrails:

• Safety: Define boundaries; certain refund or legal issues must always transfer to a human.
• Accuracy: Audit a sample of resolved conversations for each intent every week.
• Fairness: Review prompts and outcomes to check for unintentional bias.
• Privacy: Restrict personal data in prompts and maintain strict logs on sensitive access.

Set consistent review schedules. Ensure responsible owners sign off on high-risk intents, and document what your bot can and cannot do.

Takeaways You Can Act on Today

• Identify your most important intents and outcomes. Start logging them immediately.
• Publish and share a simple intent leaderboard weekly.
• Choose one low-performing intent and focus on improving the journey, then the prompt.
• Review handoff reasons regularly, prioritize removing barriers each sprint.

This method works because it matches your team’s efforts to real user problems. You’ll see meaningful progress in days, not months.

Ready to Measure What Matters?

If you’re ready to set up intent-level containment tracking and robust QA loops, we can help. Typewise integrates seamlessly with your existing stack and ensures clear, accurate, and consistent responses. Start a conversation with us at typewise.app and see how this KPI can drive lasting results.

FAQ