Author: Nadali. Prepared with AI assistance.
Takeaway: count unique customer issues, check outcomes and repeat contacts, then assess automation and its cost.
Who this is for: owners of shops, hotels and small service businesses where AI answers customers or helps support staff.
A guest receives an answer about changing their arrival date. The chat closes. The next day they call reception because the booking still shows the old date. A conversation report might look complete; the business still has one unfinished issue.
The method below is Nadali’s editorial framework, not an industry standard, a vendor formula or a comparative product test.
Read the metric’s definition first
Intercom distinguishes customer-confirmed resolution from assumed resolution when a customer leaves after Fin’s answer without requesting further help. Under its documented rules, returning to the same conversation for assistance reverses the counted resolution. These are product definitions, not independent verification of a business outcome. Intercom documentation.
Intercom also documents a denominator change: excluding conversations where Fin had no opportunity to answer increases resolution rate without increasing resolutions. Check metric versions before comparing periods. Metric update.
In Zendesk, a reply to a closed ticket creates a follow-up ticket instead of reopening the original. A reopening metric alone can therefore miss continuing issues. This describes Zendesk, not every support system. Zendesk explanation.
Check what deflection means in your system, too. If it records only avoided human handoffs, it does not establish the required outcome.
1. Fix the cohort and denominator
Take every new customer issue arriving during one week: your cohort. Define channels and categories beforehand. Exclude spam and internal tests if appropriate, recording the exclusions and their rule. Difficult, handed-off and unfinished issues stay in the denominator.
The counting unit is one customer’s issue concerning a particular order, booking or service. A message, phone call and new ticket about the same booking change belong to one record. Two independent problems inside one conversation may need separate records. Apply the same splitting rule before and after introducing AI.
Record an issue ID, first-contact date, category, linked tickets, AI involvement, human handling, outcome evidence, completion date, repeat contacts and verification status. Link records using existing operational identifiers without copying unnecessary personal data.
2. Define evidence and an observation window
For a booking change, evidence could be the correct date in the booking system. For refunds, specify the required stage: approval, payment initiation and receipt of money are different outcomes. For an information request, check the answer’s accuracy and completeness and any customer feedback. Silence alone is insufficient.
For example, choose seven calendar days of observation after the claimed resolution. Seven days is an illustrative choice, not a universal recommendation. Delivery and refund workflows need a window covering the relevant process.
Set the reporting cutoff beforehand. Count a verified resolution only when outcome evidence exists and the full observation window passed without another contact about the same unresolved issue. Label recent completions “still observing”; keep them in the overall denominator. Compare cohorts with equivalent follow-up time.
3. Connect repeat contacts and human handoffs
Look for continuations in the original conversation, new tickets, email and phone records. Check meaning: “thank you” is not a returning problem; “my refund still hasn’t arrived” may be one across another channel.
A same-issue return invalidates the earlier resolution for this check. After a fix, restart observation and count the eventual resolution once. Preserve the repeat-contact event in the history.
Separate resolutions without operational human intervention from human-assisted resolutions. A later quality audit does not constitute handling the customer’s issue. A handoff can be the right decision; assess whether it produced a completed outcome, rather than rewarding avoidance of human help.
4. Audit a human-reviewed sample
Start, for example, with 30 randomly selected issues each week. This is a manageable starting point, not a guarantee of statistical precision. Check that different issue categories and handling routes are represented. Review complaints and repeat contacts additionally, but keep that targeted sample separate from the random sample’s error rate.
A person checks the complete history, business-policy compliance, required system actions and repeat contacts. Use four labels: verified, unresolved, insufficient evidence and still observing. AI can suggest links between records; a person should check ambiguous matches.
“Three errors among 30 reviewed issues” describes the sample, not an exact population error rate. Do not label unreviewed cases verified. Never divide whole-cohort costs by only the sample’s successful cases.
5. Calculate one coherent example
An illustrative shop receives 200 unique issues. AI participates in 160; humans handle the other 40 directly. An initial report labels 120 AI-resolved: 120 / 160 = 75%.
For simplicity, staff review all 200 issues rather than a sample. Among the initial 120:
- 18 generate another contact about the same unresolved issue;
- a different 12 lack sufficient outcome evidence;
- 90 are resolved without operational human handling, have evidence and pass the complete seven-day window without a repeat.
These groups do not overlap. By the cutoff, another 60 issues have verified human-assisted resolutions, including 10 of the 18 returning issues. Their observation windows have also finished. Therefore:
- resolved without operational human handling: 90 / 200 = 45% of all issues;
- among AI-involved issues: 90 / 160 = 56.25%;
- resolved overall: 150 / 200 = 75%;
- same-issue repeat contact after the initial AI-resolved label: 18 / 120 = 15%;
- the remaining 50 issues are unfinished or unverified.
Assume costs allocated to this cohort total UAH 30,000: UAH 6,000 for tools, UAH 18,000 for human handling and UAH 6,000 for review and maintenance. Include unsuccessful attempts and rework without double-counting. Cost per verified resolution: UAH 30,000 / 150 = UAH 200. These invented amounts illustrate the calculation; they are not market prices.
What the owner should decide
Request the denominator, verified outcomes by handling route, repeats, unverified cases and costs each week. Compare like categories, channels and observation windows. A lower cost does not establish improvement if repeats or unknown outcomes increase.
The audit covers only observable channels and the chosen window. No repeat contact does not prove customer satisfaction, while seasonality or changing issue difficulty can distort comparisons. The useful question is: which customer issues reached a verified outcome, and what did the entire path cost?
Sources and definitions were checked on 8 November 2026. Nadali did not test the named services on customer data; the workflow and numerical example are editorial work.
Read also: how to measure time savings in a small AI pilot and what people should review.
