What can an AI summary really tell you about your customers?

The practical question is not simply “What do customers want?” It might be “Which delivery questions should our product page answer?” That gives the analysis a useful boundary. A decision about page content needs different evidence from a claim about the main reason customers leave.

Decide whose feedback you are counting

Start by describing the corpus: the collected texts you will analyse. Record the channels, period, languages, products and exclusions. Note whose views are missing, such as customers who never contact support. Use authorised data in approved tools and remove unnecessary personal information before processing.

Statistics Canada's guidance on non-probability sampling explains why voluntary responses can favour strongly motivated people. Applied here, a pattern in support tickets describes those tickets, not automatically the customer base. Drawing tickets at random does not remove the original selection of people who contacted support. The comments still matter; the limits affect the claims you make from them.

Choose the counted unit before reporting a frequency. Counting messages, conversations or unique customers produces different totals: a long exchange can otherwise look like several independent complaints. State the denominator too—the total number of eligible units in the collection. Remove duplicates and explain how repeated contacts are handled. One conversation may carry several themes, so theme totals need not add up to the number of conversations.

Give exploration, checking and counting different jobs

First read some feedback yourself, then let AI suggest possible themes. Define their boundaries with examples, including when several labels are allowed. A word count or a negative sentiment label cannot tell you the customer's intention, let alone explain a cancellation.

Next prepare a deliberately varied set of cases for human checking. Include different channels and languages, short and long texts, mixed requests, unclear wording and rare problems. Google’s text-classification guides recommend inspecting examples and labels and covering the inputs a system may encounter. Those guides concern supervised learning; using these principles for an AI review workflow is our adaptation.

Keep this checking set separate from the data used to estimate theme frequency. You may include extra rare cases to test whether the system can read them. Their resulting share of that set says nothing about how common the issue is. Preserve a defined counting base and report frequencies only within its stated boundaries.

Keep the evidence beside the interpretation

For each theme, retain the source identifier and an exact supporting excerpt. Check it against the original text or recording. Label translations and AI paraphrases explicitly; neither should appear as an original customer quotation.

The Government Digital Service separates observations, findings and actions in its human research guidance. MindVerve proposes carrying that distinction into AI-assisted analysis. Ask reviewers to inspect both the labels and the evidence offered for them, rather than approve a fluent summary alone.

Reviewers may disagree. Resolve the difference by returning to the source and category definition, or leave the item unknown. Keep contrary evidence and rare, potentially serious issues visible. Anthropic's support-routing documentation highlights implicit requests, emotion and multiple issues as classification difficulties. Its vendor guidance is useful context, not an independent marketing benchmark or a reason to force every comment into one theme.

Follow a delivery complaint through to a next check

Consider this fictional feedback: “The order arrived within the promised window, but I had to contact you to find out which day it would come.” The observation is that this customer reports an on-time delivery alongside difficulty finding the date. A suitable theme is unclear delivery information; delivery delay is not supported by this excerpt.

The hypothesis is that delivery information may be hard to find at a particular point in the journey. That is an explanation to investigate, not a demonstrated cause. As a next check, the content lead could inspect the confirmation message and tracking page, then ask customers to show where they look for the date.

Other comments may describe genuinely late deliveries or clear, helpful updates. Keep those alongside the proposed theme. Neither the fictional comment nor a cluster of similar comments proves why customers cancel; that causal question needs additional evidence.

Test a stronger model against a specific remaining problem

A more capable model is worth trying when task-specific reading errors persist despite adequate approved context and clear categories. If human reviewers disagree over what a category means, settle that definition first.

Compare models on the same varied cases, using human judgements as the reference. Set these cases aside beforehand and do not use them to refine the prompts. Inspect error types, fidelity to sources, treatment of ambiguity, total review and correction time, response time and cost. A model that produces smoother prose may still blur two distinct complaints.

This is a proposed evaluation method, not a claim about measured MindVerve results. A sampled quality check also cannot guarantee that every item in the wider collection has been classified correctly.

Bring a traceable record to the decision meeting

Copy this record for each finding so colleagues can check the evidence before deciding what to investigate.

1. Theme: a plain-language definition and any relevant boundaries.
2. Counted unit and denominator: what is counted, how many items carry the theme and the total eligible base.
3. Corpus boundaries: channels, dates, languages, products, exclusions and duplicate handling.
4. Source evidence: original identifiers and verified excerpts, with translations or paraphrases marked.
5. Contrary or ambiguous evidence: disagreements, unresolved cases and rare high-impact issues.
6. Next investigation and owner: the question to check, the evidence needed and the person responsible.

Where that task is defined, AI Workflow Launch can configure one research, reporting or content workflow, including scoped tool selection, tests covering typical workflow cases and failure scenarios, reusable briefs and prompts, documentation, training and handover. Licences, API usage and additional integrations are excluded; ongoing support and maintenance are separate. If the task, data or success criteria remain unclear, AI Opportunity Sprint can help establish priorities and scope.