How to audit what your AI support agent promised customers

Why conversations are not enough

Most support QA reads the conversation and scores tone, accuracy and resolution. An AI agent can pass all three and still leave a customer without the refund it promised, because the conversation never shows whether the action happened.

An audit of promises compares two things: what the agent said it did, and what your store's systems show it did.

What counts as a promise

• A refund, full or partial, with or without an amount.
• An order cancellation.
• A return or a replacement.
• A subscription paused, skipped or cancelled.
• A discount code sent.

Store credit, apologies and general reassurance are not promises you can check against a system, so leave them out.

The weekly check

• Take a sample of last week's tickets your AI agent answered, around twenty.
• Write down each promise: what kind, the amount, the order number, and whether it depended on something (for example "once we receive your return").
• For each promise, look for the matching record: a refund on the order in Shopify, a cancelled order, a return, a subscription status in Stripe, a discount code that exists.
• Sort what you find into three groups: done as promised, done differently (wrong amount or wrong order), and not done.
• Fix the customers in the last two groups first. Then look for the pattern, because the same cause usually repeats.

Turn every mistake into a guidance line

Each pattern you find becomes one sentence in your AI agent's guidance. Two that fix the most common cases: "Only confirm an action to the customer after it has run and succeeded," and "When a customer has more than one order, repeat the order number back and get a yes before acting."

Do it faster

Our free promise checker pulls the promises out of a single reply. A free 30-day audit does the whole check for every ticket, read-only, and sends you the cases to fix.