Why asking your team works better than usage data alone for AI ROI

There's a standard argument in AI tool measurement that goes: "don't trust self-reported surveys, they're biased. Trust the usage logs, they're objective." It has an appealing ring of empiricism. As a principle for measuring AI ROI, it is half-right at best.

The GAiGE is built around asking your team short questions, little and often. We get the "but surveys are subjective" objection often enough that this post is worth writing. Here's our answer.

What usage data is good at#Link to this section

Usage data (the output of the vendor's admin panel, your own network logs, or your device management inventory) answers a set of questions completely:

• Did anyone log in? A clear signal of activation.
• How often does an account get used? Session frequency per seat.
• Which features get touched? Granular event data.
• What's the usage curve over time? Adoption speed, seasonality and decline.

All of that is valuable. The question around it, the one your CFO is asking, is: was any of this worth the money?

What sits outside usage data#Link to this section

Three big gaps. They're different from each other, but each is enough on its own to make usage data an incomplete picture.

1. Use isn't value. A person can "use" a tool every day and get no meaningful output from it. Drafting work in ChatGPT and then scrapping the draft still counts as a session. Running a search in an AI assistant and not trusting the answer still counts. The vendor's usage meter clicks up; your team's productivity doesn't.

2. Value isn't use. The inverse, and more common than you'd think. A tool can deliver a great deal of value through occasional, well-chosen use. A senior engineer who opens Copilot twice a week to solve the gnarly bit of code they'd otherwise spend hours on produces little usage data and real value. The usage meter reads "low engagement: consider cutting this seat"; the work says "keep it".

3. Quality, confidence, fit and fear are questions for people. Is the output reliable? Does the person trust it? Are they worried about sharing information with it? Is the tool replacing a workflow, or layered awkwardly on top of one? These are the questions that predict renewal, expansion, and whether your AI investment will still look healthy in 18 months, and they're answered by asking.

What surveys are good at#Link to this section

Asking people, properly, is how you reach the signals usage data can't:

• Did it actually help?
• Did it make the output better, not just faster?
• Do you trust what it gave you?
• Would you miss it if it was gone?
• What's getting in the way of using it more?

These answers are subjective by definition: no log file will ever produce them. That's the job they do.

Where surveys fail (if you don't design them carefully)#Link to this section

The standard critiques of surveys are real and worth taking seriously:

• Response bias. Happy users respond more, or unhappy users respond more. Either way you get a skewed picture.
• Recall error. Asking "how has AI been this month?" is asking someone to construct a narrative from memory, and memory is compressed and biased.
• Survey fatigue. A 20-question engagement survey once a year gets ignored. Or worse, filled in at random.
• Gaming. If people think the answers will influence what tools their team gets to keep, they'll respond strategically.

Each of these has a fix. The fixes are what make the difference between useful survey data and useless survey data.

The fixes we use in The GAiGE#Link to this section

Ask little and often, not once a quarter. The browser extension shows short pulses on a cadence you control, a few times a week, and every pulse can also be answered in the AI Adoption Portal. "Roughly how much time did ChatGPT save you today?" is a very different question from "on average, over the last year, how much time would you say this tool has saved you?" The first is a recall problem of hours. The second is a recall problem of months.

Keep it short. The AI Pulse is three or four questions and takes about thirty seconds. Short and frequent beats long and ignored.

Ask anonymously by default. We tell your people: "When a pulse is anonymous, your name isn't shown with your answers to anyone, admins included." Every figure from an anonymous pulse needs answers from at least five people. In a small group, or from something written in a comment, someone may still be able to work out whose answer it is, and we tell people that too. Gaming falls away when the person answering knows who will see what.

Bound the maths. Our 2.5× cap on extrapolation, and the response rate shown beside every figure, stop one enthusiast's answers being projected across the whole team. If the response rate is 40%, we tell you, and you should discount the total accordingly.

The combined signal#Link to this section

Neither source is enough on its own. Both together give you something much better than either alone. Three examples:

High usage, low satisfaction. The "sticky but not loved" pattern. The tool has wedged itself into a workflow people can't escape, but no one's enthusiastic about it. A renewal risk when something better shows up, and a reason for AI learning now.

Low usage, high satisfaction. The expert user pattern. The tool isn't touched every day, but when it is, it solves expensive problems. Cutting the seat on activity alone would lose that value.

High usage, high satisfaction, high time saved. Your winner. Buy more licences.

In short#Link to this section

Usage data tells you who used each tool and how often. Asking your team tells you whether it helped. Every AI ROI conversation needs both.

Questions about our methodology? The 2.5× rule post is the full walk-through of how we turn pulses into defensible numbers, and How we work out hours saved shows the working.