Why asking your team works better than usage data alone for AI ROI

Usage data tells you who logged in. It doesn't tell you whether anything actually helped.

There's a standard argument in AI tool measurement that goes: "don't trust self-reported surveys, they're biased — trust the usage logs, they're objective." It has an appealing ring of empiricism. It is also, as a principle for measuring AI ROI, half-right at best.

The GAiGE is built around asking your team short, contextual questions. We get the "but surveys are subjective" objection often enough that this post is worth writing. Here's our honest answer.

What usage data is good at

Usage data — the output of the vendor's admin panel, or your own network logs, or MDM inventory — is genuinely great for a narrow set of questions:

Did anyone log in? Binary signal of activation.
How often does an account get used? Session frequency per seat.
Which features get touched? Granular event data.
What's the usage curve over time? Adoption velocity, seasonality, decline patterns.

All of that is valuable. None of it answers the question your CFO is actually asking, which is: was any of this worth the money?

What usage data misses

Three big gaps. They're different from each other, but each is enough on its own to make usage data an incomplete picture.

1. Usage ≠ value. A user can "use" a tool every day and get no meaningful output from it. Drafting work in ChatGPT and then scrapping the draft still counts as a session. Running a search in an AI assistant and not trusting the answer still counts. The vendor's usage meter clicks up; your team's productivity doesn't.

2. Value ≠ usage. The inverse, and more common than you'd think. A tool can deliver enormous value through occasional high-leverage use. A senior engineer who fires up Copilot twice a week to solve the gnarly bit of code they'd otherwise spend hours on produces minimal usage data and massive real value. The usage meter reads "low engagement — consider cutting this seat"; reality reads "this is our most profitable licence".

3. Usage data can't tell you about quality, confidence, fit, or fear. Is the output reliable? Does the user trust it? Are they worried about sharing data with it? Is the tool actively replacing a workflow, or layered awkwardly on top of one? These are the questions that predict renewal, expansion, and whether your AI investment will still look healthy in 18 months. Usage logs have nothing to say on any of them.

What surveys are good at

Asking people — properly — is the only way to get the signals usage data can't reach:

Did it actually help?
Did it make the output better, not just faster?
Do you trust what it gave you?
Would you miss it if it was gone?
What's getting in the way of using it more?

These answers are subjective by definition — no log file will ever produce them. That's not a weakness of surveys, it's the job they do.

Where surveys fail (if you don't design them carefully)

The standard critiques of surveys are real and worth taking seriously:

Response bias. Happy users respond more, or unhappy users respond more — either way you get a skewed picture.
Recall error. Asking "how has AI been this month?" is asking someone to construct a narrative from memory, and memory is compressed and biased.
Survey fatigue. A 20-question engagement survey once a year gets ignored. Or worse, fake-filled.
Gaming. If people think the answers will influence what tools their team gets to keep, they'll respond strategically.

Each of these has a fix. The fixes are what make the difference between useful survey data and useless survey data.

The fixes we use in The GAiGE

Ask in the moment, not in retrospect. The Chrome extension delivers pulses within seconds of someone actually using a tool. "Did that just save you time?" is a vastly different question than "on average, over the last month, how much time would you say this tool has saved you?" The first is a recall problem of seconds. The second is a recall problem of weeks.

Keep it short. 30 seconds max, 1-3 questions, no interstitial. Response rates on short, in-context pulses run 70- 90% in orgs we've measured. Response rates on traditional annual engagement surveys run 40-60% on a good year. Short + timely beats long + ignored.

Aggregate hard. No individual response is ever shown to admins. Your boss sees "the team rated Copilot 4.2/5", never "Sarah rated it 2/5". Gaming goes away when the person answering knows their specific answer can't be used to affect their specific situation.

Correct for response bias in the math. Our 2.5× cap on extrapolation and the response-rate flag on every report mean we can't silently project an enthusiast's answers across the whole team. If response rate is 40%, we tell you — and you should discount the aggregate accordingly.

The combined signal

Neither source is enough on its own. Both together give you something much better than either alone. Three examples:

High usage, low satisfaction. Classic "sticky but not loved" pattern. The tool has wedged itself into a workflow people can't escape, but no one's enthusiastic about it. Renewal risk when something better shows up; training opportunity now.

Low usage, high satisfaction. The high-leverage expert user pattern. The tool isn't touched every day but when it is, it solves expensive problems. Don't cut the seat based on activity alone — you'd lose the value.

High usage, high satisfaction, high time-saved. Your winner. Buy more licences.

In short

The "usage data is objective, surveys are subjective" framing is wrong because it assumes the two are competing. They're not. Usage data tells you what happened. Surveys tell you whether it mattered. Every meaningful AI ROI conversation needs both.

Questions about our methodology? The 2.5× rule post is the longer read on how we turn pulses into defensible numbers.