Why AI vendor impact surveys can't replace independent measurement
OpenAI is shipping Impact Surveys in ChatGPT, and other vendors may follow. What a vendor's own survey answers well, and what a view across all your tools adds.
Colin Cardwell
OpenAI is shipping Impact Surveys inside ChatGPT: surveys that ask the members of a workspace about the outcomes they see, with the answers shown in the workspace's analytics (Source: OpenAI Help Center, Managing impact surveys in workspace analytics). Other vendors may follow with their own.
That's good news. For a year, the measurement conversation has been stuck on activity data. "We have 412 daily active users." "You've sent 18,000 prompts." Activity tells you people are busy, not what the work is worth. Vendors moving from "are people using us?" to "are people getting value from us?" is a healthy shift.
It's also where the problem starts.
When the company selling you the AI tool is also the company grading whether it works, you have a problem the finance industry recognised decades ago, and answered with independent audit. The same answer works for AI Measurement: the questions, the scale and the comparison with the next tool need to sit with someone who doesn't sell the tool.
This piece is about why that matters, what "independent" means in practice, and how to read the vendor-side surveys arriving on your desk.
What vendor impact surveys doLink to this section
OpenAI's Impact Surveys use question sets OpenAI writes: an admin can send one, but can't change its questions (OpenAI Help Center). The answers give a picture of one tool, in one organisation, in the vendor's own terms.
That is useful. It moves the conversation past "how many seats do we have?" towards "what is the team actually getting?".
If other vendors follow, each AI tool you use may come with its own impact dashboard, and each may:
- Ask different questions
- Use different scales
- Sample at different cadences
- Define ROI differently
- Aggregate at different levels (per user, per team, per prompt)
- Publish its methodology, or not
If you use five AI tools, you could soon have five impact dashboards, none of them comparable with the others.
Why vendor-led measurement leans one wayLink to this section
Vendor-side measurement has real value when it's read for what it is: one input. The issue is treating it as a neutral score.
Three sources of bias matter more than the rest.
Methodology biasLink to this section
The vendor designs the questions. The vendor decides the scale. The vendor decides what counts as a "positive outcome". A vendor whose hero metric is "time saved" will ask about time saved. A vendor whose hero metric is "user confidence" will ask about confidence. None of those is wrong on its own. But they are not the same question, and the answers won't roll up to a comparable number.
A concrete example. If a vendor asks "how much time did this save you?" on a 0-60 minute scale, the average answer will tend to skew higher than if the same question were asked on a 0-180 minute scale. Buyers rarely see the scale. They see the average, and assume it compares.
Sample biasLink to this section
A vendor only sees its own users. If 30% of your team has drifted away from a tool, that 30% doesn't get to answer the impact survey. The signal you see is from the people who stayed. Those who stayed tend to be the more satisfied, and the people the tool failed to serve are silent.
If only the most engaged users answer, a vendor's time-saved figure describes them, not everyone you pay for. The vendor is sampling the wrong people for the question you're asking.
Narrative biasLink to this section
A vendor has every reason to lead with its best figures, because low scores make the renewal harder. The impact figures it publishes are its own reading of the data, and the less flattering evidence is less likely to make the slide.
This isn't unique to AI vendors. It's true of every category where vendors measure their own product. In AI it matters more, because the contracts are newer and buying committees have less experience comparing them.
The comparability problemLink to this section
The deeper issue is structural. Even setting bias aside, vendor surveys can't be compared across tools.
A common buyer scenario. You lead AI strategy at a 300-person organisation. Marketing, engineering, finance and legal each use a different AI tool, from four vendors, with four impact surveys.
Your CFO walks in and asks the obvious question. "Which of these is paying off?"
You can't answer that from the four vendor surveys. How four vendor surveys might differ, as an illustration:
| Tool | Who is asked | Question scale | ROI definition |
|---|---|---|---|
| Tool A | Active users, after a workflow | Time saved (0-60 minutes) | (Time × rate) ÷ seat cost |
| Tool B | Each developer, weekly | Confidence (1-5) | "Acceptance rate" |
| Tool C | Each user, monthly | Satisfaction (-100 to +100) | Self-reported minutes per task |
| Tool D | Each team, quarterly | Several factors, not published | Not published |
Four vendors and four methods, so the numbers don't add up to one answer.
And the buying decision is rarely "which one of these is the worst?". It's "given a finite budget for next year, where do we double down, where do we hold, and where do we cut?". That decision needs comparable measurement across every AI tool you use. Vendor surveys, by design, can't provide it.
What "independent" actually meansLink to this section
Independence has four practical meanings, and any AI Measurement vendor (including us) should be willing to answer all four directly.
1. Who designs the questionsLink to this section
Independent measurement uses the same question, on the same scale, for every tool. The question is designed by an organisation whose commercial interest is in the measurement being trustworthy, not in any specific tool looking good. The methodology is published. The scale is fixed. The wording is open to review.
2. Who owns the dataLink to this section
If the AI vendor is the one collecting the answers, the AI vendor is the one deciding what to publish, what to keep, what to share with its sales team, and how the totals look on a slide. Independent measurement holds the data outside the AI vendor's environment. The organisation owns the answers, and the AI vendor sees none of them.
3. Who publishes the methodologyLink to this section
You should be able to read exactly how every score in the report is calculated: what counts as "active", what counts as "time saved", how unused seats are worked out, and how satisfaction is normalised across tools. If you can't read the methodology, the score is opinion, not measurement.
4. Who is rewarded by which outcomeLink to this section
This is the load-bearing one. A vendor selling you Tool X benefits if Tool X looks good. A vendor selling you measurement of every tool benefits if the measurement is trustworthy enough that you keep using it. Those are different incentives, and they pull in different directions.
A CFO framework: four questions to ask any AI impact dataLink to this section
Here is a quick check any CFO can run on any AI impact data crossing their desk.
1. Who chose the questions?
If the same company that sells the tool also designed the survey, treat the result as one input, not the answer. There may still be useful signal. You just can't rely on it as a like-for-like comparison.
2. Who chose the scale?
A 5-point scale and a 100-point NPS produce different averages from the same underlying sentiment. If the vendor can't tell you why it chose its scale, the number doesn't travel.
3. Who is in the sample?
A vendor's impact survey usually polls the active users of the vendor's tool. The leavers, the people who never adopted it and the quietly disengaged are missing from the denominator. The question to ask is "what percentage of seated users are in your sample, and what do you know about the rest?".
4. Who saw the result before you did?
If the vendor saw the result first and decided what to show you, you are looking at a curated summary. If the answers come to you first, and the vendor isn't the one deciding what you see, you are looking at measurement.
Run those four questions over the next vendor impact dashboard that arrives on your desk. You'll usually find one or two of them hard to answer.
A useful inputLink to this section
Vendor impact surveys are a useful input to AI Measurement. OpenAI shipping Impact Surveys is a positive development, and others adding their own would be too. (See also our piece on why asking your team adds what usage data can't.)
The job of an independent, cross-vendor measurement layer is to:
- Use one methodology across every AI tool you use
- Sample the whole seated population, not just the most engaged
- Publish the question wording and the scale
- Hold the answers outside any AI vendor's environment
- Present the result like for like, so the CFO's renewal decision rests on evidence
That's what The GAiGE is built to do. It is independent of every AI vendor, and it measures ChatGPT, Copilot, Claude, Gemini and the rest the same way, so the question "is this paying off?" has a real answer.
Where to go nextLink to this section
If this describes a problem you already have (impact numbers from five vendors that don't add up to one decision), the sample AI Impact Report (no sign-up needed) shows one methodology across a sample organisation's tools.
Whichever vendor surveys arrive, the four questions above are the ones to keep asking, and "trust us" isn't an answer to any of them.
Colin Cardwell
Founder of AiGILE and The GAiGE. Colin has spent thirty years designing custom software for tough business problems.
Connect on LinkedIn (opens in a new tab)