What's the best way to measure AI? Five options compared
Annual surveys, vendor dashboards, SaaS spend management tools, consultant audits and short pulses on a regular cadence: five approaches, what each is good for, and how to pick the right combination.
Colin Cardwell
AI costs have grown faster than the means of measuring them. Here are the five approaches organisations use today: what each is good for, where each falls short, and how to pick the right combination for your team.
Every leadership team we talk to is asking some version of the same question. We've bought seats in five different AI tools across the organisation. Some people seem to love them. The bills are real. Is any of this actually working?
It's a fair question. It's also surprisingly hard to answer, because AI tools sit in an awkward spot: they don't fit neatly into the measurement frames we already have. They're not a hire, where you measure outputs. They're not a piece of software with a clear process metric, like Salesforce or HubSpot. They're a general-purpose lever, used differently by every person who touches them.
So how do you actually measure it? Five approaches are in common use today. Each has a sweet spot. None is a universal answer. The trick is picking the right one (or, more often, the right combination) for the question your team is genuinely trying to answer.
1. Annual engagement and pulse surveysLink to this section
Examples: Culture Amp, Lattice, 15Five, Officevibe, the Gallup Q12.
These are the broad employee-experience tools that many mid-sized organisations already run. Quarterly or annually, you ask your team a battery of questions about how they're feeling, and you slot in two or three AI-specific questions next to the engagement and management items.
Sweet spot: tracking sentiment at a high level over time. Useful if your leadership team needs a single "are people happy with the AI rollout?" number for a board report.
Where it falls short: who answers. Annual surveys rarely reach everyone, and the people who answer lean towards the enthusiasts and the complainers. You're also asking people to remember three months back to a tool they used twice, so the answers reflect impressions, not behaviour. And because AI questions live alongside thirty others, they get a few seconds of attention, not real reflection.
Use this if you're already running an engagement programme and you want a coarse trend line. Don't use it to make tool-by-tool budget decisions.
2. Vendor usage dashboardsLink to this section
Examples: the admin panel that ships with ChatGPT Enterprise, Microsoft Copilot's usage report, Claude for Work's analytics, GitHub Copilot's seat dashboard.
Every major AI vendor ships an admin dashboard for the seats you've bought. They tell you who's logged in, who's active, how many messages or completions or suggestions per user per week, sometimes with a feature breakdown.
Sweet spot: answering the activation question. "Did Sarah, who I gave a Copilot seat to, actually use it?" The data is objective, comes with your subscription, and is granular. Vendor dashboards answer the activation question completely; whether the tool is paying off is the question around it.
Where it falls short: use and value are different things. Someone can use a tool a hundred times a week and find it unhelpful. Someone else can use it twice and save themselves a day. The dashboard counts the activity; whether it paid off needs asking.
Use this to spot dormant seats and to flag patterns worth a question (a tool with strong adoption in Engineering but none in Marketing: is that right?), alongside a measure of whether each tool is paying off.
3. SaaS spend management platformsLink to this section
Examples: Zylo, Productiv, Vendr, Spendflo. Some also touch the AI-cost-management space (Zylo's AI Discovery, Productiv's AI tooling visibility).
These platforms started life as SaaS cost-rationalisation tools: surface every subscription, find the duplicates, negotiate them down at renewal. Most have a story about AI now: which tools are spreading across your organisation, how the costs are trending, whether seats are sitting idle.
Sweet spot: cost visibility. If you've lost track of how many AI subscriptions are scattered across company cards, departmental budgets and shadow procurement, these tools find them and consolidate them.
Where it falls short: cost-management platforms show what you pay, and for what. Whether it's paying off is the next question, and not the one they're built for. Even the dormant-seat warnings only get you to "cancel the seats nobody uses". They don't tell you which tools are quietly working but under-adopted, or which ones people use a lot but secretly resent.
Use this if your AI costs are out of control and you need a consolidation exercise.
4. Consultant-led audit (one-off engagement)Link to this section
Examples: a six-week engagement with a Big Four consultancy, an AI-specialist boutique, or your in-house transformation team running a structured audit with interviews, focus groups, and a final report.
Done well, this is the most credible single snapshot you can get. A skilled consultant will interview a cross-section of your team, observe actual usage, build a quantitative case study or two, and hand you back a board-ready report with concrete recommendations.
Sweet spot: depth. You get qualitative texture (the "why" behind the numbers) plus a strategic recommendation written for your specific context. Hard to beat for a board presentation or a budget decision that justifies the engagement's cost.
Where it falls short: it's a snapshot, not a sensor. Six weeks after the report is delivered, your team has bought two new AI tools, dropped a third, and the usage patterns have shifted. A few months on, the report is a historical document. The other problem is cost: a serious AI audit from a large consultancy is a significant one-off cost. You'll do it once and then go without measurement until you can justify another round.
Use this for a one-off strategic reset: the moment you're presenting AI ROI to the board for the first time, or making a multi-year platform decision. Don't expect it to be your ongoing instrument.
5. Short pulses on a regular cadenceLink to this section
Example: The GAiGE. (We're aware this is our blog. Stay with us: the trade-offs get the same treatment as everyone else's.)
The approach we built. The browser extension shows each person a thirty-second pulse when one comes due: three or four short questions, three times a week per tool by default, on a cadence admins control. Every pulse can also be answered in the AI Adoption Portal. The answers become a dashboard of hours saved, satisfaction, adoption and where people need AI learning, and, with Pro, ROI by tool.
Sweet spot: per-tool numbers you can defend, kept current. The methodology is published: a 2.5× cap on scaling up reported time, your own blended hourly rate, and pulses that are anonymous by default, on which every figure needs answers from at least five people. Every number shows how it was worked out, and reported hours sit beside estimated hours, so you can choose how strong a claim to make.
Where it falls short: it works best with the browser extension (Chrome or Edge), which is highly recommended; anyone without it answers in their AI Adoption Portal. Larger organisations can install it for everyone through Google Workspace or their device management, and each person still allows it on all sites themselves.
Use this when you want a defensible per-tool ROI number on an ongoing basis: for the CFO who keeps asking, for renewal decisions, for spotting which tools your team secretly resents before they drop out of use. Pulses ask about every tool you register, in the browser or on the desktop, because they arrive on a schedule.
What the right answer usually looks likeLink to this section
The combination that usually makes sense is two or three of these, layered:
- Vendor dashboards for activation and seat-utilisation hygiene (included with your subscriptions, and already there).
- Short pulses on a regular cadence for the ongoing per-tool ROI signal that drives renewal and rollout decisions.
- A consultant audit every 18 to 24 months, for strategic resets and big platform decisions.
The annual employee survey can include two or three AI questions for the sentiment trend line, but it's not where you'll find the answers that matter. SaaS spend management is worth it only if your subscription sprawl is out of control.
Where to startLink to this section
If you're just beginning, the order we'd suggest is the reverse of how most organisations actually start. Most start with the annual survey and the vendor dashboards because they're already paid for. Neither, on its own, tells you whether a tool is paying off.
Start with the question your CFO will ask in six months, work backwards from that, and pick the measurement that answers it. For most teams that's a continuous per-tool ROI signal. If you'd like to see what that looks like in practice, we've published the full methodology behind The GAiGE in our 2.5× rule post, and a longer piece on why asking your team adds what usage data can't.
Colin Cardwell
Founder of AiGILE and The GAiGE. Colin has spent thirty years designing custom software for tough business problems.
Connect on LinkedIn (opens in a new tab)