Why we built The GAiGE, and what most companies get wrong about measuring AI ROI
Three years ago a customer asked us how to measure whether their AI tools were working. We didn't have a good answer, but here's how we ended up with one.
Colin Cardwell
When we started AiGILE, we built a ten-step approach to AI adoption for mid-sized organisations. Step one was pick the right problems to attack. Step two was pick tools that fit those problems. And so on down to step ten, which was some variation of measure the effectiveness of the tools you're using. It was the obvious final step. No one finishes an adoption plan without it.
It was also the step that kept embarrassing us. Customers would nod politely through the first nine steps and then ask, at the end, "OK, so how do we actually measure this?" And we didn't have a good answer. Vendor dashboards didn't exist yet in any meaningful form. Our fallback was "send a survey". Our customers dutifully sent surveys. The response rates were dismal, the results were thin, and nobody learned very much.
That didn't slow AI down. Adoption kept accelerating (wildly, in some places) in a way that was mostly unplanned, mostly uncoordinated, and mostly invisible to the people paying for it, especially at mid-sized organisations without a dedicated AI officer to care about the question. Tools got bought, rolled out, used, ignored, renewed. Nobody really knew what any of it was doing.
We realised, somewhere in the middle of that, that the problem wasn't going to measure itself.
What usage data answers, and the question around itLink to this section
By year two, vendors had started publishing basic usage telemetry. Seats, sessions, prompt counts. That was real progress over having nothing, and we saw CTOs gratefully latch onto it.
Usage data answers one question completely: who opened each tool, and how often. What it can't tell you is how your team is using a tool, what for, how well it's going, whether they'd recommend it to a colleague, or why three people on the same team have quietly stopped. Those answers, and a defensible ROI number, need asking.
The Chrome extension idea, and why it changed thingsLink to this section
We knew asking people was the right direction. The catch was always who answered. An email survey is easy to put off, and the people who answer it lean towards the keen: it arrives at the wrong moment, it costs you ten minutes, and the reward for finishing is that you've filled in a form. We needed something different.
About a year ago we started playing with a different idea: a Chrome extension that asked a few short questions on a light, regular cadence, rather than a quarterly survey. Three or four questions, not twenty. Seconds, not minutes. More people would answer because there was less in the way.
There's a personal angle there too. Before AiGILE, our founder, Colin Cardwell, spent years in game design and gamification. Friction and reward are the two levers game designers think about every day. A badly designed survey is all friction, no reward. A well-designed one is quick, specific, and slightly satisfying to complete. That's the difference between a survey people put off and one they answer.
We started calling them Pulses, because the product needed to feel different: a brief check-in rather than a corporate form. The name set the design brief.
The methodology we didn't know we neededLink to this section
Here's the admission. Our original plan was to build the extension, add up some numbers, and worry about the methodology behind "ROI" later. Good enough, ship it, we'd refine.
The tool we were building The GAiGE with (Claude) pushed back, repeatedly, every time we reached for a shortcut. It kept asking awkward questions. "What happens when one user reports 20 hours saved and the rest report 1? Do you really want to extrapolate that? What about response-rate bias? What are you capping at?"
Every one of those was a question we'd planned to deal with after launch. Every one of those, it turned out, would have embarrassed us if a real customer had asked it first. So we leaned in, talked it through, argued with it, tested it, and ended up with a methodology we can publish: the 2.5× cap on scaling up reported time, the response-rate-aware reporting, the rules on who sees which answers, and the normalised health scores on every gauge. That work, more than any other single thing, is what separates The GAiGE from a slick survey tool. It's also the work we didn't plan to do.
We mention that because we're living in an era where "built with AI" is usually a boast about speed. For us it's also a quality story. The product is more rigorous because we had a collaborator who wouldn't let us off the hook on the hard questions.
Why usLink to this section
A fair question to ask: with AI-assisted development as cheap and fast as it now is, why does AiGILE specifically get to build this, rather than any of the dozens of teams who could spin up a similar product in a weekend?
Three years of working inside mid-sized organisations, helping them adopt AI, gave us a set of intuitions we didn't know we had. We know which metrics executives actually trust in a board pack and which ones they'll ignore. We know how teams actually use these tools (not how the vendor demo says they'll use them). We know which questions put people on the defensive if asked the wrong way, and which ones draw out candid feedback. Call it accumulated pattern recognition.
Game design matters here too. A weekend's worth of vibe-coded MVP can produce a working survey tool. The gap between that and a pulse people answer week after week is where we've spent most of our time.
Where we are, and what we'd like you to doLink to this section
It's early days for The GAiGE. We've made a lot of decisions we believe in, and we've published the methodology so you can push back on any of them. We also know there's plenty we haven't built yet, and plenty of signal we haven't yet worked out how to show well.
We'd like you to join us on that journey. Give it a go free for 30 days with your own tools. No credit card needed. That's long enough to install the extension, roll it out to your team, and see whether the dashboard tells you something you didn't already know. What you pay is one price for the size of your organisation, set in bands. We think that's where this category is going.
Three years on from that first embarrassed moment, when a customer asked how we'd measure it, we've finally got a good answer.
The methodology post: The 2.5× rule. The privacy posture: Anonymous or identifiable. If you've got questions or pushback, talk to us.
Colin Cardwell
Founder of AiGILE and The GAiGE. Colin has spent thirty years designing custom software for tough business problems.
Connect on LinkedIn (opens in a new tab)