Your Dealership's 'AI Visibility Score' Is Mostly Noise. New Research Shows Why a Single Reading Can't Be Trusted.

Somewhere in your inbox right now is a pitch for an "AI visibility dashboard." It promises to show you exactly where your dealership ranks when a shopper asks ChatGPT, Gemini, or Perplexity "what's the best Ford dealer near me." Maybe you already bought one. Maybe a vendor emailed you a screenshot last week — "You're #3 in ChatGPT!" — and you either celebrated or panicked depending on the number.

A new research paper says that number is mostly noise.

Not wrong, exactly. Just far less certain than the confident little digit on the dashboard makes it look. And for dealers being sold on precise "AI rank" scores — and being asked to make hiring, firing, and budget decisions off them — that distinction is the whole ballgame.

The development: a single AI-visibility reading is closer to a coin flip than a fact

The paper, titled "Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement," comes from Ronald Sielinski, co-founder and chief data scientist at IQRush, a company that measures AI search visibility. (Worth noting up front: he sells software in this exact space, which makes it notable that his own research is a warning label on how these numbers get used.) It was also written up this week by Matt G. Southern at Search Engine Journal.

Here's the setup. The researchers took three ordinary shopping topics — bird feeders, running gear, and adult multivitamins — generated 200 real-world questions for each, and asked them across SearchGPT, Google Gemini, and Perplexity. Then they did the thing almost nobody selling you a dashboard does: they asked the same questions over and over, daily for nine days and in one case every ten minutes for four hours, and watched how much the answers moved.

They moved a lot.

The clearest example from the paper: on SearchGPT, for running gear, Tom's Guide showed up in about 9.5% of citations and Runner's World in about 6.0%. On a dashboard, that's a clean story — Tom's Guide is winning by 3.5 points, go copy what they're doing. But once you account for how much those numbers bounce between runs, the ranges overlap completely. In the paper's words, the two are statistically indistinguishable. The 3.5-point "lead" is inside the margin of error. It might be real. It might be the model having a different afternoon.

And that wasn't a fluke pairing. The paper found that overlapping confidence intervals "are the norm rather than the exception for domains that appear to differ in citation share by less than 5–7 percentage points." Its blunt conclusion about the single-number dashboards: they "provide a misleadingly precise picture of domain performance."

How much sampling would it actually take to trust the number?

This is the part that should stick with you. The researchers worked out how many times you'd have to ask before a ranking settled down enough to believe. Per Search Engine Journal's write-up, across 30 platform-topic tests the answer ranged from 33 to 94 separate queries — and in three of those tests, the ranking never stabilized at all, even after 125 questions. All three failures were on SearchGPT, the ChatGPT-powered engine, which also flat-out returned no citation on 17% of the questions in one topic.

Put plainly: to know your "AI rank" for one question on one engine with any confidence, you might need to ask it fifty times. The tool that pinged an engine once and printed you a number did not do that. It handed you one frame of a movie and called it the plot.

Why this should feel familiar

If you've been in this business a while, you've watched an industry fall in love with a precise-looking number before.

Remember when "Google PageRank" was a green bar from 0 to 10, and vendors sold dealers on nudging it from a 3 to a 4? Or when every SEO report led with a single "keyword ranking" — you're #1 for "Toyota dealer Springfield!" — as if search results were the same for everyone, every time, forever? They never were. Rankings were already personalized, localized, and shuffled constantly. The single number was always a simplification we agreed to pretend was solid because it fit in a slide.

AI search took that same comforting oversimplification and cranked the underlying randomness way up. Traditional Google, for all its updates, gives roughly the same page the same result minute to minute. A generative engine is probabilistic by design — it can answer the identical question two different ways ten minutes apart. So the "your rank is X" number, already shaky in the SEO era, is now built on sand that actively shifts while you're standing on it.

We wrote last week about how AI citations are volatile over time — the spot you win in March can be gone by July. This new research adds a second, quieter problem underneath that one: even at a single moment, a single measurement is unreliable. It's not just that the picture changes between months. It's that any one snapshot is blurry to begin with.

What this means for your dealership

Here's how this actually lands on your rooftop, because it's not abstract.

That vendor screenshot proves almost nothing — in either direction. A one-run "you're #3 in ChatGPT" is not evidence your GEO investment is working, and a one-run "you dropped to #9" is not evidence it's failing. Both are single pulls from a noisy machine. Don't sign a contract on the good screenshot. Don't fire your agency over the bad one.

Small movements are not results. If a dashboard shows your AI visibility "up 4 points" this month, that is almost certainly inside the noise floor — the same 5-to-7-point range the researchers found swamps most differences. Celebrating it is like celebrating a good coin-flip streak. The only movements worth reacting to are large and repeated.

"Precision" is a sales tactic, and now you can test for it. Any tool or vendor that hands you an exact AI rank — "#3," "72/100 visibility score" — from what is clearly a single check is overselling certainty they don't have. You're allowed to ask the question the research implies: how many times did you sample this, and what's the range? A tool honest enough to sometimes say "not enough data to tell" is worth more than one that prints a confident number every time.

The engines are not equal, and ChatGPT is the wildest. SearchGPT was the noisiest and most likely to cite nothing at all. If your shoppers lean on ChatGPT — and many now do — expect its readings to swing hardest, and weight them least when they're based on a single look.

What to do about it

You don't need a statistics degree. You need to stop treating a single AI number as a scoreboard and start treating your AI presence as a trend you sample. Concretely:

• Sample, don't snapshot. When you check whether AI recommends your store, ask the same real shopper questions — "best [your brand] dealer near [your city]," "is [your dealership] a good place to buy" — several times across a few days, not once. You're looking for a pattern: are you consistently present, or only occasionally? Presence beats rank.
• Watch for absence, not position. The most useful signal isn't whether you're #3 or #6 today. It's whether your name shows up at all, most of the time. Being reliably in the answer is the win. An exact rank is mostly noise; consistent inclusion is real.
• Grade vendors on honesty about uncertainty. Ask any AI-visibility tool or partner how often they sample and whether they report ranges. "We ran it once" is a red flag. Ranges and repeated sampling are the tell of someone who actually understands the medium.
• Keep building the foundation, because that's what isn't noisy. The one thing that genuinely moves your odds across every engine is being the kind of business AI wants to cite: a fast, machine-readable site, accurate structured inventory, real answers to real buying questions, and a strong Google Business Profile and review presence. That work pays off no matter which way the daily dice land.
• Ignore anyone selling a guaranteed AI rank. Same rule as the "get cited in AI" pitch: if the number itself is this unstable, nobody can sell you a fixed position in it. They can only sell you a screenshot.

The bottom line

The comfortable version of AI search is a clean dashboard with your dealership sitting at a confident number. The research says that number is a single frame from a shaky camera — useful as a rough hint, dangerous as a decision. The engines are probabilistic, the readings bounce, and the "precision" on the screen is mostly manufactured.

That's not a reason to ignore AI search. It's a reason to measure it like a grown-up: sample it, look for consistent presence over exact rank, and put your real money into the durable foundation instead of chasing a number that changes when the model blinks. The dealers who get this will stop overreacting to screenshots and start winning the only thing that lasts — being genuinely worth citing.

Want to know whether AI is consistently recommending your store, not just whether it did once on a Tuesday? Book a no-pressure walkthrough and we'll show you how your dealership actually reads to an AI right now — and where the real, durable gains are.