Google Just Made Incrementality Testing Free and Open Source. The Method Needs More Geography Than One Rooftop Has.

Google shipped something on September 10 that almost no dealer will hear about from a vendor, because it makes a specific kind of vendor report much harder to defend.

Meridian GeoX left beta. In Google's own announcement, the line is short: "Now, GeoX is generally available globally in Meridian." Google describes the thing itself as "our global open-source library for running causal geo-experiments across any advertising platform," and says advertisers can use it to "get started with running independent experiments, or incorporating incrementality results into your MMM to help boost accuracy."

Read the phrase "across any advertising platform" twice. This is Google publishing free software whose purpose is to test whether advertising, including Google's own advertising, caused anything. Brooke Osmundson covered the launch at Search Engine Journal on the same day, and notes Google first previewed GeoX in May. The code sits on GitHub under an Apache-2.0 license, and installs with a single pip command.

What a geo experiment actually does

Strip the language off and the method is old, simple, and brutal.

You split your markets into two groups. One group keeps running the advertising. The other group gets it turned off, or turned up. Then you wait, and you compare what the two groups sold. The difference is the ad's real contribution, because everything else that happened to both groups happened to both groups.

That is the whole idea. It is the only measurement method on the board that can tell you the sale would not have happened anyway, and every attribution model your vendors use is a substitute for it. Last click, first click, data-driven, position-based: all of them start by assuming the sale belongs to somebody, then argue about who. A holdout test starts by asking whether the sale was ever in question.

Google Research's paper on GeoX, by Steven Ye, Fabian Sinn, Yunxiao Li and Yang Jiao, is blunt about why nobody runs these. Two barriers held adoption back: "the high costs required to overcome large variance across geographical regions, and the unreliability of conventional analysis methods under real-world autocorrelation and non-stationary trends."

In plain terms, markets are noisy and different from each other, so you have to give up a lot of revenue in the dark markets before the signal beats the noise, and the older math often gave you an answer you could not trust anyway. GeoX packages stratified sampling, Time-Based Regression, Synthetic Control, Synthetic Difference-in-Differences and a placebo inference engine to attack both problems at once.

The 2015 study every dealer should know about

The most important advertising experiment ever run on this question was run by eBay, and it should be taught in every dealer twenty group.

Tom Blake, Chris Nosko and Steven Tadelis shut off eBay's paid search advertising on brand keywords across Yahoo and Microsoft, and kept Google running as a control. They published the result in Econometrica in 2015.

Almost all of the click traffic and the sales that had been credited to those brand ads showed up in organic search instead. The company had been paying for people who were already coming. On non-brand keywords the finding was more layered: new and infrequent shoppers responded to ads, but the heavy users who were going to buy regardless consumed most of the budget, and the average return came out negative.

Now hold that next to a franchised dealer's account. You are buying your own store name. You are buying your own OEM's model names in your own market. Your monthly report shows those campaigns converting at a cost per lead that makes them look like the best money you spend, and they convert well for exactly the reason eBay's did. The shopper already knew the name. They typed it.

Nobody at a dealership has ever been able to settle that argument, because settling it required an experiment nobody had the tooling or the nerve to run. Google just put the tooling on GitHub.

Before anyone quotes the launch numbers at you

The performance claims attached to this launch deserve the same skepticism the tool is designed to enforce.

Google says the native multicell execution delivers "budget savings of more than 31% for large advertisers," plus better than 94% faster design generation. PPC Land, covering the beta exit, put the provenance on record: "The figure is a vendor number. No sample size, test period, comparison set or definition of 'large advertiser' was given during the stream, and the comparison is against unnamed open-source alternatives."

PPC Land's conclusion is the right one to carry: "The 31% and 94% figures will hold or fail on independent replication that has not yet happened." The method is sound and worth your attention. The marketing numbers on top of it are unverified, and a tool built to demand causal proof arriving with unreplicated vendor statistics is worth a small smile.

What this means for your dealership

The unit of measurement is a market, and that decides who can play. A geo experiment needs enough separate geographies to split into a treated group and a control group. A single-rooftop store selling into one metro has one geography. You cannot hold out half of yourself. This is a structural limit of the method rather than a licensing restriction, and no amount of budget fixes it.

Dealer groups are the ones holding a live option here. If you run twelve, twenty or forty rooftops across separate DMAs, you have the variation this method needs, and you have been making tier-two and tier-three budget decisions without ever testing them. That is the single largest unexamined line item in most group marketing budgets.

The data requirement is specific and most stores do not have it yet. Google's GeoX user guide states: "All experiments require gathering historical geo-level KPI daily time series data." For heavy-up or go-dark tests you also need historical daily geo-level spend. Daily, by market, going back. If your group's reporting lives in monthly PDFs from four vendors, the first project is plumbing rather than statistics.

Your attribution report was never answering this question. A dealer report showing branded search at a low cost per lead is describing which channel got credit. It carries no information about whether the lead existed without the ad. We have already covered how easily that credit lands in the wrong bucket inside GA4. Those are different questions, and after eBay, the honest posture on branded paid search is that it is unproven at your store rather than proven good.

Watch what Google did not publish. The GeoX overview and user guide do not state a minimum number of geos, a minimum history length, or a minimum detectable effect you should expect. The paper leans hard on lowering the minimum detectable effect without putting a number on the floor. Anyone who sells you a dealer-flavored version of this should be made to state their assumptions in writing.

There is a free metric in the same announcement worth five minutes. Google also introduced Data Strength Uplift in Google Ads, which "calculates the additional conversions recovered by your first-party data setup to help quantify impact." That is a much smaller idea than GeoX and it requires nothing of you except looking at it.

What to do about it

1. If you are a single rooftop, do not buy a geo-experiment product. The method does not fit your footprint. Spend the attention on a clean before-and-after test instead, and accept its weaker evidence honestly.
2. If you are a group, inventory your data first. Can you produce daily sales and daily ad spend, split by rooftop and market, for the last two years? That answer determines whether any of this is available to you this year.
3. Pick the campaign you would defend hardest. Branded paid search is the correct first candidate, because it is the one everyone is certain about and the one the literature has already embarrassed once.
4. Have your agency put the design on paper before any money moves. Which markets go dark, for how long, what result would change the budget. A test with no pre-committed decision rule is theater.
5. Ask every vendor whose report claims credit whether they have ever run a holdout. The answers will sort your vendor list faster than any deck review, and they guard against firing the wrong vendor for the wrong reason.
6. Send GeoX to whoever owns analytics, not to whoever owns marketing. It is a Python library that wants GPU support for heavy simulation. The person who can actually evaluate it does not sit in the marketing meeting.

The part worth watching

The interesting thing about a company open-sourcing the instrument that grades its own product is that the instrument does not care who built it. GeoX runs against any platform, so the first groups that adopt it will end up with a defensible number for Google, Meta, streaming, direct mail and radio on the same axis.

Dealers have argued about that comparison for thirty years using numbers that were never comparable. The tool to settle it is now free, and the constraint is no longer cost. It is whether your organization can produce clean daily data by market, and whether anyone in the building wants to know the answer badly enough to turn something off.

If you want help working out whether your store or your group has the data to test any of this, and what your branded search is actually worth, book a time with us and we will go through it with you.