Phantom Users: Where the 'Bots' in Your Mobile Analytics Come From and How to Tell Them from Fraud

A few days ago I almost sent Google Ads support an angry email about bots. The picture was convincing: we launched an ad campaign in the US, and the same phone model immediately popped up in analytics — OnePlus 8 Pro, a 2020 device. In three days it was 17 “new users” out of 73. The next most common model appeared twice. They behaved like robots: opened the app, started the first level, dropped off. Not a single purchase, not a single finished run.

It looked like classic fraud: a device farm installing the app for a paid install. The email was ready. Then I looked at the dates and noticed the “bot” spikes didn’t line up with the campaign launch. They lined up with the days I uploaded new builds to Google Play.

It wasn’t a farm. Most likely, these were Google’s own test devices. This happens often enough that anyone who looks at mobile analytics should know about it.

What the pre-launch report is and why it ends up in analytics

Google Play has a built-in automated test: the pre-launch report. Every time you upload a build to Play Console, Google installs it on real devices in Firebase Test Lab and runs a crawler. The robot opens the app, taps buttons, walks through screens, and after a few minutes produces a report: crashes, accessibility issues, screenshots. According to the Play Console docs, the report is generated automatically on upload and is usually ready within an hour.

For QA this is a useful free run. But it has a side effect the docs only mention in passing: these are real phones running a real build. The analytics SDK inside the app doesn’t know a robot launched it. It honestly sends first_open, sessions and first-level events. In reports, this looks like new users.

That’s where the wrong conclusions start:

“We have bots.” A batch of identical devices that do nothing looks like fraud.
“Retention dropped.” The robot never comes back on day two. Every build upload adds users with D1 = 0 to the cohort.
“The funnel broke.” The robot goes through the first steps and quits, so early-step conversion drops.
“Users don’t pay.” The same docs state plainly that test devices can’t make purchases.

If you upload builds often — say, five times in two days before a release — the phantoms add up. In my case it was dozens of “users” a week on top of real installs.

Signs: test device or real fraud

Google’s robot and a bot farm look similar, but there are differences you can see in ordinary analytics.

Looks like the pre-launch report:

Spikes in new users match the times you uploaded builds to Play Console, not the ad campaign launch. The robot runs within an hour of upload.
The app version of these “users” is the one you just uploaded, sometimes not yet released to production.
The install source is (direct) or (not set), not your ad campaign.
The country is often unknown ((not set)), sessions are short — a minute or two.
Zero purchases, even though the app has in-app purchases and real users make them.

Looks like real fraud:

The spike matches the launch or a budget increase of a campaign and lasts as long as the campaign runs.
Installs are attributed to the ad network and counted as paid conversions.
Device models and OS versions are distributed unnaturally: lots of identical devices, but on the normal production version.
There are clicks and installs but not even a first open, or opens come at suspiciously even intervals.

The key difference: you don’t pay for Google’s robots. The Play Console docs say Google Ads already excludes traffic from the pre-launch report address ranges. The robot installs the app directly, without an ad click, so there’s nothing to count as a paid install. It pollutes your analytics but doesn’t eat your budget. Fraud eats your budget.

A five-minute check

Before writing to an ad network’s support, I now run four checks.

Source. In Firebase or GA4, open the suspicious segment (e.g., “device model = X”) and look at First user source / medium / campaign. If it’s (direct) / (none), these aren’t ad installs.
Timing. Compare the first_open time of the suspicious users with your build upload times. A match within an hour is almost certainly the pre-launch report.
App version. If the “bots” run the version that had just been uploaded while real users are on the previous one, it’s the robot.
The report itself. Play Console → Test and release → Testing → Pre-launch report → the version → Devices. If the suspicious model is in the report’s device list, case closed.

If all four checks point to ads, that’s a reason to complain. But then you have data, not a hunch, and support treats you very differently.

How to filter out test devices

Don’t turn off the pre-launch report: it catches crashes before your users do, for free. It’s better to teach the app to recognise that a robot launched it.

The Test Lab system flag. Firebase Test Lab devices have a system setting firebase.test.lab set to "true". It’s described in the Firebase docs, with an example check via Settings.System.getString(contentResolver, "firebase.test.lab"). In Unity you can read the same value through AndroidJavaClass.

Then you have two options:

Don’t collect analytics at all. Check the flag at startup and, if it’s Test Lab, disable data collection before the first event (in Firebase that’s setAnalyticsCollectionEnabled(false)). The check has to run as early as possible, or first_open will already be sent.
Tag instead of disabling. Set a user property like traffic_type = test_lab and filter it out in reports with comparisons and audiences. I prefer this option: the robot’s crashes and performance stay visible, while product metrics are calculated without it.

What won’t work. In GA4 you can’t drop traffic by device model with data filters: there are only filters for developer traffic and internal traffic by IP (see the GA4 help). And IP isn’t visible in Firebase reports. So tagging from inside the app is the most reliable way.

For other ad networks. Google Ads excludes pre-launch traffic on its own. Other networks may count it their own way. You can get the current Test Lab IP ranges with gcloud beta firebase test ip-blocks list and pass them to the network’s anti-fraud settings.

Takeaways for QA

Analytics is a test environment too. Any automated run on real devices sends real events: the pre-launch report, your device farms, CI autotests. If you don’t tag them, product decisions are made on dirty data.
Check that the tagging works. The Test Lab flag is a test case of its own: upload a build, wait for the pre-launch report, and make sure those devices are either absent from analytics or tagged.
Separate “bots” from “tests” before you complain. A fraud complaint without checking the source is wasted time and lost credibility with support.
Upload builds less often if you’re watching early metrics. Five uploads in two days means five batches of phantom users dragging down the new cohort’s D1.

Checklist: phantom users in analytics

A suspicious spike of new users on one model — compare with build upload dates.
Check First user source for the segment: ads or (direct).
Check the app version for the segment: freshly uploaded or production.
Open Pre-launch report → Devices and look for the suspicious model.
Make sure no money was charged for these installs in the ad account.
Add a firebase.test.lab check to the app as early as possible at startup.
Decide: disable analytics on Test Lab or tag users with a property.
Add a test case: after a build upload, pre-launch devices don’t show up in product reports.
Tag your own autotests and device farms the same way.
For third-party ad networks, pass Test Lab IP ranges to their anti-fraud.

The nastiest thing about phantom users isn’t that they exist, it’s that they’re believable. Identical devices, zero conversion, short sessions — it all adds up to a fraud story. Five minutes of checking the source and timing saves you both a support email and a wrong product decision.