Hi @Azu_Saucedo. I’ve seen this type of question arise in many different businesses over the last decade or so, and the outcome has always been the same…namely that spending too much time on analysing differences between digital tracking and your back end is wasteful, and yields little benefit to the business. The time is better spent elsewhere, “proving” cause of a variance is a bit like finding a needle in a haystack.
There is not really such a thing as an “expected standard” of variance between digital analytics tools (whether Amplitude or another) and the “source of truth”…it varies significantly from industry-to-industry and business-to-business. For example, I’ve seen businesses have a 30% variance while others only have 5-10%. As you rightly point out, this can be due to ad blockers, but also things such as “private browsing modes” in browsers, and not forgetting cookie consent modules (no consent = no tracking).
I’ve always advised a focus on comfort that the tracking setup is done correctly, i.e. thorough QA that your events are firing when they should be. That’s no mean feat of course, but for me that’s an absolute key to getting trust in your digital data, not an over focus on comparison to back-end systems. Digital data is known for its nuances in a complex world of native apps and web apps, and should absolutely not be used for “official” counts. For example, if the need is to tell the business how much money has been taken, the back end data should be the source, not Amplitude (or similar)…but if the need is to observe patterns and trends in digital measures and to uncover insights, then yes, Amplitude absolutely fits here.
Hope that’s of some use!
@dangrainger I’ve been working with @Azu_Saucedo on this investigation. Thanks for confirming this based on your experience helping other companies answer this question. Your findings align with our research thus far, and it’s helpful to hear from someone who has spent considerable time digging into this with other businesses.
Agree with Dan here 
.
As Dan mentioned, the problem of “siloed source/s of truth” arises most commonly depending on the type of data the team is looking at - financial vs product usage.
Within the event analytics space, ensuring attention to your event triggers and correct instrumentation is much valuable than comparing absolute counts between two different reporting systems. It could be done as an one-off exercise , but doing that at a regular cadence would only make your team lose trust in your data systems over a period of time.
I would always love to avoid the problem of having to compare between legacy source/s of truth and numbers from tools like Amplitude. But the opportunity to implement a company’s event tracking setup from scratch with Tool XYZ is hard to come by in most cases, unless for very early stage companies. So existing source/s of truth will always be a constant companion that we will have to deal with :)
Thank you for your responses @Saish Redkar and @dangrainger . I definitely agree with you! I have two follow up questions:
- How to explain variances between a third tracking system such as GA where GA and Amplitude don’t match either?
- When I have been asked in the past about the variances with source of truth, I have provided the same response as Dan mentions but it doesn’t seem to be enough. Have you found a way to communicate this to data driven but non technical stakeholders?
Thank you again for sharing your experiences!
