Our app is evolving, we would like to make sure the tracking of our events is stable.
It would be damageable, and hard to fix, if we realise after releasing the new version of our app, that some changes in the code made some event not be tracked anymore (refactoring, replacing features, ...).
But that is what tests are for ! A test could perform “a normal walkthrough” of the app, and check that the event trace of this walkthrough is stable.
How does those tests look like ? Does the Amplitude community has best practices for this use case, maybe a playbook, and eventually point to code or a git repository ?
Best answer by jarren.patao
@castor-radieux sure! After redirecting the API Key to send events from the newest version of my project with the TestProject API Key, I do all formal testing of my events to make sure they end up in Amplitude accordingly.
I use a couple of our tools to verify my actual event behavior:
Once I verify that my events are being performed as expected, I then revert the API Key to my Production Project where I ensure my new release is detected so I can track any changes in metrics between versions.
Here are key things which have helped us over the past few years to contain the chaos during wiring multiple events and maintain sanity wrt the data governance side of things before sending events into Amplitude -
We are heavy on cross-portfolio analysis and have multiple apps instrumented. Creating and maintaining an updated data dictionary / taxonomy sheet is the first and easiest step. Ensure that the events + event properties currently being sent in Amplitude are part of your data dictionary first.
The Govern Feature is another really useful one. Using this one for triggering alerts for event validation errors and event/property planning.
For making sure that the existing instrumentation isn't affected by refactoring and adding on new features/events, you can use some of the Amplitude features -
Creating charts for existing events and setting up custom monitor alerts. You can set threshold conditions in there and get alerts if the event counts go down below your expectations after the release.
Creating test scripts using the Dashboard REST API. You can write some code by using these endpoints and calculating your own threshold from the data points returned. Running these for all your events before and after your release might help you in your instrumentation walkthrough.
I'm curious to hear more on this from the other users in this community and how they are tackling this use case.
@Saish Redkar brings up a lot of great points about the tools that are useful for ensuring your instrumentation is in line with your expectations.
I actually create multiple projects where before any new release, I just redirect the API Key used to the test project. You are able to create as many projects as you’d like at the moment so we usually advise testing in a completely separate environment with your own CD/CI tools so you aren’t polluting your Live environment with testing data.
@castor-radieux sure! After redirecting the API Key to send events from the newest version of my project with the TestProject API Key, I do all formal testing of my events to make sure they end up in Amplitude accordingly.
I use a couple of our tools to verify my actual event behavior:
Once I verify that my events are being performed as expected, I then revert the API Key to my Production Project where I ensure my new release is detected so I can track any changes in metrics between versions.
The next step would be to automatise those tests in some ways, to free up some time, remove the risk inherent to manual testing, and test events that you did not consider could be affected.
Thanks for your feedback @castor-radieux! I definitely understand how automating some testing could free up some time. I’ll provide this feedback to our team to see if there is anything we might be able to address on our end!
If you don't have an Amplitude account, you can create an Amplitude Starter account for free and enjoy direct access to the Community via SSO. Create an Amplitude account. You can also create a Guest account below!
If you're a current customer, select the domain you use to sign in with Amplitude.
If you don't have an Amplitude account, you can create an Amplitude Starter account for free and enjoy direct access to the Community via SSO. Create an Amplitude account. Want to sign up as a guest? Create a Community account.
If you're a current customer, select the domain you use to sign in with Amplitude.
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.
