Skip to main content

Hi, we’ve decided to use SplitIO for AB testing in my company, but since we also use Amplitude, I’d really like to have the ability of analysis based on ab tests user segments in Amplitude too

So, we send a getTreatment(“test_name”) call to SplitIO that returns the test and the version that we should serve to the client. We then serve different versions to the client based on that. However we also want to register that in Amplitude so that we can do our full funnel analysis, retention analysis, stickiness analysis based on the cohort users belong to.

My approach would be to register each test and variant as a user property in Segment/Amplitude and this way we could easily do every analysis we want by simply grouping the users by the version that was served to them. Would that be the best practice?

In one tutorial (albeit by Mixpanel) they suggest that we should fire an event when the test is served to the user. I really dislike this approach because IMHO the test version served to the user is a user property just like any other user property such as the version of the app served or if the user was seeing the app in dark_mode or not.

What do you think the best approach here would be? 

  1. call identify(“user_id”, {ab_tests: ,“testA--version1”,”testB--version2”]})
  2. fire an event user_served_testA{ test_variant: “version1”} and user_served_testB{ test_variant: “version2”}
     

Best regards

Luka

Hi @luigi7up ,

 

In general, it would be advised to use the Identify API to update the user with the appropriate user properties. The only caveat with this is that I would wait for about 20 seconds before sending an event after the Identify event. This is to give plenty of time for the identify call to be processed and for the user properties to attach. Then the user properties should propagate to the next event, allowing you to work with your Experiment data. 

 

I hope this helps! Let me know if you have any further questions!

 

Kind Regards,
Denis

HI @Denis Holmes 

I didn’t know that identify() comes with a delay of 20 sec ?!
 

What is Amplitude’s recommendation when it comes to ab testing then? What is the correct setup ?

  1. Firing an ab_test served event when the UI is presented?
  2. Adding a ab_test: “xxx” property to an existing event in the UI
  3. Registering a user property by sending them within an event instead of calling the identify call separately?
  4. Using the identify() call before the experiment events?

Best regards

Luka

HI @luigi7up ,

 

The identify API does not come with a delay of 20 seconds. It is usually relatively instant. I would just give some space to ensure that it goes through in case of any delays. You could ultimately reduce this to even 5 seconds, should be more than enough. To be fair, 20 is a bit excessive so apologies for the confusion. 

 

You could do either way Luka but to keep costs down, I would suggest using the Identify API to add a user property for the experiment. This will attach to the user profile and will appear on the next event that is sent in and any following events. My suggestion would be to use the Identify API and/or, if you already have Experiment events, you could send it with the first experiment event. Other than that, the Identify API could be used and then when the events after the call come in, they will have the user property from the Identify API automatically attached. Does that make sense?

 

Kind Regards,
Denis 

Reply


Terms & ConditionsCookie settings