Question

Can Amplitude do network experimentation?

  • 12 April 2024
  • 3 replies
  • 212 views

In my product, users can send each other messages.  If we want to experiment on how messaging works, we can’t run a typical A/B experiment because we need both the sender & recipient to be in the same variant.  One standard way of addressing this is running experiments in different geographic locations, with the assumption that each location has a mostly-closed network.  This hasn’t worked for us, unfortunately, as our separate locations aren’t very similar in the first place so we can’t compare them.  Some companies have the ability to expose experiments based on network clustering.  Is that a feature that Amplitude has?

Side note: I have bet on with my engineer that the answer is no. =D  So you won’t disappoint me if that’s the case!

The best alternative I can think of is to use BigQuery to manually separate users into clusters, normalize on relevant attributes, try to pair similar clusters to assign to different variants, and upload those assignments as a user property, but I have no way of ensuring similar levels of exposure.  It seems like there’s just too many factors where things can go wrong. =/


3 replies

Badge
Hi Alexandra,

Thank you for contacting Amplitude Support. Happy to help!
Jake here.

The short answer to your question is no, we don’t support network clustering for targeted audiences.

One way is that you can choose to target certain audiences with a particular variant if desired
(it’s called “stratified sampling” and it’s an advanced feature on our platform) - link

Another alternative that I would suggest is bucketing by a non-user ID by utilizing our group bucketing feature.
If this customer has a “pair ID” or “conversation ID” that they can bucket by, then the experiment will work - without the need for fancy math 🙂
For this to work, you would need an account add-on feature to utilize. I can loop in AE regarding the issue if you are interested!

I hope this helps, and please let me know if you have further questions.

Best regards,
Jake


P.S. Checkout upcoming events and user meetups on our events page.

Thanks for this!  Question about bucketing by pair id.   Is it correct to say that since a user can message several different people (but no group messages) that a user will end up with multiple pairs to bucket?  So e.g. their conversation with Person A could be in control, but their conversation with Person B could be in treatment?

Badge

@Alexandra Ressler 

 

Yes, you are correct, I am afraid that is the limitation that we have for peers to peers message testing.

🙏

Reply