Firing fetch_experiment and exposure together (when fetch_experiment return variant)

kannan.ganesan1989

Hello,

We are using Ruby SDK initialize_remote to use fetch_experiment and exposure events.

My question is, is it best practise to fire exposure event when ever we fetch_experiment returns experiment?

Problem we think. Firing exposure and fetch separately will introduce unpredictability as customer might drop without seeing the experiment itself.

Your thoughts on this?

Find more posts tagged with

Event

Experiment

Accepted answers

tsegalla

Hi kannan.ganesan1989 and marcel ueno, great question.

Regardless of when you fire each, first make sure that you’re always calling .fetch() first before calling .variant() / firing exposure. This order is important to ensure we are assigning the user to a test and a variant first before considering them exposed so that your data is as accurate as possible.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other. In this case, there’s no difference between the two.
If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow. For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it. For that reason, this user should not be considered to be part of the experiment results.
- Amplitude uses the exposure as the denominator in every experiment metric you set up. This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results. Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

All comments

marcel ueno

I am with the same doubt, I think it makes sense to expose when it’s fetched. The only reason I wouldn’t do this is if we get it for another reason than provide the experience to the user. I would love to hear from someone else.

tsegalla

Hi kannan.ganesan1989 and marcel ueno, great question.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other. In this case, there’s no difference between the two.
If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow. For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it. For that reason, this user should not be considered to be part of the experiment results.
- Amplitude uses the exposure as the denominator in every experiment metric you set up. This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results. Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

kannan.ganesan1989

Hi kannan.ganesan1989 and marcel ueno, great question.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other. In this case, there’s no difference between the two.
If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow. For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it. For that reason, this user should not be considered to be part of the experiment results.
- Amplitude uses the exposure as the denominator in every experiment metric you set up. This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results. Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

Thanks 👍

Quick Links

Unanswered Questions

playing videos 9:16 (vertical) in an iOS guide?
Hi all, We are getting set up on Amplitude and want to be able to have demo videos automatically show up in a guide in a 9:16 ratio (vertical/portrait) and not in the crunched landscape orientation that the current frame delivers for Vimeo and YouTube. We are using the iOS SDK, which seems like it may be the limiting…
Trying to import new user attribute from CSV but it doesn't work
I have a csv with 2 columns user_id and lead_source. I went to Catalog and selected CSV under Lookup Tables. Step 1: I brought in the CSV and Step 2: I mapped the first column (user_id) to User ID (Amplitude User). It showed me a sample of my CSV and showed both columns at the bottom of Step 2. I clickd Finish. I went to…
How to filter by Event properties in Segmentation chart
I am unable to find my Event Properties in the list of “+ Filters” in Segmentation chart.
notifications.integration.saveFailure - Unable to add Facebook Cohort as a Destination
Myself and other team members have tried using the Facebook Cohorts integration however we always receive the same error ‘notifications.integration.saveFailure When looking in Meta Business Manager it says that the integration is active however when going into the Destinations tab in Amplitude, it is blank I have tried the…
User ID is converting to a big number
I am setting user property wallet address, and using the same variable to set user ID. However, while walletAddress works (0xA66...54), User ID seemingly converts the hex to a big number (1.2905460614871781E+48) which is preventing us from using filtering and views. Any ideas?