Skip to main content
Solved

Firing fetch_experiment and exposure together (when fetch_experiment return variant)

  • February 9, 2023
  • 3 replies
  • 363 views
K

Hello,

 

We are using Ruby SDK initialize_remote to use fetch_experiment and exposure events.

My question is, is it best practise to fire exposure event when ever we fetch_experiment returns experiment?

 

Problem we think. Firing exposure and fetch separately will introduce unpredictability as customer might drop without seeing the experiment itself.

 

Your thoughts on this?  

Best answer by tsegalla

Hi @kannan.ganesan1989 and @marcel ueno, great question.

Regardless of when you fire each, first make sure that you’re always calling .fetch() first before calling .variant() / firing exposure.  This order is important to ensure we are assigning the user to a test and a variant first before considering them exposed so that your data is as accurate as possible.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

  • If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other.  In this case, there’s no difference between the two.
  • If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow.  For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it.  For that reason, this user should not be considered to be part of the experiment results.
    • Amplitude uses the exposure as the denominator in every experiment metric you set up.  This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results.  Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

View original
Did this topic help you find an answer to your question?

3 replies

M
  • marcel ueno
  • New Member
  • 1 reply
  • February 9, 2023

I am with the same doubt, I think it makes sense to expose when it’s fetched. The only reason I wouldn’t do this is if we get it for another reason than provide the experience to the user. I would love to hear from someone else.

T
Team Member
Forum|alt.badge.img+5
  • tsegalla
  • Team Member
  • 22 replies
  • Answer
  • February 9, 2023

Hi @kannan.ganesan1989 and @marcel ueno, great question.

Regardless of when you fire each, first make sure that you’re always calling .fetch() first before calling .variant() / firing exposure.  This order is important to ensure we are assigning the user to a test and a variant first before considering them exposed so that your data is as accurate as possible.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

  • If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other.  In this case, there’s no difference between the two.
  • If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow.  For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it.  For that reason, this user should not be considered to be part of the experiment results.
    • Amplitude uses the exposure as the denominator in every experiment metric you set up.  This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results.  Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

K
M
2 people like this
  • K
    kannan.ganesan1989
  • M
    marcel ueno
K
tsegalla wrote:

Hi @kannan.ganesan1989 and @marcel ueno, great question.

Regardless of when you fire each, first make sure that you’re always calling .fetch() first before calling .variant() / firing exposure.  This order is important to ensure we are assigning the user to a test and a variant first before considering them exposed so that your data is as accurate as possible.

In terms of best practice on how close together these calls get made, it really depends on where your test is taking place in your user flow:

  • If the change that you’re testing takes place on a home page/screen immediately after initializing, then it makes sense that the .fetch() call and the exposure would be called one right after the other.  In this case, there’s no difference between the two.
  • If the change that you’re testing takes place further down the user flow, we recommend calling .fetch() upon app initialization, and then calling .variant() / firing exposure closer to when the user would actually encounter your change in the flow.  For example, if you’re running an experiment on your pricing page, a user might be assigned via .fetch() on the home page for the experiment—but if they don’t visit the pricing page, they'll never actually be exposed to it.  For that reason, this user should not be considered to be part of the experiment results.
    • Amplitude uses the exposure as the denominator in every experiment metric you set up.  This means that while we can still measure if a user was assigned to a test and a variant upfront, we only include users who were exposed in the results.  Doing it this way helps ensure that you’re only analyzing user behavior as a result of them having experienced the change, reducing any noise from users that wouldn’t see the change.

I hope this is helpful!

Thanks 👍

Reply


Helpful Members this Week

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings