Product Experimentation

Amplitude Experiment is our product that allows teams to remotely modify and configure the product experiences for unique audiences.

  • 84 Topics
  • 168 Replies

84 Topics

Amplitude Admin
Community Manager
Amplitude AdminCommunity Manager
published in Product Experimentation

Welcome to the product experimentation forum!

Amplitude Experiment is our product that allows teams to remotely modify and configure the product experiences for unique audiences This is where you can ask and get feedback on questions such as: ✅ Am I able to export the data I have in Experiment similar to Analytics?✅ How can I re-assign variants to my users?✅ Why are my users not getting targeted the right variant?✅ What happens if we merge projects in Experiment? Helpful resources: ❓Overview of Amplitude Experiment❓The experimentation workflow: from configuring to troubleshooting your experiment❓Amplitude Experiment key terms ❓Set up mutually-exclusive experiments❓Understand the Experiment Analysis view❓Advanced topics in Experiment: duration estimates, sequential testing for statistical inference, MDE and confidence intervals, randominzation, performance and scaling, sample ratio mismatches❓Amplitude Academy course: Getting Started with ExperimentStill have questions? Post them here! 

3871
T
1 year ago
D
Novice
dargeliesNovice
asked in Product Experimentation

Experiment Flag Slack Integration / Webhook

Hey there,Recently we had an incident that was caused by a feature flag being toggled. The incident commander was not aware of the feature flag being toggled to understand what changed in the system. Typically, we push system changes like this to Slack with automations to make it more visible. I noticed there is a Slack integration for some Amplitude features and webhooks for custom monitors, but didn’t see this. What we’re looking for is an integration or webhook that will notify us when a feature flag state is changed. Is there any way to do something like this? Thanks

3
jarren.patao
Community Manager
21 hours ago
L
Helpful Hero
liukaichiHelpful Hero
asked in Product Experimentation

Question about using Bucketing Slat

I know this is documented quite well here: https://www.docs.developers.amplitude.com/experiment/general/evaluation/implementation/#activationBut this experiment is very important, so I just want to make sure I have this right. We have 3 flags that we want to ensure get to the same users. If we use the same Bucketing Salt for each of the flags, then if we keep the segments and rollout numbers the same, then the same users should receive all 3 flags. Is this correct? Are there any pitfalls that we should be aware of when doing something like this? Thanks in advance.

2
Thao Nguyen
Team Member
13 days ago
A
Alexandra ResslerNew Member
asked in Product Experimentation

Can Amplitude do network experimentation?

In my product, users can send each other messages.  If we want to experiment on how messaging works, we can’t run a typical A/B experiment because we need both the sender & recipient to be in the same variant.  One standard way of addressing this is running experiments in different geographic locations, with the assumption that each location has a mostly-closed network.  This hasn’t worked for us, unfortunately, as our separate locations aren’t very similar in the first place so we can’t compare them.  Some companies have the ability to expose experiments based on network clustering.  Is that a feature that Amplitude has?Side note: I have bet on with my engineer that the answer is no. =D  So you won’t disappoint me if that’s the case!The best alternative I can think of is to use BigQuery to manually separate users into clusters, normalize on relevant attributes, try to pair similar clusters to assign to different variants, and upload those assignments as a user property, but I have

3
J
Novice
16 days ago
M
Micah BlackburnNew Member
asked in Product Experimentation

New Developer Here With Some UTM Questions

Hello, I am developing amplitude with my ecommerce website smartbettor.ai. I have currently implemented amplitude to track events using the java script sdk. I track payments with stripe and want to track sign ups. After a successful stripe checkout users are redirected to a registration form. The only way to complete sign up is to complete this registration form so I have our ‘sign up’ event being the submission of this form. There are 2 problems with this currently.  Numbers between what stripe says the customer sign ups are and what amplitude is saying are very different. Why would amplitude not be able to track every time this event happens and have significantly less than what stripe says? We want to track utm source. We are starting ad campaigns and want to know what website the user came from. I added these lines which I assumed would track: includeUtm: true, // Track UTM parameters includeReferrer: true, // Track referrer includeGclid: true, // Track Google Click Identifier incl

0
M
1 month ago
M
Mohammad GhasemiNew Member
asked in Product Experimentation

attribution currently is unavailable

when i want to use attribution, it shows me : attribution currently is unavailablehow can i do for that?

0
M
2 months ago
L
Liron BurrackNew Member
asked in Product Experimentation

amplitude/experiment-js-client SDK vs amplitude-js SDK?

Does amplitude-js npm package includes the amplitude/experiment-js SDK inside it?what is the difference between them? 

0
L
2 months ago
Rune Andersen
Well-Versed
Rune AndersenWell-Versed
asked in Product Experimentation

Any Payload size limit in a variant ?

Questions is:Are there a limit for how large (line, characters, mb) a JSON Payload can be as a part of Feature flag.For a client im implementing Experiment/Feature Flags for IOS and Android. One of the Feature flag planned are to be used as a “All Users 100%” with a payload with info about shops etc. Example of this is an object of all shops, and info for each shop.Intention is to make only one “App Data” feature flag that will grow overtime.For now it’s only 74 lines, and est 1200 characters. But it’s intended to grow over time.  

0
Rune Andersen
Well-Versed
2 months ago
M
Michael McNicholasNew Member
asked in Product Experimentation

Does anyone have any experience using Amplitude Experiment and Google Tag Manager together?

Does anyone have any experience using Amplitude Experiment and Google Tag Manager together? Our goal is to have Amplitude Experiment set-up like a typical client side tool since it would be difficult to integrate our servers-side code due to some vendor issues. (long-term addressable, but not in the near term) So the basic thought is1> Fire Amplitude Analytics in GTM with the  “Amplitude Analytics Browser SDK” GTM community template. 2> Fire a script tag to initialize the Expierment. <script src=“https://unpkg.com/@amplitude/experiment-js-client@1.9.0/dist/experiment.umd.js”></script> 3> Create a Google Tag Manager Variable which is a custom javascript variable and captures which the experiments and variations the visitor is in. 4> Use those values in the GTM variables to execute or not, the code in our experiments. This is not pretty.  But I haven’t come across much content on Amplitude Experiment, since it is a newer product and not for the lower tier plans. a

2
M
3 months ago
F
Federico HuneeusNew Member
asked in Product Experimentation

Trouble tracking exposure events. Assignment events work, exposure doesn't

Hey guys, can someone give me a hand. I’ve followed pretty much every step in the guides to track exposure, but nothing gets registered on my dashboard. What I’ve done is loading the experiment.start() and then running the line const variant = experiment.variant(‘experiment-one’) [experiment-one is the flag], but it doesn’t work.i tried both the Amplitude and Segment integration, but no luck. This is one way I tried (with Segment → )export async function loadScript() { const promise = new Promise((resolve, reject) => { const script = document.createElement("script"); script.type = "text/javascript"; // script.async = true; // script.defer = true; script.text = ` !function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","page

0
F
3 months ago
D
Daniel BarriaNew Member
asked in Product Experimentation

Evaluation REST API not working

Hey everyone! Here is a situation that I am facing beign new using amplitude. We’re trying to use amplitude Evaluation REST API to get users’ variant for a feature flag we’ve created. All seems to be alright, deployment key is created, feature flag is created and active but for some reason when I try to get user variant it always returns an empty objecthttps://www.docs.developers.amplitude.com/experiment/apis/evaluation-api/#exampleI’ve tested from that page and also doing a cURL. Any help would be awesome! Thanks

3
D
3 months ago
N
NitishKumarNew Member
asked in Product Experimentation

In the funnel of "Play Song or Video" to "Purchase Song or Video", what percentage of users do "Play Song or Video" 3 times, before conversion (Purchase Song or Video)

I have created this chart https://app.amplitude.com/analytics/demo/chart/6yds5ago/edit/rz913e1d, but want to know if this is correct, please help.

1
Saish Redkar
Expert
3 months ago
N
NitishKumarNew Member
asked in Product Experimentation

 What are the Average and Median times taken to do "Send Message" after "User Sign Up"? What is the distribution chart of "Time Taken" for this conversion?

This is in B2B Chat Analytics, I tried but don’t know if my answer is correct. Hope someone can help me with this. https://app.amplitude.com/analytics/demo/chart/hjqmr0zx/edit/cbmsp21w

1
Saish Redkar
Expert
3 months ago
N
NitishKumarNew Member
asked in Product Experimentation

 What are the Average and Median times taken to do "Send Message" after "User Sign Up"? What is the distribution chart of "Time Taken" for this conversion

This is in B2B Chat Analytics, I tried but don’t know if my answer is correct. Hope someone can help me with this. https://app.amplitude.com/analytics/demo/chart/hjqmr0zx/edit/cbmsp21w

1
Saish Redkar
Expert
3 months ago
P
Prashant PadhyNew Member
asked in Product Experimentation

What is the conversion from "Add Item to Wishlist" to "Complete Purchase" completed within 1 day, from Japan users, for "Item_ID" = 100, in June?

E - COMMERCE TOOL

0
P
3 months ago
N
NitishKumarNew Member
asked in Product Experimentation

What is the true conversion(Event level conversion) from "Select Song or Video" to "Play Song or Video", when the Song or Video is selected from "Main Landing Screen". Compare this conversion number by different "Sources" , in the same chart.

Hi, I feel my answer is a little close https://app.amplitude.com/analytics/demo/chart/hjqmr0zx, Is this correct, if not I need little assistance.

1
Saish Redkar
Expert
3 months ago
N
NitishKumarNew Member
asked in Product Experimentation

What is the conversion from "Play Song or Video '' to "Purchase Song or Video '' , if the user follows a playlist (Follow Playlist) within 24 hours of first use?

Hi, I tried to do it this way https://app.amplitude.com/analytics/demo/chart/new/3w9uxdfi, the problem I am not able to figure out this part,  “if the user follows a playlist (Follow Playlist) within 24 hours of first use”. Hoping that the chart above is the correct answer for the question (if not please help me on what to change). How to implement this 24 hours part?

3
N
3 months ago
D
Dave PorterNew Member
asked in Product Experimentation

Separate Amplitude Experiment deployment or not for SSR server?

BackgroundThe examples in the server side rendering guide for Amplitude Experiment have…a “server” deployment key for initialising the node SDK used to fetch initial variants, see https://www.docs.developers.amplitude.com/experiment/guides/server-side-rendering/#initialize-the-server-sdka “client” deployment key for initialising the javascript SDK ExperimentClient with those initial variants, also used on the server. See first “if” branch here https://www.docs.developers.amplitude.com/experiment/guides/server-side-rendering/#initialize-the-client-sdk-on-render. The “client” deployment key matches that used to initialise the ExperimentClient in the browser in the second “if” branch.However, the Amplitude Experiment Node SSR Demo app here https://github.com/amplitude/experiment-node-server/tree/main/packages/ssr-demo uses an identical “client” deployment key for all three. (https://github.com/amplitude/experiment-node-server/blob/main/packages/ssr-demo/lib/experiment.ts and https://githu

0
D
4 months ago
C
Casey HNew Member
asked in Product Experimentation

Redirect A/B test with Amplitude

Hi, Has anyone ever succeeded in settng up a redirect A/B test using Amplitude Experiment? We have been looking for instructions to set up redirect test but couldn’t find any. 

0
C
5 months ago
V
Valentin IvanovNew Member
asked in Product Experimentation

Missing User ID in Start Session - GTM

Hi everyone, for my project I use GTM template “Amplitude Analytics Browser SDK” to initialise the Amplitude but I set userId and identify user in the project itself(by calling the newly created amplitude object), because GTM doesn’t have access to the information needed. Also I set timer with 30 retries in case that object is not initialised yet.Now what happens is that for majority of users but not all(around 70%) I can’t see information from identify, as well as User ID, in the user properties of Start Session event . However in the future events of same session the information I send with identify method is present but User ID is still missing. This causes the table in User Look-Up to show Amplitude ID instead of user Id for these events(but if you open the page of the user, he has correct Id). This is applicable only for the 70% of users I mentioned at the beginning, the rest are receiving identify data and userId in Start Session, so further events have userId set.Does anyone kno

0
V
6 months ago
T
Thomas GroendalNew Member
asked in Product Experimentation

How do I correlate time in platform to retention?

We’re working on a tool that is a platform for investigations in a virtual machine. We have a thesis that more time spent in the machine itself is going to be indicative of happy customers, as we meet a broader range of their investigative tasks in platform. I want to test this by seeing if time in platform is predictive of retention. I can measure time in platform on average, but I can’t figure out how to link it back to retention.I’m also open to criticism of the idea in general. 

1
Saish Redkar
Expert
6 months ago
N
Novita OliveraNew Member
asked in Product Experimentation

Change experiment key name

Can we change the experiment key name for existing Amplitude experiment?

0
N
6 months ago
L
Novice
lanceNovice
asked in Product Experimentation

Session metrics in experiments

Is it possible to track session metrics (e.g., Time spent per user, Avg length, etc.) in an Experiment Results chart? In the past we’ve tried using User Session charts but we aren’t cleanly able to create separate treatment and control segments and it does not have the same functionality in leveraging the only after Exposure event… feature in an Experiment Results chart 

0
L
Novice
7 months ago
J
Jacky LaiNew Member
asked in Product Experimentation

What is the best way to differentiate the path in funnel analysis?

I have one button, and one dropdown on the same page. There will be 3 events for each:1. button shown → button clicked → target page2. dropdown shown → dropdown clicked → target pageMy question is,Q1.what’s the granularity of time interval amplitude is using?Let’s say, if the granularity is 1 minute, within 1 minutebutton shown → dropdown shown → button clicked → dropdown clicked → target pageSince all the events happen in one minute, how to tell which scenarios (button or dropdown) completed the flow?Thanks.

1
dangrainger
Super Helper
7 months ago
M
Manuel FenklNew Member
asked in Product Experimentation

Amplitude Integration via Kotlin Multiplatform (cross-platform)

Hey folks, we are currently on the way towards migration from GA towards Amplitude. We have one iOS and one Android App, and we want to take this opportunity to integrate Amplitude via Kotlin Multiplatform Mobile (KMM).Goal: One business logic for the whole analytics integration in order to reduce the overhead of maintaining it on 2 platform and - most important - consistency among the platforms.Here is a video from GDG in Melbourne last week, talking about that in a more general way:  Does someone here in the community maby has experiences with such an integration? If yes, I would be very happy to get some insights about it. How did go, pitfalls, would you do it again, etc :)Thanks a lot!Best, Manuel 

1
M
7 months ago

Terms & ConditionsCookie settings