Question

Data insufficient prompt in T Test Analysis Experiment

  • 12 June 2023
  • 3 replies
  • 113 views
V

Hi Amplitude Team!
I would like to ask regarding the Amplitude Experiment. The case is that we have a sample size requirement of 57,000, two sided, and 95% confidence level. However when we wanted to analyse the result of the experiment and changed the Statistical setting to the required, the data showed that it “needs more data to be conclusive” and the statistical in the data is 0% when the supposedly performance is -1.8% (with MDE of 1%).
 


Is there any reason why this prompt is showing? Thank you!

3 replies

N
Rank
Userlevel 2
Badge +5

Hey @Vania Alya,

This is a statistical analysis issue more than a tool issue, what the tool is showing is:
your audience sample per variant must be at least 57000 users to perform some statistical calculations on how those users interact with the variants you propose to them, the exposure remaining is 0 because you already hit the sample size target (each variant is exposed to more than 60k users each).
As you can see in your image both variants behave similar (there’s no statistical significant difference between them) that can be related to either your changes were too granular (you should cause a greater minimal detectible effect in your experiments), or you need way more time for your users to interact with your experiments to produce a statistical significant difference. 
These are the most tricky results you can have while doing an experiment because you should choose based on your experience what to do next, stop the experiment and start a new experiment that has a greater minimal detectible effect or keep running the experiment (this option is tricky and I personally would recommend to use it in specific cases, you should know your product).

Hope it helps! :)

V

Hi @Naryie Vasquez , Thank you so much for the explanation that you have given and very much appreciated. However i wanted to raise a couple of insights that made me question it. So :
1. We hit the MDE (The MDE was 1% and the relative performance was -1.81%)
2. If we use the sequential method, it gives a 99% significance, thus giving the question why using t test it gives a 0% significance

3. We hit the samples needed (the samples needed was 57,000 and for each sample we gathered ~61000)

So was wondering if there are any other explanation because as of now it hit around 90k exposures it still gives 0% significance.. Thank youu

N
Rank
Userlevel 2
Badge +5

Hi @Vania Alya, 
As you can see in the docs says:

The Analyze tab will now show the T-test results of your experimental data. Please note that sequential testing is the default test type. If you change the test type to a T-test, Amplitude Experiment will not save that change when you refresh or leave the page
It looks like your experiment was made using sequential as the default method, that maybe can explain the change.
The other thing is MDE is something we couldn’t truly calculate but estimate (that’s the reason why we do experiments), you have different scenarios in which you should choose which method is best to analyze your experiment using either the sequential method or the t-test one, given your initial hypothesis, you can find a larger explanation about that on this doc. Once you solve the question Which method is better?, you should consider that some experiments you can have either null hypothesis, false positives or false negatives, so choosing the method that better adjust to your initial conditions is imperative.
Take into account that reaching the sample size doesn’t guarantee your results will be statistically significant, so that the choice now is either stop the experiment and start other with a greater MDE or keep the current experiment running.

Reply


Terms & ConditionsCookie settings