Update by Stephen Choi, Marina Sergeyeva, & Shelley Wang, our lead engineers on diagnostics and warnings
One area that has become increasingly important for our customers is the ability to be warned when experiments are not quite going as expected. This meta-analysis of the quality of experiment data ensures that our customers can trust the analysis and results of each experiment.
Last month we released diagnostic run-time charts, as the first step in this direction, giving users the ability to track the health of their experiments in real-time. Watching the assignment and exposure event data allows anomalous data to surface that is otherwise hidden in other tools.
We’re excited to continue to expand on that work this month with two new updates: SRM Detection warnings and Experiment guardrail warnings.
SRM Detection Warnings
2 new charts & 2 new warning boxes
Sample Ratio Mismatch (SRM) occurs when there is an uneven or unintended distribution of exposures to the control and variant(s) in an experiment. As an example, if you’ve set a 50/50 allocation in your experiment so that 50% of your users see the control and 50% of your users see the variant, you would expect to see a 50/50 split in the actual exposures as planned. However, due to various reasons outside of our control, sometimes more users can see the control or the variant. This is usually an unintended bug in your code or deployment of the experiment in the feature flags.
When an SRM occurs, you may see something like 55% of users seeing control and 45% of users seeing the variant. This indicates something has gone wrong with the true randomization of the experiment and that the analysis & results should be highly scrutinized.
We will be publishing a blog on SRMs in the near future with more in-depth explanations on why they’re important to catch. Subscribe to our blog so you don’t miss that!
What we shipped!
We are adding two new charts to help users to get information to indicate whether there is a sample ratio mismatch ratio in the running experiment as well as warning boxes when an SRM has been detected.
Assignment to Exposure conversion chart
This chart provides information on how many assignment events actually have converted to an exposure event. It is another form of displaying the assignment and exposure events shown in the Monitor tab already. If there is a significant difference in the conversion for each variant, then it is a good indication of why an SRM, if at all, is happening.
Variant Jumping chart for each variant
Variant jumping is when the same user sees two or more variants. This is another good indication of an SRM if the numbers are significantly high.
SRM Detection Warnings
There will now be a warning banner in the Monitor tab and on the summary card in the Analyze tab if a significant SRM has been detected. We also provide a help center article which provides recommended actions.
Experiment Guardrail Warnings
Making foundational changes to the setup of an experiment after it has already started running is not best practice and can lead to unexpected results that are difficult to interpret. Customers have asked us to provide ample warning if they are about to make a change to the configuration of an experiment that could lead to invalidating the analysis. These new guardrail warnings will generate a pop-up message when a user is about to make a change that could affect the validity of an experiment.
When experiments influence product decisions the data must be trusted. If there are other areas in the product where you worry about data quality and experiment integrity, please let us know!
