We have two minor updates to quickly talk about to the lifecycle component of Experiment.
Lifecycle is the structure of Experiment in stages with Plan, Configure, Monitor, and Analyze as seen across the top of the page.
We’re so grateful for all of our customers who provide us feedback on the overall lifecycle of their experiments. We’re always aiming to have this lifecycle match what happens in reality. Two areas we’ve received feedback on are from users wanting to have a better understanding of how long it will take for an experiment to finish running as well as a bit more clarity on what to do when an experiment ends.
Analyze: New Duration Estimation
We have now added a Duration Estimate to the summary card while your experiment is running. This callout provides an estimate of how much longer we believe it will take for the analysis to reach statistical significance as it relates to your primary metric. We calculate this largely based on how much traffic the experiment has been receiving so far and project that forward as an estimate. We also provide an estimated best case and worst case scenario.
Analyze: Completing an Experiment
Previously when an experiment reached statistical significance, we presented users with a button labeled “Make a Decision”. While we ultimately think this is the best outcome for an experiment, to make a well informed product decision based on the data, sometimes an experiment does not end in making a product decision. Sometimes users would not click this button because they weren’t ready to make a final decision.
We’re now changing that button to read “Complete Experiment”, making that button more prominent in a few places, as well as making it more dynamic based on what our recommendation is. We’re also adding the ability to go back and revisit a decision, allowing users to change their previous decision. We hope this brings a bit more clarity to the purpose of this button to complete an experiment and take a next step.
Thank you again to everyone that continues to drive incredible decisions through Experiment and we’re always looking for ways to improve the product, both big and small.
