Correlation vs. Causation: Why Experiments Are the Next Big Thing for Data-Driven Leaders

Some good books on experimentation: https://www.quora.com/What-are-some-good-books-to-read-on-the-practical-side-of-online-experimental-design/answer/Ronny-Kohavi
 

If you’re interested in learning more about the culture and many real-life examples, I teach a 10-hour interactive zoom class: https://bit.ly/ABClassRKLI, partly based on the book https://experimentguide.com 

Thank you @ronnyk! These books and your course are amazing resources! 😃

Amazing Coffee Chat this morning with Wil and Rachel! Adding some of the interesting questions we didn’t get to in the chat here for continued discussion :)

First question: Can you use historical data instead of running an online experiment to prove causation?

Another great question from our chat: How do you usually decide the % of users to roll out the experiment to?

Also from our chat: How do you know when to close out an experiment? How many results are good enough?

> Can you use historical data instead of running an online experiment to prove causation?
No.  You can not prove causality from observational data without significant assumption (which can only be claimed in very limited settings, that is, not in real-life business scenarios). 
Recommender systems have tried to find functions that work well; see great answer at https://www.quora.com/What-offline-evaluation-metric-for-recommender-systems-better-correlates-with-online-AB-test-results

This is why A/B tests, or controlled experiments, are the gold standard in science.

For slides about famous claims from observational data that were later proven wrong, see https://bit.ly/CODE2018Kohavi

> How do you know when to close out an experiment? How many results are good enough?

See https://www.quora.com/How-long-should-you-run-an-A-B-test-on-your-site-before-you-declare-one-a-winner/answer/Ronny-Kohavi

> How do you usually decide the % of users to roll out the experiment to?

The simple answer is split all your users across the variants (this gives you the maximum statistical power to detect small differences).
But don’t start an experiment that way, as new code is more buggy.  Start with a small percentage (e.g., 1%-5%) and make sure there are no egregious issues, then ramp-up to 50/50% or 1/3*3 (for A/B/C).  See https://experimentguide.com 

Thank you @ronnyk. Great stuff! 😀