Hello,

Happy to discuss this with you! Could you tell me more about the metrics you think would be beneficial to add to this feature? Also, could you share your use cases.

Thanks,

Sydney

Hello,

I was able to sync internally and get you some more information on Releases. Releases can be used to annotate chart content when product changes go out and collect corresponding analyses about the performance of the feature. Automation of analysis is something we've considered doing with the structured data but we don't have a timeline for that at this time.

Hope this helps you utilize Releases let me know if you have any thoughts or feedback you want to share!

Best,

Sydney

Hello Sidney, thanks for following up.

Automation Analysis of Release done by Amplitude is really something I would look forward.

Keep in mind that those kind of release analysis are not straightforward (almost impossible) to do from the user interface. Those factors are all messing with the analysis at the same time:
- Lots of different events, no automatic detection to see which ones are affected.
- Adoption rates will mess with any chart that is not about percentage.

- Releases might start on a different day.
- Seasonality factors can kick in and create False Positives.
- Releases can be different on an hourly level, but small changes will only be visible after one week (and a new release kicks in).

But maybe we can start simple:
what is the best way to compare the occurrence of one event between two releases ?

Hi Castor,

I would say to use an Event Segmentation chart. You can view a specific event done over a time frame. The Releases will show as a vertical line on the chart and you will be able to see the occurrences of the event between the two Releases frames.

You can see an example here on our demo:

https://analytics.amplitude.com/demo/chart/new/2maa34b

These two lines are different Releases:

Hello Sydney, thanks for your contribution.

I will call the initial release release A. I will call the release associated to the first vertical line release B, and to the second one release C.
I encounter several issue with this approach, as it echoes my points from above:

• We do see the actual date of the release, but not the adoption rate at any point. What was the traffic / usage of the different release? On Dec. 28th, was release B accessible to 100% of the users or only rolled out to a few testers? If adoption was low, would you still trust the occurence of the event in release B? 
  If the release was done at 23p.m.; would you still trust the occurence of the event in release B?
  In our case, users are free to update their app whenever they want, so adoption rate could be anywhere and split between any number of versions.
• Jan 1st is a bit lower than Dec 25th; though it is the same day of the week. Is the difference statistically significant, if yes it could be due to the release ? How does Amplitude support us  in assessing this ?
• Let’s say the release affects the event collection negatively. How does Amplitude support us in the explore/exploit tradeoff ? (keep the release live to gather more data; stop it to restore the “positive” behaviour).
• Your app is unaffected by Christmas and new year, so we can not really discuss the seasonality issues .
  
   

I see your points based on what you have said there are a couple of notes I can make. I would like to start by saying as a Platform Specialist I can help you understand a feature. Regarding how to interpret your data or best business use cases you will need to decide that on your end or possibly with your Customer Success Manager if you are on our paid plan!

• We do see the actual date of the release, but not the adoption rate at any point. What was the traffic/usage of the different releases? On Dec. 28th, was release B accessible to 100% of the users or only rolled out to a few testers? 
• You can view the usage of users on the release is in the "Metrics" section it will show the number of unique users that have been exposed to your Release, and the percentage of your active User base that represents see here for the formula. 

• If adoption was low, would you still trust the occurence of the event in release B? 
  If the release was done at 23p.m.; would you still trust the occurence of the event in release B?
• Whether or not to “trust an occurrence of an event” would depend on what you decide. Perhaps other community members can give input here! One thing you can do is a group by ‘Version’ on a chart to see users’ performance of an event in each ‘Version.’

• Jan 1st is a bit lower than Dec 25th; though it is the same day of the week. Is the difference statistically significant, if yes it could be due to the release? How does Amplitude support us in assessing this?
• Generally, interpreting the data is left to the customer as you will know more about your product and releases than we will. We do have paid add ons like Root Cause Analysis and Anomaly and Forcast that help you determine if data points are anomalies and their possible causes.

• Let’s say the release affects the event collection negatively. How does Amplitude support us in the explore/exploit tradeoff? (keep the release live to gather more data; stop it to restore the “positive” behaviour).
• We will not be in control of whether or not a release is live. That will be determined on your end and it will be your responsibility to update or keep the release live on Amplitude. As for exploring whether or not the release is affects event performance that will also be up to you to decide but you can possibly use the features I mentioned above Root Cause Analysis and Anomaly and Forecast to help you!

I would think of releases as a way to track what version your users are using. Our tools can help you uncover some causes but ultimately interpretation will be up to you to decide what is or is not significant for your product.