Solved

Comparing user retention of two cohorts to identify activation event

  • 8 January 2023
  • 9 replies
  • 261 views
A
Rank
Badge

Hi there,

I’m looking to validate an activation event based on whether or not it results in 2x user retention.

To do this, I’ve created two user cohorts:

  • Group one: users that have signed up, then performed the activation event (aha moment) within a day
  • Group two: users that have signed up, but have not performed the activation event (aha moment) within a day

If it worked out, group one should have 2x group two’s 6 month retention.

To find out if this is true/false, I took the following steps: 

  1. Create a retention report of users returning to performing the aha moment month over month
  2. Set performed by Users parameter to group one
  3. Measured 6 month retention rate
  4. Create a second retention report of users returning to performing the aha moment month over month
  5. Set performed by Users parameter to group two
  6. Measured 6 month retention rate

To my surprise, group two has higher retention... This seems odd, so I feel I may be doing something wrong. Is there a mistake with the way I’m doing this? If not, is there a better way to do measure if an activation event results in 2x retention?

Thank you!

icon

Best answer by timothy-permutable 11 January 2023, 09:09

View original

9 replies

timothy-permutable
Rank
Badge +10

Hi! Your method sounds good so I suspect there might be a mistake, probably in step 2 or 5. Can you share a screenshot of the segmentation panel of your chart(s)?

Timothy | Product Analytics Consultant - permutable.co
A
Rank
Badge

@timothy-permutable yeah thanks for your help, here it is:

 

 

Notice anything off on your end?

timothy-permutable
Rank
Badge +10

I’m surprised that you’re using “starting event” = “meaningful usage”. This means that for group two you are looking at users who didn’t perform the activation in the first day, but DID perform it later (because only users who do the starting event are included in the chart). I would have expected “starting event” = “user:created” to get the 2X result you described.

 

With the current configuration, effectively delayed activation, I find it understandable that the retention could be higher in group two.

Timothy | Product Analytics Consultant - permutable.co
A
Rank
Badge

@timothy-permutable ah i didn’t think about it.

So retention is the measure of a user doing something and coming back to do it again correct? It seems that wouldn’t be account creation rather it’d be the return of a user doing the meaningful event right?

 

For example, if this was Amplitude, they’d measure retention as starting event: created report and then returned to create another report MoM. They’d test if a user has activation event to sign up and create a report within a day. How would you do it if this was the measure of retention?

A
Rank
Badge

Btw if my approach is flawed in some way. I’d love to know how I can best compare retention of users and how it differs based on those who make an activation event.

Goal: confirm if an event is an accurate activation event as described by it’s need to result in 2x retention

timothy-permutable
Rank
Badge +10

The best practice for retention is to set the starting event to something that happens early in the experience (e.g. install, signup) and only happens once. Then the return event should be whatever you want to measure as the action that they come back and repeat.

 

Given your description you should go for starting event “user:created” and return event “meaningful usage”. Then your comparison of activated vs not activated will be a fair one. What is the result when you do that? Do you get 2X?

Timothy | Product Analytics Consultant - permutable.co
A
Rank
Badge

@timothy-permutable sorry I believe in Mixpanel it may have been different which caused confusion that starting and ending event would be the same event that we want them to repeat (ie they book a uber and return to book an uber - starting and return event being the same event that we want them to repeat and in this case I want meaningful usage to repeat)

Now that I switched it up, I see it’s 40% vs 24%. Not exactly double. What do you think is this good enough or I should change the activation event?

timothy-permutable
Rank
Badge +10

I think that 2X thing is a good guide, but what you have is good enough. I would say what matters more is that you believe the activation event represents the user discovering the value of your product :)

 

Now it’s time to stop measuring and start making changes to your product to increase activation! Good luck!

Timothy | Product Analytics Consultant - permutable.co
Jeremie Gluckman
Rank
Userlevel 6
Badge +9

Thank you @timothy-permutable for your help here! 😃

Reply


Terms & ConditionsCookie settings