Skip to main content

Hi there,

I’m looking to validate an activation event based on whether or not it results in 2x user retention.

To do this, I’ve created two user cohorts:

  • Group one: users that have signed up, then performed the activation event (aha moment) within a day
  • Group two: users that have signed up, but have not performed the activation event (aha moment) within a day

If it worked out, group one should have 2x group two’s 6 month retention.

To find out if this is true/false, I took the following steps: 

  1. Create a retention report of users returning to performing the aha moment month over month
  2. Set performed by Users parameter to group one
  3. Measured 6 month retention rate
  4. Create a second retention report of users returning to performing the aha moment month over month
  5. Set performed by Users parameter to group two
  6. Measured 6 month retention rate

To my surprise, group two has higher retention... This seems odd, so I feel I may be doing something wrong. Is there a mistake with the way I’m doing this? If not, is there a better way to do measure if an activation event results in 2x retention?

Thank you!

Hi! Your method sounds good so I suspect there might be a mistake, probably in step 2 or 5. Can you share a screenshot of the segmentation panel of your chart(s)?

@timothy-permutable yeah thanks for your help, here it is:

 

 

Notice anything off on your end?

I’m surprised that you’re using “starting event” = “meaningful usage”. This means that for group two you are looking at users who didn’t perform the activation in the first day, but DID perform it later (because only users who do the starting event are included in the chart). I would have expected “starting event” = “user:created” to get the 2X result you described.

 

With the current configuration, effectively delayed activation, I find it understandable that the retention could be higher in group two.

@timothy-permutable ah i didn’t think about it.

So retention is the measure of a user doing something and coming back to do it again correct? It seems that wouldn’t be account creation rather it’d be the return of a user doing the meaningful event right?

 

For example, if this was Amplitude, they’d measure retention as starting event: created report and then returned to create another report MoM. They’d test if a user has activation event to sign up and create a report within a day. How would you do it if this was the measure of retention?

Btw if my approach is flawed in some way. I’d love to know how I can best compare retention of users and how it differs based on those who make an activation event.

Goal: confirm if an event is an accurate activation event as described by it’s need to result in 2x retention

The best practice for retention is to set the starting event to something that happens early in the experience (e.g. install, signup) and only happens once. Then the return event should be whatever you want to measure as the action that they come back and repeat.

 

Given your description you should go for starting event “user:created” and return event “meaningful usage”. Then your comparison of activated vs not activated will be a fair one. What is the result when you do that? Do you get 2X?

@timothy-permutable sorry I believe in Mixpanel it may have been different which caused confusion that starting and ending event would be the same event that we want them to repeat (ie they book a uber and return to book an uber - starting and return event being the same event that we want them to repeat and in this case I want meaningful usage to repeat)

Now that I switched it up, I see it’s 40% vs 24%. Not exactly double. What do you think is this good enough or I should change the activation event?

I think that 2X thing is a good guide, but what you have is good enough. I would say what matters more is that you believe the activation event represents the user discovering the value of your product :)

 

Now it’s time to stop measuring and start making changes to your product to increase activation! Good luck!

Thank you @timothy-permutable for your help here! 😃

Reply


Terms & ConditionsCookie settings