A Specific Profile ID is not pulled in via Dashboard API

Hi @Audino.Anter 

Does this chart use a group by clause?

I’m not 100% sure, but it looks like the existing chart API response might be running into result pruning.

I don’t see any mention of pruning in the API results, but I know that segmentation chart does undergo pruning when a group by is used. It should have been the other way round i.e. API response should list out all the ids if you are grouping by user id.

Hi @Saish Redkar ,

I just noticed that the chart has a warning “The results of this query are pruned”.

Yes, the chart use a group by clause as below,

Is there any alternative to build a similar dashboard that contains all the data above, where everything can be pulled via API?

Thanks!

I think the Rest API should give all the results even with a double group by.

For double group-bys, results may be limited to 500 group by results, in which each group by value pair is counted as a single result in the event seg chart.

Can you do a csv export of your chart view and compare the total item rows you get vs the one displayed on the chart and the API response? That might help to some extent.

Tagging the platform specialists for added insights into this -

  @belinda.chiu  @Denis Holmes@eddie.gaona 

Thanks @Saish Redkar I’ll make sure this gets submitted as a ticket as well. 

Hello @jmagg 

Thank you for the detailed insights!

I would like to ask, is there any other Amplitude API that would provide the entire bulk data without any limitation?

I noticed the documentation below that there are many different types of apis.

https://www.docs.developers.amplitude.com/analytics/api-reference-overview/#analytics-and-data-apis

I am just wondering which of the api above should be used that contain all the data below has the highest data rows limit?

https://analytics.amplitude.com/adform/chart/i8u86h8

https://analytics.amplitude.com/adform/chart/8hsipyp

https://analytics.amplitude.com/adform/chart/m1aw35y

I will scale down the date filter and make it 30 days instead.

Sincerely,

Audino

Hi @Audino.Anter 

Of all the APIs listed in there, only Dashboard REST API will be able to help you get data from your existing charts. All the other APIs are used for various other purposes.

As Julia mentioned, the CSV export of the existing chart from the UI has the highest chance of giving the maximum count.

If you want the highest coverage across a larger time interval, then you should look at exporting your event data with Export API or directly to some supported destinations (like snowflake). You can then do a distinct count on the desired property using custom SQL.

However, if you are just looking to get list of unique user ids satisfying a given condition, then creating a cohort and then exporting it is also a possible way. You have a limit of 1M users using this approach.

Ditto @Saish Redkar!

Hi @Saish Redkar ,

Sorry for bothering you again.

I tried to pull a small amount of data from the Amplitude Dashboard API. I pulled 1 hour worth of data from one of the dashboards above.

I noticed that the API Limit value is 1000 based on the previous response. When I exported the CSV, there were around 800 rows of data, but when I pulled via Dashboard API only ~700 worth data were exported.

Is it possible that not all data get extracted even if it’s below the 1000 rows limit?

Thank you for the support!

Hi @Audino.Anter -- would you mind sharing the URL to the chart you’re referencing in your most recent message please? I would be happy to take a look and see if I can determine why the CSV returns 800 rows whereas the Dashboard API returns only 700 rows.

Hi @jmagg ,

This is the dashboard
https://analytics.amplitude.com/adform/chart/thjs64j

I tried to pull the data for only 1 hour. The export CSV gave me 824 rows of data (831 - 7 rows of header). I also confirm that there’s no null value for the 3 or 4 digits agencyID on the csv file.

The Dashboard API returns 799 data. I used CTRL+F [‘ to calculate the number of 3 or 4 digit IDs minus one for the date 2023-04-03.

I pulled the raw data with the Python code below, to confirm that I did not make any change in the Amplitude Data being pulled.

def pulldata(url):    response = requests.request("GET", url, headers=headers, data=payload)    if response.status_code == 429:        time.sleep(300)        pulldata(url)    data = response.json()    df = pd.json_normalize(data)    df = df[["data.series", "data.seriesLabels", "data.xValues"]]    df.to_csv("checking.csv")

if __name__ == "__main__":    pulldata("https://amplitude.com/api/3/chart/thjs64j/query")

The ‘segment’ which refers the 3 or 4 digits AgencyID also shows 799 matches only, I print the raw data just to be sure.

I also checked this with other dates for 1 hour worth of data, and they all provide different number of rows, where the data pulled via API is smaller than the export CSV although it’s less than 1000 rows.

Thank you!

Hi @Audino.Anter -- thank you for the details. I was able to reproduce this on my end. Since I’m unsure why the CSV returns 824 rows and the API only 799, I’ve reached out to the Engineering team for assistance. I’ll be in touch as soon as I hear back from them!

Hi @Audino.Anter -- Engineering took a look and believes this may be a bug. That said, I filed a bug report so they can look further into it. I’ll keep you posted!

Hi @jmagg , thank you for your help. I will wait for an update from your side!

Hi @jmagg ,

Did the engineering team provide estimated time when the bug will be fixed?

Thank you

Hi @Audino.Anter - thank you for the nudge! I actually just heard back from Engineering yesterday and have some information to relay to you. Here’s a note from Engineering --

I read the code and just found the backend has another limit on the number of event properties for each segment group. It’s 10 or limit/number of segment groups, whichever is larger. So even though the overall result does not exceed the limit (which is 1000 for Dashboard REST API), the result in a single segment group can exceed the limit and get pruned. The more group segments there are (in the ticket case, it’s the number of different agencyId), the more likely the result of each group segment (agencyId) is pruned. There are over 400 different values of agencyId so the limit for each of them is 10. If there are more than 10 profileId for an agencyId, only 10 of them will be kept.

For CSV export, the overall limit is 10000, so the limit for each segment group is max{10, 10000/~400} ~= 25. That caused the result in CSV have more rows than the REST API. So this is an expected behavior.

Does this help clarify?

Hi @jmagg ,

Yes, it clarifies my doubts. Thank you!