What's the best way to manage the large number of Transfer Configurations created by the BigQuery integration?

Hi Community,

Does anyone who uses the BigQuery integration have the same problem as me? The integration has created literally tens of thousands of transfer configurations over the last year, which has made the GCP BigQuery Data Transfers UI more or less unusable as a result.

If so: What do people usually do about this?

Is it really necessary for the integration to create a new configuration every hour, or am I missing something?

Find more posts tagged with

Integrations

BigQuery

Export data to bigquery

Accepted answers

All comments

Yuanyuan Zhang

Hi Padraig,

This is Yuanyuan from the Amplitude Support team - happy to take a look for you.

Could you please share a bit more context about the transfer configurations you mentioned? Do you use this BigQuery import connection? Screenshots would be very helpful but if there is any private information you want to share, please let me know. Then I can start a private thread with you to continue the discussion.

Looking forward to hearing back from you!

Best,
Yuanyuan

P.S. Checkout upcoming events and user meetups on our events page.

Paddy Alton

I think I can give you some high level details here. We use the built-in BigQuery destination - here is a screenshot of the regular load jobs that run:

You can see that records get exported every half hour.

Over on the GCP side, this gets implemented via a series of one-off data transfer configurations from an AWS bucket under Amplitude’s control - I can show you some of the info from that:

Happy to start a private thread if you need additional details.

Yuanyuan Zhang

Hi Padraig,

Thanks for the extra information! Let me check with the engineer and get back to you.

Best,
Yuanyuan

P.S. Checkout upcoming events and user meetups on our events page.

Yuanyuan Zhang

Hi Padraig,

The engineer let me know that this is a known issue that is in our backlog. Before they can releases a solution, you are free to free to clean up the previously created transfer configs. Sorry for the inconvenience. I will keep you updated!

Best,
Yuanyuan

P.S. Checkout upcoming events and user meetups on our events page.

Yuanyuan Zhang

Hi Padraig,

Hope you had a nice weekend! The engineer let me know that they will allocate sometime in Q1/2025 or Q2/2025 to fix this issue. In the meantime, please kindly clean up the previously created transfer configs as you need.

Best,
Yuanyuan

P.S. Checkout upcoming events and user meetups on our events page.

Quick Links

Unanswered Questions

Import data from one amplitude account to another
How do I transfer data from one amplitude account to another? It seems all the community chats about this have completely broken links. Easy to export, no way to import, very frustrating. :(
@amplitude/plugin-web-vitals-browser build fails on CRA v4 (react-scripts@4.0.3)
When using @amplitude/plugin-web-vitals-browser with Create React App v4 (react-scripts@4.0.3), the build fails with: ./node_modules/@amplitude/plugin-web-vitals-browser/node_modules/web-vitals/dist/web-vitals.js<br /><br />SyntaxError:…
Question about Amplitude Export API Data Delay
Hello, I'm using SpringBoot and Kotlin to integrate with the Amplitude Export API. The official Amplitude documentation (https://amplitude.com/docs/apis/analytics/export) states that "data is available to export at a minimum within 2 hours of when the servers received it." However, there's no clear information about the…
Retrieve via the API users based on a user property
Hello, Is there a way to make a query like: “Give me all the users that have the user property X equal to 10”, by using just any of the APIs, without making a cohort first through the Amplitude’s UI? Thank you!
400 Status Code Error When Loading "Page Viewed: Cart" Data
Hi, I’m trying to create a chart in Amplitude using the Page Viewed: Cart event, measured as Uniques, over the past 7 days. However, I keep getting this error on the graph: "An error occurred while loading your data. (400 status code)" Other events seem to load fine, so it looks specific to this event or configuration. Has…