Hi Community,
Does anyone who uses the BigQuery integration have the same problem as me? The integration has created literally tens of thousands of transfer configurations over the last year, which has made the GCP BigQuery Data Transfers UI more or less unusable as a result.
If so: What do people usually do about this?
Is it really necessary for the integration to create a new configuration every hour, or am I missing something?
Question
What's the best way to manage the large number of Transfer Configurations created by the BigQuery integration?
+5Hi Padraig,
This is Yuanyuan from the Amplitude Support team - happy to take a look for you.
Could you please share a bit more context about the transfer configurations you mentioned? Do you use this BigQuery import connection? Screenshots would be very helpful but if there is any private information you want to share, please let me know. Then I can start a private thread with you to continue the discussion.
Looking forward to hearing back from you!
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
This is Yuanyuan from the Amplitude Support team - happy to take a look for you.
Could you please share a bit more context about the transfer configurations you mentioned? Do you use this BigQuery import connection? Screenshots would be very helpful but if there is any private information you want to share, please let me know. Then I can start a private thread with you to continue the discussion.
Looking forward to hearing back from you!
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
I think I can give you some high level details here. We use the built-in BigQuery destination - here is a screenshot of the regular load jobs that run:
You can see that records get exported every half hour.
Over on the GCP side, this gets implemented via a series of one-off data transfer configurations from an AWS bucket under Amplitude’s control - I can show you some of the info from that:
Happy to start a private thread if you need additional details.
+5Hi Padraig,
Thanks for the extra information! Let me check with the engineer and get back to you.
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
Thanks for the extra information! Let me check with the engineer and get back to you.
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
+5Hi Padraig,
The engineer let me know that this is a known issue that is in our backlog. Before they can releases a solution, you are free to free to clean up the previously created transfer configs. Sorry for the inconvenience. I will keep you updated!
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
The engineer let me know that this is a known issue that is in our backlog. Before they can releases a solution, you are free to free to clean up the previously created transfer configs. Sorry for the inconvenience. I will keep you updated!
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
+5Hi Padraig,
Hope you had a nice weekend! The engineer let me know that they will allocate sometime in Q1/2025 or Q2/2025 to fix this issue. In the meantime, please kindly clean up the previously created transfer configs as you need.
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
Hope you had a nice weekend! The engineer let me know that they will allocate sometime in Q1/2025 or Q2/2025 to fix this issue. In the meantime, please kindly clean up the previously created transfer configs as you need.
Best,
Yuanyuan
P.S. Checkout upcoming events and user meetups on our events page.
Reply
Welcome to the Amplitude Community!
If you don't have an Amplitude account, you can create an Amplitude Starter account for free and enjoy direct access to the Community via SSO. Create an Amplitude account. You can also create a Guest account below!
If you're a current customer, select the domain you use to sign in with Amplitude.
analytics.amplitude.com analytics.eu.amplitude.comor
Welcome to the Amplitude Community!
If you don't have an Amplitude account, you can create an Amplitude Starter account for free and enjoy direct access to the Community via SSO. Create an Amplitude account. Want to sign up as a guest? Create a Community account.
If you're a current customer, select the domain you use to sign in with Amplitude.
analytics.amplitude.com analytics.eu.amplitude.comor
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.