Solved

Backfill historical data to BigQuery

  • 18 December 2022
  • 5 replies
  • 311 views
R
Rank
Badge

I have a BigQuery data destination which works great. We were able to export all of our data since we started collecting data on August 10, 2022.

 

We also recently imported historical data into Amplitude going back to October 2021. The issue is that the BigQuery backfill option is saying that it first saw data on August 10, 2022 (which is true) but we want to backfill the newly imported historical data all the way from October 2021. 

Is this possible? The calendar date picker doesn’t let me go beyond August 10, 2022.

icon

Best answer by rubenugarte 27 December 2022, 18:28

View original

5 replies

jarren.patao
Rank
Userlevel 4

Hi @rubenugarte, in this case, one workaround that I might try to suggest would be to send a manual event to be captured to this project. Since the date picker won’t allow you to go beyond what was currently seen within Amplitude, you could potentially send an event via our APIs on Oct. 2021 so our systems are able to query that far back.

 

Otherwise, it might be good to attempt setting up a secondary BQ destination to pull all data from the associated bucket.

Do it. Do it right. Do it right now.
R
Rank
Badge

@jarren.patao thank you. We fired an event using the HTTP API. Do you know how long it might take for the BigQuery backfill page to refresh the first seen event (if at all)?

R
Rank
Badge

I also tried to setup a secondary BigQuery destination but it still limits the backfill to the Aug 10, 2022 date.

jarren.patao
Rank
Userlevel 4

Thanks for your response @rubenugarte. I’ve got to check in with our product team to understand what might be happening here and how to circumvent this behavior, but I’ll get back here when I can.

Do it. Do it right. Do it right now.
R
Rank
Badge

@jarren.patao update here. Turns out Amplitude was able to export the historical data through a backfill. It seems that the “First Event Date” means that any events imported after that date still get exported. It’s a little confusing but it worked. 

Reply


Terms & ConditionsCookie settings