Solved

Unable to fetch new file from GCS Bucket

  • 15 April 2023
  • 7 replies
  • 140 views

I am new to Amplitude, I have configured a source to GCS for event data import. It imported first time when I have configured, but later whenI am pushing another files to GCP bucket with same extension and format it is not reading that file.

icon

Best answer by Esther Trapadoux 20 April 2023, 07:02

View original

7 replies

Userlevel 5
Badge +8

Hi @anandraprinc welcome to the community! Happy to help troubleshoot this. First off, when importing data from a Google Cloud Storage, we have some documentation here: https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/ 
 
Would you be able confirm that all pre-requisites are met for this integration and also where you might be seeing the sync behavior that you've described? 
 
After getting a bit more information around the behavior you're experiencing, I may be able to help further.

Hello @Esther Trapadoux Thanks for your response.

I have followed the same documentation given here  https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/ 

I have done all the required setup and It also picked up data first time. 

But when the new files comes to GCP Bucket , It did not picked up.

I am not understanding the configuration given here https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/#storage-organization-requirements I think I might have missed something.

Thanks

Awaiting your response

I am running into the same issue. I followed the storage organization requirements to the best that I could. My files are in my bucket organised as follows:


<bucket-name>/amplitude/2023/04/19/11/whatthetest.json

My folder is amplitude. so that is the prefix. But the data is still not getting imported.

Userlevel 5
Badge +8

Hi @anandraprinc and @mukkkkkk got it, and thanks for following the steps in the documentation. It could be a matter of event mapping as well, have you also seen the Guided Converter Creation section to see if there are any steps that may have been missed during setup: https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/?h=gc#guided-converter-creation 
 
A few more things to check otherwise:
 
Make sure you’ve taken care of some prerequisites.

  • Make sure you have admin permissions for your Amplitude org.
  • Make sure you have a GCS service account with the appropriate permissions. Learn more.
  • Make sure that a project exists to receive the data. If not, create a new project.
  • Make sure your GCS bucket has data files ready to be ingested. They must conform to the mappings that you outline in your converter file.
  • Make sure the data in your GCS bucket follows the format outlined in Amplitude's HTTP API v2 spec.

Thanks @Esther Trapadoux . It worked . I was doing mistake while storage organization requirements configuration. 

Thanks you so much.

 

Userlevel 5
Badge +8

@anandraprinc YAY!!!! Glad you got it to work. 🎉

Worked for me as well. It just took a day or so for it to start working. Not sure why that was the case but I just had to wait

Reply