Unable to fetch new file from GCS Bucket

I am new to Amplitude, I have configured a source to GCS for event data import. It imported first time when I have configured, but later whenI am pushing another files to GCP bucket with same extension and format it is not reading that file.

Find more posts tagged with

Integrations

Event

GCS

Import

Sources

Accepted answers

Esther Trapadoux

Hi anandraprinc and mukkkkkk got it, and thanks for following the steps in the documentation. It could be a matter of event mapping as well, have you also seen the Guided Converter Creation section to see if there are any steps that may have been missed during setup: https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/?h=gc#guided-converter-creation

A few more things to check otherwise:

Make sure you’ve taken care of some prerequisites.

Make sure you have admin permissions for your Amplitude org.
Make sure you have a GCS service account with the appropriate permissions. Learn more.
Make sure that a project exists to receive the data. If not, create a new project.
Make sure your GCS bucket has data files ready to be ingested. They must conform to the mappings that you outline in your converter file.
Make sure the data in your GCS bucket follows the format outlined in Amplitude's HTTP API v2 spec.

All comments

Esther Trapadoux

Hi anandraprinc welcome to the community! Happy to help troubleshoot this. First off, when importing data from a Google Cloud Storage, we have some documentation here: https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/

Would you be able confirm that all pre-requisites are met for this integration and also where you might be seeing the sync behavior that you've described?

After getting a bit more information around the behavior you're experiencing, I may be able to help further.

anandraprinc

Hello Esther Trapadoux Thanks for your response.

I have followed the same documentation given here https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/

I have done all the required setup and It also picked up data first time.

But when the new files comes to GCP Bucket , It did not picked up.

I am not understanding the configuration given here https://www.docs.developers.amplitude.com/data/sources/google-cloud-storage/#storage-organization-requirements I think I might have missed something.

Thanks

Awaiting your response

mukkkkkk

I am running into the same issue. I followed the storage organization requirements to the best that I could. My files are in my bucket organised as follows:

<bucket-name>/amplitude/2023/04/19/11/whatthetest.json

My folder is amplitude. so that is the prefix. But the data is still not getting imported.

Esther Trapadoux

Make sure you have admin permissions for your Amplitude org.
Make sure you have a GCS service account with the appropriate permissions. Learn more.
Make sure that a project exists to receive the data. If not, create a new project.
Make sure your GCS bucket has data files ready to be ingested. They must conform to the mappings that you outline in your converter file.
Make sure the data in your GCS bucket follows the format outlined in Amplitude's HTTP API v2 spec.

anandraprinc

Thanks Esther Trapadoux . It worked . I was doing mistake while storage organization requirements configuration.

Thanks you so much.

Esther Trapadoux

anandraprinc YAY!!!! Glad you got it to work. 🎉

mukkkkkk

Worked for me as well. It just took a day or so for it to start working. Not sure why that was the case but I just had to wait

Quick Links

Unanswered Questions

Experiment - Exposures Without Assignment (User is Assigned in Event Stream)
We are running an experiment, and in charts seeing a higher number of exposures than assignments. When we drill into users who have an exposure and not an assignment, they do end up having an “Assignment” event in their user event stream. What is preventing them from being included in the roll-up assignment calculation?…
Experiments: Saved pages
When im creating we experiment, there is option “Saved Pages” here… I searched everywhere and I could not find a single mention of this in docs and i could not figure out how to set this up. My goal is to define a list of pages, ideally by importing a csv. I know i could create rules, but defining that for tens or hundreds…
401 Invalid API Key
I am trying to integrate aplitude in my kotlin android application. My dashboad is on US servase zone. I am sure that i copied the right api key for my project but logs shows that the api key is invaled with the staus code 401 Amplitude( Configuration( apiKey = BuildConfig.AMPLITUDE, context = androidContext(), serverZone…
Web experiment real-time change
Hi everybody, I am running an experiment using the visual editor. I have made some copy changes to our subheading. However, when I land on the testing page, I can see the change from version A to version B happening in real-time. I already have a few colleagues complaining that it does not look good. Is there anything I…
Amplitude Experiments A/B Testing: Redirect from Control to Treatment Not Working
The redirect from Control to Treatment does not work automatically under the following conditions: * When the user lands on the Control page via a link. * When the user lands on the Control page via a window.location.href redirect. * When the user lands on the Control page via a header('Location: <target_page>') redirect…