Skip to main content

I’m trying to import events via parquet files on s3. Trying to keep thing super simple but still having issues:

  1. Should I be able to see a preview here? It must be doing some parsing because the config converter can identify the columns:
  1. Given the 5 columns, I’m using the following conversion but getting errors for the files:
{
    "converterConfig": {
        "fileType": "parquet",
        "compressionType": "none",
        "convertToAmplitudeFunc": {
            "event_type": "type",
            "ts": "timestamp",
            "profile_id": "profileId",
            "device_id": "deviceId",
            "kafka_ts": "kafka_timestamp"
        }
    },
    "keyValidatorConfig": {
        "filterPattern": ".*\\.parquet"
    }
}

What am I missing/misunderstanding? I also have error logs but I’m having trouble following what they mean.

Hi @pachung121, 

Thank you for writing this post! 

Could you please confirm if your file meets all the requirments listed here: https://www.docs.developers.amplitude.com/data/sources/amazon-s3/?h=s3+import#file-requirements? Would it be possible to share the error logs? It might give us some clues where to look into next. Thanks. 

Hi @Yuanyuan Zhang! I think I figured it out using the error logs.

Looks like the configuration requires event_type and time, with time in ms.

i.e., in the code above, I had to rename the ts key to time and change the value to a timestamp in ms value.

Reply


Terms & ConditionsCookie settings