Solved

s3 Converter Configuration Reference

  • 17 August 2022
  • 2 replies
  • 151 views

I’m trying to import events via parquet files on s3. Trying to keep thing super simple but still having issues:

  1. Should I be able to see a preview here? It must be doing some parsing because the config converter can identify the columns:
  1. Given the 5 columns, I’m using the following conversion but getting errors for the files:
{
"converterConfig": {
"fileType": "parquet",
"compressionType": "none",
"convertToAmplitudeFunc": {
"event_type": "type",
"ts": "timestamp",
"profile_id": "profileId",
"device_id": "deviceId",
"kafka_ts": "kafka_timestamp"
}
},
"keyValidatorConfig": {
"filterPattern": ".*\\.parquet"
}
}

What am I missing/misunderstanding? I also have error logs but I’m having trouble following what they mean.

icon

Best answer by pachung121 19 August 2022, 19:20

View original

2 replies

Userlevel 5
Badge +5

Hi @pachung121

Thank you for writing this post! 

Could you please confirm if your file meets all the requirments listed here: https://www.docs.developers.amplitude.com/data/sources/amazon-s3/?h=s3+import#file-requirements? Would it be possible to share the error logs? It might give us some clues where to look into next. Thanks. 

Hi @Yuanyuan Zhang! I think I figured it out using the error logs.

Looks like the configuration requires event_type and time, with time in ms.

i.e., in the code above, I had to rename the ts key to time and change the value to a timestamp in ms value.

Reply