s3 Converter Configuration Reference

pachung121

I’m trying to import events via parquet files on s3. Trying to keep thing super simple but still having issues:

1. Should I be able to see a preview here? It must be doing some parsing because the config converter can identify the columns:

2. Given the 5 columns, I’m using the following conversion but getting errors for the files:

{<br />    "converterConfig": {<br />        "fileType": "parquet",<br />        "compressionType": "none",<br />        "convertToAmplitudeFunc": {<br />            "event_type": "type",<br />            "ts": "timestamp",<br />            "profile_id": "profileId",<br />            "device_id": "deviceId",<br />            "kafka_ts": "kafka_timestamp"<br />        }<br />    },<br />    "keyValidatorConfig": {<br />        "filterPattern": ".*\\.parquet"<br />    }<br />}

What am I missing/misunderstanding? I also have error logs but I’m having trouble following what they mean.

pachung121

Hi Yuanyuan Zhang! I think I figured it out using the error logs.

Looks like the configuration requires event_type and time, with time in ms.

i.e., in the code above, I had to rename the ts key to time and change the value to a timestamp in ms value.