Solved

Importing non-events disguised as events from Snowflake causes error.

  • 21 February 2023
  • 3 replies
  • 95 views

If the title makes you scratch your head, fear not, I scratched mine so much it hurts.

 

Also, I am as new as it gets to Amplitude, so I might be missing something really basic.

 

All our marketing spend, coming from a dozen or so partners is stored in Snowflake, where it is also harmonised to our in-house tracking setup. It is not event-like by nature, but I was asked to try to bring it to Amplitude nonetheless, so I tried to make it appear as event-like as possible.

Below my attempt, as simple as pretending all our costs pertain to one user, mr. string-zero.

 

CREATE OR REPLACE VIEW a_db.a_schema.a_view_exposed_to_amplitude
AS
SELECT
'0' AS user_id,
to_timestamp_ntz(a_date_field) AS ts,
date_part(epoch_millisecond, a_date_field::timestamp) AS epoch,
'costs_per_cid' AS event_type,
object_construct('cid', cost_token, 'clicks',clicks, 'usd_costs', costs, 'impressions', impressions) AS event_properties,
object_construct() AS user_properties
FROM a_db.a_schema.a_cost_table
;

All went well as I created the connection, the SQL in the connection setup is as simple as it gets

SELECT
user_id AS "user_id",
event_type AS "event_type",
event_properties AS "event_properties",
epoch AS "time", -- must be in milliseconds since epoch (timestamp)
user_properties AS "user_properties", -- must be JSON
ts AS "update_time_column"
FROM a_db.a_schema.a_view_exposed_to_amplitude

But once the initial ingestion job kicked in, it brought in 0 events, just a couple million errors :) 

All looking the same

"error":"events must have either user_id or device_id or both (Missing device or user id)

 

From my perspective the dataset I created has a user_id. Granted, it’s ‘0’ for all of them, but is that the issue or am I missing something more fundamental?

icon

Best answer by Saish Redkar 21 February 2023, 18:09

View original

3 replies

Userlevel 7
Badge +10

Hi @Igor Markovic 

Quoting from https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/#device-ids-and-user-ids-minimum-length

“Device IDs and User IDs must be strings with a length of 5 characters or more. If an event contains a device ID or user ID that's too short, the ID value is removed from the event.

Since “0” is considered invalid as a user_id, Amplitude is rejecting these events.

Hope this helps.

Thank you @Saish Redkar this was indeed the issue, and enforcing a longer user id string solved it. 

I am encountering an additional error now, 

User 1z2y3x4w5v is throttled by daily limit.

But this one is easier to understand. I guess there is a configuration, somewhere, that limits the number of events per distinct user over a certain period of time.

Userlevel 7
Badge +10

Amplitude has an ingestion limit of 500k events per user id on a rolling 24 hour window per project. If that gets breached, you are most likely to get this error.
You can read more here - https://www.docs.developers.amplitude.com/analytics/apis/batch-event-upload-api/#toomanyrequestsfordeviceerror

Reply