Skip to main content
Solved

Importing non-events disguised as events from Snowflake causes error.

  • February 21, 2023
  • 3 replies
  • 298 views

If the title makes you scratch your head, fear not, I scratched mine so much it hurts.

 

Also, I am as new as it gets to Amplitude, so I might be missing something really basic.

 

All our marketing spend, coming from a dozen or so partners is stored in Snowflake, where it is also harmonised to our in-house tracking setup. It is not event-like by nature, but I was asked to try to bring it to Amplitude nonetheless, so I tried to make it appear as event-like as possible.

Below my attempt, as simple as pretending all our costs pertain to one user, mr. string-zero.

 

CREATE OR REPLACE VIEW a_db.a_schema.a_view_exposed_to_amplitude
AS 
SELECT
	'0' AS user_id,
	to_timestamp_ntz(a_date_field) AS ts,
    date_part(epoch_millisecond, a_date_field::timestamp) AS epoch,
    'costs_per_cid' AS event_type, 
    object_construct('cid', cost_token, 'clicks',clicks, 'usd_costs', costs, 'impressions', impressions) AS event_properties,
    object_construct() AS user_properties
FROM a_db.a_schema.a_cost_table
;

All went well as I created the connection, the SQL in the connection setup is as simple as it gets

SELECT
  user_id AS "user_id",
  event_type AS "event_type",
  event_properties AS "event_properties",
  epoch AS "time", -- must be in milliseconds since epoch (timestamp)
  user_properties AS "user_properties", -- must be JSON
  ts AS "update_time_column"
FROM a_db.a_schema.a_view_exposed_to_amplitude

But once the initial ingestion job kicked in, it brought in 0 events, just a couple million errors :) 

All looking the same

"error":"events must have either user_id or device_id or both (Missing device or user id)

 

From my perspective the dataset I created has a user_id. Granted, it’s ‘0’ for all of them, but is that the issue or am I missing something more fundamental?

Best answer by Saish Redkar

Hi @Igor Markovic 

Quoting from https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/#device-ids-and-user-ids-minimum-length

“Device IDs and User IDs must be strings with a length of 5 characters or more. If an event contains a device ID or user ID that's too short, the ID value is removed from the event.

Since “0” is considered invalid as a user_id, Amplitude is rejecting these events.

Hope this helps.

View original
Did this topic help you find an answer to your question?

3 replies

Saish Redkar
Expert
Forum|alt.badge.img+10
  • Expert
  • 1380 replies
  • Answer
  • February 21, 2023

Hi @Igor Markovic 

Quoting from https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/#device-ids-and-user-ids-minimum-length

“Device IDs and User IDs must be strings with a length of 5 characters or more. If an event contains a device ID or user ID that's too short, the ID value is removed from the event.

Since “0” is considered invalid as a user_id, Amplitude is rejecting these events.

Hope this helps.


  • Author
  • New Member
  • 1 reply
  • February 22, 2023

Thank you @Saish Redkar this was indeed the issue, and enforcing a longer user id string solved it. 

I am encountering an additional error now, 

User 1z2y3x4w5v is throttled by daily limit.

But this one is easier to understand. I guess there is a configuration, somewhere, that limits the number of events per distinct user over a certain period of time.


Saish Redkar
Expert
Forum|alt.badge.img+10
  • Expert
  • 1380 replies
  • February 22, 2023

Amplitude has an ingestion limit of 500k events per user id on a rolling 24 hour window per project. If that gets breached, you are most likely to get this error.
You can read more here - https://www.docs.developers.amplitude.com/analytics/apis/batch-event-upload-api/#toomanyrequestsfordeviceerror


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings