Experiment Assignment and Exposure discrepancies

Martin E

We have recently integrated Amplitude analytics and have more recently been testing Amplitude Experiments on our website. For our first test experiment, the ratio between assignment and exposure is roughly 2:1, so there is double the amount of assignments as there is exposures and we don’t know why that is?

We also checked in Pathfinder and could see the following problem: Sometimes exposures seem to occur before the assignment event, is that an issue?

 

 

We included the Amplitude browser and experiment sdks via the npm modules @amplitude/experiment-js-client and @amplitude/analytics-browser We then initialize the experiment sdk like so:

  private _experiment = Experiment.initializeWithAmplitudeAnalytics(<br />    environment.amplitudeDeploymentKey,<br />    { fetchOnStart: true }<br />  );<br />

as well as the browser SDK:

amplitude.init(environment.amplitudeApiKey, { defaultTracking: true });

and here is the code that checks the variant with additional experiment.start() call just in case:

  hasVariantValue(<br />    experimentFlag: string,<br />    variantValue: string<br />  ): Promise<boolean> {<br />    return new Promise((resolve, reject) => {<br />      this.initExperiments().then(<br />        () => {<br />          const variant = this._experiment.variant(experimentFlag);<br />          resolve(variant.value === variantValue);<br />        },<br />        (err) => {<br />          reject(err);<br />        }<br />      );<br />    });<br />  }<br /><br />  initExperiments(): Promise<boolean> {<br />    if (this._initialized) {<br />      return Promise.resolve(true);<br />    }<br /><br />    return new Promise((resolve, reject) => {<br />      this._experiment.start().then(<br />        () => {<br />          this._initialized = true;<br />          resolve(true);<br />        },<br />        (err) => {<br />          reject(err);<br />        }<br />      );<br />    });<br />  }<br />

and is then used like so:

const amplitude = AmplitudeService.getInstance();  <br />amplitude<br />    .hasVariantValue('alt-title-searchbox', 'treatment')<br />    .then((active) => {<br />      console.debug('[amplitude] split test active = ', active);<br />      if (active) { /* split test active */ }<br />     });<br />

We do not set any additional device or user ids.