[dashboard] UI experiments

[geropl]

Description

Since we have segment for tracking we gained the ability to make judgements around dashboard decisions based on data. But since our turnarounds are so slow we very seldomly make use of our "data-driven" superpower 🦸 .

This is an attempt to improve on this: It introduces dashboard-local Experiments. Every user on login gets assigned a random mix of experiments, which may influence the UI behavior (rendering, routing, etc.), based on their respective configured likelihood. The set is stored in localStorage and reported to segment with every interaction.

As a result, we can measure differences in behavior by comparing the experiment group with the rest - "control group" (*) - and make judgement calls on how to improve based on data and not opinions.

Further thoughts:

• we could easily extend this into server by persistinguiExperiments in User.featureFlags, for instance. This has the benefit that our analytic calls on the backend are able to report uiExperiments as well, to offer a complete picture. I went with "dashboard-only" for now to stay simple.

(*) Note that the results are only statistically sound with this approach when we're running a) 1 experiment at a time, or b), they are absolutely independent. Still, I hope are all aware and can use this to gain some valuable insights. 🙂

Related Issue(s)

Fixes #

How to test

• go to https://gpl-ui-experiments.staging.gitpod-dev.com/projects
• see that you have an empty array of experiments set in your local storage:
  

Release Notes

NONE

Documentation

[geropl]

/hold

I wanted to have a discussion about this before merging.

[JanKoehnlein]

/lgtm

Feel free to unhold

[roboquat]

LGTM label has been added.

Git tree hash: 1d47de0fc7d5b009f1570273efabb94e9dc93688

[geropl]

Thx @JanKoehnlein for the review. Waiting for @jakobhero before merging as we already talked about this.

[geropl]

/unhold

to move forward. Still looking forward for your feedback @jakobhero , and happy to fix and adjust in a follow-up PR! 🙏

[laushinka]

Let's do this.

/lgtm

[roboquat]

LGTM label has been added.

Git tree hash: db801324e644e4f307b8d3407329aa79dca88faf

[laushinka]

/approve no-issue

[roboquat]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JanKoehnlein, laushinka

Associated issue requirement bypassed by: laushinka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• components/dashboard/OWNERS [JanKoehnlein,laushinka]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jakobhero]

to move forward. Still looking forward for your feedback @jakobhero , and happy to fix and adjust in a follow-up PR! 🙏

i had a chance to test Experiments (after the fix) with a couple of Segment calls and they look good to me! I appreciate the nested structure as it's going to avoid bloating the structure of event tables once we have many experiments running simultaneously (Segment stringifies nested props before sending them to the data warehouse), but it might cause problems for Mixpanel as it cannot show more than one layer of properties (but it should still be able to store this data). I'm happy to go with this for now and try how difficult it will be to handle this downstream. Thanks for this effort, it's going to be a great addition @geropl !