Implement analytics on changes to the `.gitpod.yml`

[loujaybee]

Context: This one is important as it will give us data around when a user is updating this file as part of their onboarding journey. e.g. are users updating the file on their first visit? second visit? etc

We would like to know: if / when a user is changing or updating their gitpod.yml.

Ideally, we would gather fine-grained data on which properties the user is updating on the YAML file. This is so we can optimize how and when we present certain features to our users.

Considerations:

• We could possibly implement this via a keystroke event within VS Code, but we'll need to set a reasonable debounce/throttle on the event to ensure we do not send many events.
• We will need to decide which attributes/aspects of the gitpod.yml that are interesting for us to track and why.

Note: If necessary we can break this ticket down, simply knowing that the user has changed the file / added it would deliver value at this stage.

[filiptronicek]

@loujaybee this is a very interesting concept. I wonder if we could do more things on the client by just adding a tailored extension, which would take care of the whole gitpod.yml walkthrough process. This would mean nothing is sent back - good for our servers and also 0 latency. I imagine in this process there won't be too many features which we couldn't implement via an extension, but just a thought 💡!

[loujaybee]

This one is a little tricky, so probably best if @akosyakov is able to collaborate with whoever picks this up to come up with a plan. As with all of the analytics tickets, having something working initially is better than trying to overthink it, and come up with something super comprehensive.

[loujaybee]

There is potentially some tracking added already here in the supervisor [1].

Let's investigate if we have that data, and if those events are working as expected.

[akosyakov]

There is potentially some tracking added already here in the supervisor [1].

@jakobhero Could you check please whether we have config-changed in Segment, but which does not make to BigQuery for some reasons?

[jakobhero]

whether we have config-changed in Segment, but which does not make to BigQuery for some reasons?

Hi @akosyakov, i had another look at the events coming into Segment and config-changed never makes it there. Also, the BigQuery destination is not configured to block any of the events. Are we sending the events to the source with the write key of production trusted?

[akosyakov]

I think it is configured here:

gitpod/components/common-go/analytics/analytics.go

Lines 42 to 46 in 0851301

	switch os.Getenv("GITPOD_ANALYTICS_WRITER") {
	case "log":
	return &logAnalyticsWriter{}
	case "segment":
	return &segmentAnalyticsWriter{Client: segment.New(os.Getenv("GITPOD_ANALYTICS_SEGMENT_KEY"))}

and sent here:

gitpod/components/common-go/analytics/analytics.go

Lines 92 to 102 in 0851301

	func (s *segmentAnalyticsWriter) Track(m TrackMessage) {
	defer recover()
	_ = s.Client.Enqueue(segment.Track{
	AnonymousId: m.AnonymousID,
	UserId: m.UserID,
	Event: m.Event,
	Timestamp: m.Timestamp,
	Properties: m.Properties,
	})
	}

@csweichel Could it be that GITPOD_ANALYTICS_WRITER and GITPOD_ANALYTICS_SEGMENT_KEY not configured in production?

[jakobhero]

Could it be that GITPOD_ANALYTICS_WRITER and GITPOD_ANALYTICS_SEGMENT_KEY not configured in production?

For what it's worth, dashboard and server side analytics calls also depend on the configuration of those as far as i know and they don't seem to be affected.

[akosyakov]

For what it's worth, dashboard and server side analytics calls also depend on the configuration of those as far as i know and they don't seem to be affected.

yes, i see it in each workspace