[META] Regarding the release CI runners: GitHub or the Haskell GitLab?

[Kleidukos]

Due to a host of issues that have been noted through continuous usage of the Haskell GitLab CI by @hasufell, he suggests that we discuss a migration from the GitLab runners to GitHub's own runners.

The proposal (originally here) is transcribed below:

---

Migrating tooling to github / providing github action runners to Haskell infrastructure

Current state

1. GHCup relies on Gitlab CI for testing and releases
2. GHCup maintainers also use Gitlab CI to build darwin M1 stack binaries
3. HLS and Cabal rely on Gitlab CI for releases only
4. projects like bytestring rely on very slow emulation on github actions or other semi-private runners

The problem

1. GHC Gitlab is focussed on providing working GHC binaries and environment for GHC developers.
   That makes other projects or less popular Systems (like FreeBSD) second class citizens.
   The current state of priorities was communicated here
2. All runners on GHC Gitlab are custom and are often maintained by different parties. Getting
   support for broken runners is not always easy. In fact, I have a more direct line to Github staff
   (who are old Gentoo dev colleagues) than to GHC gitlab devops or runner maintainers. On Github,
   we would only maintain a subset of runners ourselves (e.g. darwin M1) instead of all of them.
3. Maintaining two sets of CI configurations for HLS and Cabal is not only a maintenance burden, but also
   means that these projects don't regularly test their PRs and changes against ALL architectures, but
   only do so at release time (which may be too late). All those projects have communicated that they
   don't want to migrate fully to Gitlab last I asked (and actually proposed this).
4. Ideally, all core tooling and core libraries should have access to all relevant architectures that GHC
   supports... on Github actions.

The solution

1. migrate GHCup fully to Github
2. migrate release CI of HLS and Cabal to Github
3. buy our own runners (for e.g. darwin M1) with Opencollective money from GHCup and HLS and with help from HF funding
   and maintain them across our teams
   • darwin M1 prolly at 150$ per month per box (e.g. Macstadium)
   • AWS instances (FreeBSD and aarch64 Linux) for 730 hours (one month) somewhere around 50-150$ per box, depending on instance
   • initial cost may be 500-1000$ per month for a kickstart

HF has indicated (but not promised) that they may help with resources or funding of this solution.

Side effects

• may reduce runner pressure and load on Gitlab CI, freeing up resources for GHC development
• will reduce pressure on devops, because CI issues affect less projects

Remarks

Matthew's concerns on environment and reproducibility

Matthew was concernd about not using gitlab environments (probably the ci docker images) to match the environment where GHC
is built.

Note from Julian: However, we don't do that in HLS CI anyway, at the moment, only partly.

Zubin remarked we can switch CI to gitlab without switching source control

HLS and Cabal could theoretically use gitlab CI exclusively for both testing and releases and then have hooks and
integration with github.

Note from Julian: that would only fix problem 3.

[Mikolaj]

There were scattered bits of discussion of this proposal on #hackage a couple of days ago involving @hasufell, @chreekat, @gbaz and me, but copy-paste from matrix is unreadable, so perhaps that should be screen-shot or, better, somebody on IRC could copy-paste?

Edit: never mind, the participants seem happy to continue without the copied backlog.

[Mikolaj]

Related: #8044

Edit: was wrong link; corrected.

[gbaz]

Thanks for pasting this! While a good start, it doesn't really cover the current status of the new infrastructure which ghcup now uses, nor have an arch-to-arch comparison of which arches are supported on both setups at the moment.

[hasufell]

Thanks for pasting this! While a good start, it doesn't really cover the current status of the new infrastructure which ghcup now uses, nor have an arch-to-arch comparison of which arches are supported on both setups at the moment.

I think comparing for runners doesn't make any sense, because it does not reflect the end-user experience, which is the main issue.

Regardless, I tried to make a comparison. On gitlab, I simply filtered by runner tags. The tags are messy and confusing, so it's possible it doesn't show the correct picture.

	Gitlab	Github	Remarks
x86_64-linux	13	+1000
x86_64-darwin	1	+1000
x86_64-windows	2	+1000
aarch64-linux	14	2
aarch64-darwin	5	2	these can also execute x86_64
x86_64-freebsd	?	+1000	via Cirrus CI, which gitlab could use as well, theoretically

Two main things to note:

• all runners on gitlab are self-hosted
• only two architectures on github are self-hosted
• we're planning to expand the number of github connected runners... this is just the start of the project

[Mikolaj]

Transcript of today's chat with @chreekat and @hasufell:

28/01/2023, 4:03:19 pm - mikolaj: chreekat: do you think you could find a moment at some point and advise re our gitlab config? Most of the builds fail and perhaps you would suggest a trivial fix or even add a line or two and solve all our problems? 
28/01/2023, 4:03:27 pm - mikolaj: it's this pipeline: https://gitlab.haskell.org/haskell/cabal/-/pipelines
28/01/2023, 4:03:42 pm - mikolaj: with this config: https://github.com/haskell/cabal/tree/master/.gitlab
28/01/2023, 4:03:58 pm - mikolaj: it was used for the cabal prerelease just prereleased
28/01/2023, 4:04:05 pm - mikolaj: with mixed effects
28/01/2023, 4:04:18 pm - mikolaj: meaning, it seems what builds, works, but not much builds
28/01/2023, 4:04:34 pm - mikolaj: thanks!
28/01/2023, 4:34:15 pm - chreekat: mikolaj: surveying the failed jobs, my guess is the following:

The two ARM Linux jobs (aarch64 and armv7) fail because of a missing glibc version. An oversight of the Docker image in use?

The i386 Alpine job failed because GHCUp can't find a GHC 9.2.3 that runs on that platform. Is it supported by GHC?

The FreeBSD job failed because the name of a package in Ports changed. The job sets up a fresh FreeBSD in Vagrant every time so it's sensitive to changes in the operating system like that. There may well be other problems there
28/01/2023, 4:34:45 pm - chreekat: That message originally had paragraphs, but I guess IRC happened to it 
28/01/2023, 4:40:22 pm - mikolaj: chreekat: thank you; is there a cheap way to fix those? E.g,. can I copy-paste their respective GHC configs and be done or better link to them so that their updates propagate?
28/01/2023, 4:42:57 pm - maerwald (@maerwald:libera.chat): use 9.2.2
28/01/2023, 4:44:07 pm - maerwald (@maerwald:libera.chat): and then complain to GHC HQ for bad platform coverage
28/01/2023, 4:47:58 pm - mikolaj: maerwald: thank you
28/01/2023, 4:48:09 pm - mikolaj: chreekat: do you mind if I copy this to https://github.com/haskell/cabal/issues/8674 ?
28/01/2023, 4:48:51 pm - mikolaj: we have to decide whether to completely move away from gitlab or not, but it's really hard to decide without any person permanently responsible for our CI
28/01/2023, 4:49:00 pm - chreekat: mikolaj: sure, ping me as well so I get an email
28/01/2023, 4:49:14 pm - chreekat: I will have to look into this more on Monday :)
28/01/2023, 4:49:28 pm - mikolaj: thank you so much
28/01/2023, 4:50:13 pm - mikolaj: (re decision: because we'd be deciding for that volunteer that we hope to find, without that volunteer having any say, so not much point)
28/01/2023, 4:50:17 pm - maerwald (@maerwald:libera.chat): Man of Letters: my offer doesn't stand indefinitely btw. Once I'm done with HLS CI, I probably won't have capacity or motivation for this anymore.
28/01/2023, 4:50:30 pm - mikolaj: understood
28/01/2023, 4:54:27 pm - mikolaj: BTW, this was all spurred by https://discourse.haskell.org/t/cabal-3-10-last-call-for-contributions/5322/6?u=mikolaj  --- we need all platforms even for prereleases [edit: and nightlies once we have them]
28/01/2023, 4:54:31 pm - maerwald (@maerwald:libera.chat): Mac runners are a *major* headache btw. So one of the primary goals is to have as few platforms as possible self-hosted. There's no reason to do that other than "github doesn't provide it"

[Bodigrim]

I don't see any good reason to stay on GitLab. GHC runners can barely cope with GHC workload itself, and despite all efforts I am yet to see an MR I do not have to babysit and rerun flaky jobs. Until recently the major benefit of GitLab was an access to ARM machines, but now thanks to @hasufell and @angerman github.org/haskell has them as well.

[hasufell]

Is this moving forward into any direction? Does cabal have any governing structure to make decisions?

All information is available, please make a decision.

[chreekat]

To share what I feel about the matter:

1. I was asked to see what it takes to run Cabal release pipelines on GitLab. I was able to double the number of platforms built for Cabal 3.8.1. My aim was to build for every platform that GHC 9.6.1 is built for, and Centos7 is the only remaining holdout.

2. While GHC CI is still unreliable, that's mostly because of GHC. I don't think GitLab is in that bad of a state.

3. Recall that those ARM runners that are now available on github.org/haskell are the same runners used on GitLab, and thus have the same level of reliability.

4. My professional intuition is that building Cabal releases on the same platforms as GHC releases is valuable.

5. While Microsoft is a major sponsor of the Haskell Foundation, they're not paying us to use GitHub, and I have strategic concerns about relying heavily on systems outside our control when alternatives exist.

6. Nonetheless, it looks like I'm the only person in the room with the ability and interest in supporting Cabal's CI on GitLab. Due to my bigger priorities, I can only promise ~1 day of work per release right now. Compared to the amount and quality of effort being put into GitHub CI, I am not sure it's the right decision to stick with GitLab.

I, also, am curious to know what decision is made and who has the power to make it.

[hasufell]

@chreekat GitHub provides virtually unlimited runners for most of the platforms. This allowed me in HLS to scale up the bindist builds for 5+ distros and 3+ versions of those and that PER GHC. In parallel.

One such build on gitlab would lock up all the existing Linux runners (around 50 I believe).

The point is to reduce the amount of self hosted runners as much as possible.

[Mikolaj]

@hasufell: this needs to be discussed (thank you for taking part). We are busy with 3.10 release. The governing structure is cabal devs (gathering fortnightly, e.g., today, details on #hackage) and the github repo maintainers and the release manager makes the final decision for the particular release.

[hasufell]

The governing structure is cabal devs (gathering fortnightly, e.g., today, details on #hackage)

What's the list of cabal devs?

and the github repo maintainers

Who is that? I happen to be org admin, does that make me repo maintainer?

and the release manager

Who is that and how are they appointed? Just volunteers raising their hand?

[Kleidukos]

@hasufell The following teams are related to cabal:

• @haskell/cabal-contributors
• @haskell/cabal-core
  And @Mikolaj is lonely as a @haskell/cabal-triagers but I don't know if that one shouldn't be removed. :)