Use GHC CI infrastructure to produce release artifacts.

[phadej]

The current CI is good for daily development, but GHC CI has wider platform coverage, which would be great for providing binary releases.

Yet, the setup have to be made.

[bgamari]

Anyone interested should get in touch with me. I'm happy to advise.

[samuela]

Hey @bgamari I'm interested in taking a stab at this! What needs to be done?

[bgamari]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

I have put up a quick experiment here. Feel free to use it as a starting point. The remaining work is the following:

• fix the logic (i.e. the find command in my experiment) for locating and copying the executable
• modify the Linux job in my experiment to rather use the CentOS 7 build instead of Debian 9 (this ensures that we build against an old glibc, allowing us to use the same executable across all glibc-based distributions)
• Possibly freeze the Hackage index date?
• Add jobs for (using the GHC .gitlab-ci.yml as a template):
  • AArch64 Linux
  • ARMv7 Linux
  • Darwin (macOS)
  • Windows
  • x86-64 FreeBSD
  • x86-64 Alpine Linux (musl)
  • any other platforms that I have forgotten
• add --enable-split-sections to the cabal command line
• build and package the documentation?
• possibly package the build products into a tarball?
• cache the Cabal store?

The platforms which don't use Docker are a bit tricky; on these platforms (Windows, Darwin, FreeBSD) GHC's .gitlab/ci.sh script is responsible for installing a GHC binary distribution. You'll need to replicate this functionality.

I would be happy to set up a chat to discuss any details if this would be helpful. Alternatively, find me in #ghc on irc.freenode.net.

Thanks for volunteering!

[hasufell]

fix the logic (i.e. the find command in my experiment) for locating and copying the executable

    - mkdir -p out
    - cabal update
    - cabal new-install --install-method=copy --overwrite-policy=always --installdir=$(pwd)/out cabal-install

If you want to use new-build and then find the exe, do:

    - mkdir -p out
    - cabal update
    - cabal new-build
    - cp "$(cabal new-exec --verbose=0 --offline sh -- -c 'command -v cabal')" out/cabal

[bgamari]

Thanks @hasufell! I adopted the v2-install approach in my branch.

[samuela]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

Thanks for all the info! Since this project is hosted on GitHub whereas GHC is hosted on GitLab, we'll need to do something different than .gitlab-ci.yml, right? It looks as though cabal is currently using GitHub Actions and Travis. So is the ultimate goal to do these builds on cabal GH Actions, cabal Travis, or ghc gitlab CI?

[samuela]

@bgamari Oh, I think I may have misunderstood! So your fork lives on gitlab and has a .gitlab-ci.yml file. So which of these CI systems is/should be producing the bindists?

[bgamari]

@samuela the plan here is to use GHC's GitLab CI to prepare the cabal-install binary distributions since then we can easily piggy-back on the existing work put into GHC's infrastructure (not to mention, neither GitHub Actions nor Travis support the range of platforms supported by GHC).

[samuela]

Ok, so in order to start adapting https://gitlab.haskell.org/bgamari/cabal-build-test/-/commit/50b81bcf1208dbd7ea566d0a2e0fa428dbdd4d00 to work with centos, we need a new docker image in ci-images. Here's an attempt: https://gitlab.haskell.org/samuela/ci-images/-/commit/b50c3c8049321d8542e7bc29bb03c4e815765621 but it fails with

Step 10/25 : RUN curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx
 ---> Running in 1ee41616a1d9
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   178  100   178    0     0    217      0 --:--:-- --:--:-- --:--:--   217
xz: (stdin): File format not recognized
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/bash -o pipefail -c curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx' returned a non-zero code: 2

So we'll need ghc-8.8.3-armv7-centos7-linux.tar.xz first...

[hasufell]

There is armv7 for https://downloads.haskell.org/~ghc/8.2.2/ (debian 8). You could try to bootstrap from there.

[bgamari]

@samuela I would say let's stick with building on Debian 9 on ARMv7 and AArch64. We don't currently build CentOS bindists on these platforms; it's something that ideally we would ideally do but this will take time and most people tend to use Debian-based distributions on these platforms anyways.

[samuela]

Ok, I've started some work on this branch. Here are the pipeline results. So far armv7-linux-deb9 and aarch64-linux-deb9 seem to be working but x86_64-freebsd failed.

[hasufell]

@samuela nice, a few comments

1. I think the FreeBSD runner doesn't have a cabal-3.x yet. I also manually update it in the ghcup CI. You can download FreeBSD binaries here: https://hasufell.de/d/d3e215db133e4fcaa61e/ (I signed the hash file). @hvr hasn't responded to my request of adding those tarballs to downloads.haskell.org yet.

2. I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage? @phadej there seems to be no cabal mirror on gitlab.haskell.org yet?

[samuela]

I think the FreeBSD runner doesn't have a cabal-3.x yet. I also manually update it in the ghcup CI. You can download FreeBSD binaries here: https://hasufell.de/d/d3e215db133e4fcaa61e/ (I signed the hash file). @hvr hasn't responded to my request of adding those tarballs to downloads.haskell.org yet.

Hmm, this is unfortunate. Ideally we could just update the Docker image? I guess that would require having the downloads up on downloads.haskell.org? For the sake of this issue should we just go ahead and skip freebsd? Or will we need all the platforms in order to switch over?

I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage?

I'm a little confused where this CI is supposed to live. We've got a bunch of different repos flying around across GitHub and GitLab! I'm happy to put them wherever. I interpreted this comment to mean that they should be added to GHC's GitLab repo, but perhaps I misunderstood?

[bgamari]

Hmm, this is unfortunate. Ideally we could just update the Docker image?

@samuela, right, I'm not opposed to updating the Docker image to pull FreeBSD binaries from @hasufell's server for now.

Or will we need all the platforms in order to switch over?

We don't, although it would be nice to do it in one swoop.

I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage?

No, https://gitlab.haskell.org/bgamari/cabal-build-test is a fork of this repository and builds Cabal and cabal-install from said repository.

I'm a little confused where this CI is supposed to live. We've got a bunch of different repos flying around across GitHub and GitLab! I'm happy to put them wherever. I interpreted this comment to mean that they should be added to GHC's GitLab repo, but perhaps I misunderstood?

The idea is that we add a .gitlab-ci.yml to this repository (as I did in my experiment). https://gitlab.haskell.org/ can then host a mirror https://github.com/haskell/cabal which CI will run on. The binary artifacts that are produced from this CI can then be used as release artifacts.

[samuela]

So a lot of magic is happening in GHC's .gitlab/ci.sh. It seems like a bit of code smell to copy-pasta the whole thing into cabal to get builds working on non-Docker platforms. A few options off the top of my head:

• Just copy-paste the script into cabal.
• Download .gitlab/ci.sh off of the GHC repo, and then run it from there. Has the disadvantage that GHC can unknowingly break cabal CI.
• Move .gitlab/ci.sh into ci-images or another independent repo.

@bgamari @hasufell wdyt?

[hasufell]

@samuela I think the idea was to start off where @bgamari left in his experimental branch: https://gitlab.haskell.org/bgamari/cabal-build-test/-/blob/build-test/.gitlab-ci.yml

The GHC one is certainly overkill.

[bgamari]

Precisely. Very little of the complexity (maybe none?) in the GHC script is needed by Cabal, which is a standard Haskell package. As suggested by @hasufell, starting with my experiment is probably the best way to begin.

[23Skidoo]

This would be pretty great.

[samuela]

Ok, I've been working on this branch and specifically .gitlab-ci.yml. The following architectures are working so far:

• aarch64-linux-deb9
• armv7-linux-deb9
• x86_64-linux (deb9)
• x86_64-freebsd
• x86_64-linux-alpine
• x86_64-darwin

I think the only thing missing now is windows...

[phadej]

@Mistuke do you have your cabal building scripts somewhere publicly available?

[phadej]

Also @hvr, could you share the scripts you use.

The setup samuela made seems simple enough, but it misses making actual tarball and providing license report + plan.json which IIRC we want to provide along (or inside) the tarballs.

[Mistuke]

@phadej my CI runs a version of this

> cd "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c"
> del /F /S /Q "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> mkdir "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin" 
> cabal v1-install alex --bindir="C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> cabal v1-install happy --bindir="C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> cabal --version
> alex --version
> happy --version 
> cabal  update
> cabal  --store-dir="E:\CabalStore" configure --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" all --allow-older=cabal-testsuite:Cabal
> cabal  --store-dir="E:\CabalStore" build --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" cabal
> cabal  --store-dir="E:\CabalStore" build --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" cabal-install
> bash -c "find dist-newstyle/build/x86_64-windows/ghc-8.4.3/ -iname cabal.exe -print0 | xargs -I{} -0 cp -v {} ."
> E:\GHC\tools\cv2pdb.exe cabal.exe
> strip -s cabal.exe
> E:\GHC\tools\upx.exe -9 cabal.exe
> gpg --output cabal.exe.sig --detach-sig cabal.exe
> E:\GHC\tools\7zip\7z.exe a -bt -y cabal.zip cabal.exe cabal.exe.pdb cabal.exe.sig

[samuela]

I'm not at all familiar with windows, and I also don't have a windows machine to test on. But if anyone else is familiar/can contribute then I think we'd be all done here! Is there anything else we'd be missing at that point?

[bgamari]

I can handle Windows. Thanks @samuela!

[hasufell]

Can we trigger those release builds for the previous 2-3 versions?

[phadej]

Please hold your horses. In other words do not use these builds for anything yet. Let's make the setup work first and then propagate it. I don't want to see yet another feature left half-way because "it seems to work".

[hasufell]

Let's make the setup work first and then propagate it.

What's left to do? :)

[samuela]

• fix the logic (i.e. the find command in my experiment) for locating and copying the executable

All done here.

• modify the Linux job in my experiment to rather use the CentOS 7 build instead of Debian 9 (this ensures that we build against an old glibc, allowing us to use the same executable across all glibc-based distributions)

Out of scope per #6616 (comment).

• Possibly freeze the Hackage index date?

Unclear whether this is required?

• Add jobs for (using the GHC .gitlab-ci.yml as a template):

  • AArch64 Linux
  • ARMv7 Linux
  • Darwin (macOS)
  • Windows
  • x86-64 FreeBSD
  • x86-64 Alpine Linux (musl)
  • any other platforms that I have forgotten

These are all done except for windows.

• add --enable-split-sections to the cabal command line

Done in https://gitlab.haskell.org/samuela/cabal-build-test/-/commit/1964e1c8.

• build and package the documentation?
• possibly package the build products into a tarball?
• cache the Cabal store?

Not sure which of these are hard requirements. Right now, we're not doing any of them AFAIK.

[hasufell]

@phadej so what is blocking further steps?

[phadej]

I'll take a look.

[phadej]

I'm having really a bad day, but I say it here anyways. I don't see how we can officially support FreeBSD or ARM as e.g. Windows eats occasionally days of work.

In other words, it would be simpler and clearer for us to provide binaries for GHC's Tier 1 platforms only. We could have scripts for say FreeBSD or/and ARM in place, but if those fail, it won't hold release process or change any priorities.

EDIT: I'd rather support FreeBSD and ARM than Windows, but if one have to choose (and we seems to be forced to) than Windows wins.

[Mistuke]

I'm having really a bad day, but I say it here anyways. I don't see how we can officially support FreeBSD or ARM as e.g. Windows eats occasionally days of work.

I'm sorry you're having a bad day, I know what those are like. But I wonder why you think Windows is so much harder. I build cabal nightly in different configuration and have done so for 2 years now with the only time I had to make any changes was when I hit a GHC bug recently.

So I wonder if Windows isn't more difficult because of the CI setup, and not Windows itself.

In other words, it would be simpler and clearer for us to provide binaries for GHC's Tier 1 platforms only. We could have scripts for say FreeBSD or/and ARM in place, but if those fail, it won't hold release process or change any priorities.

EDIT: I'd rather support FreeBSD and ARM than Windows, but if one have to choose (and we seems to be forced to) than Windows wins.

What exactly is the thing that is increasing your maintenance burden. I'd be happy to go over the configuration with you instead of having you stab around in the dark. You know where to find me.

[phadej]

What exactly is the thing that is increasing your maintenance burden. I'd be happy to go over the configuration with you instead of having you stab around in the dark. You know where to find me.

This CI issue is just a reminder that heterogeneous targets are tricky. There are occasionally issues with Windows, like this CI or segfaulting (https://gitlab.haskell.org/ghc/ghc/issues/17926). We don't have even thought about signing cabal binaries for macOS (which I guess will be required), I hope that whatever GHC will do could help cabal as well there.

So adding ARM and/or FreeBSD to the set which one should keep working all the time doesn't feel good now. I do think that they will have less unique issues then mac or win, but I don't want to promise anything.

I'm also really worried what the rumored ARM based mac would mean to all of us.

[phadej]

And to clarify even more. I somehow know Linux, and understand mac. So can reasonably deal with most issues on those. If it weren't you @Mistuke, than Windows story would be a lot worse. You deserve a lot of thanks for helping us.

But I have no idea about FreeBSD or OpenBSD, and what issues ARM might bring to the table. It's an unknown territory with no "support network". I guess if someone who would champion these areas (and don't want to change or rewrite all the rest of cabal because of that), then it would be great.

[hasufell]

@phadej FreeBSD support should really be no issue. I can spin up a FreeBSD vm in a few seconds, do tests, build stuff. In fact, the first few ghcup versions were all manually built on FreeBSD with manually built cabals. I haven't experienced major issues. The most problematic OS is macOS and it isn't easy to get your hands on a VM.

[samuela]

@phadej I'm sorry to hear this is turning out to be so stressful. I understand first-hand that maintaining an OSS project involves a lot of work, much of it thankless. I think we should brainstorm ways to make this all easier to maintain.

It sounds to me like there are two questions at play here: (a) How much can/should be put into CI? and (b) What platforms should be supported and at what "support tiers"?

In terms of (a), I claim that having a CI should greatly reduce the workload and stress required in publishing a release and maintaining stability across a variety of platforms. Codifying all of this ad hoc logic into CI means that we have a single, reproducible, and version-controlled source-of-truth for the build and release process across all the platforms. This way, eg. @Mistuke could contribute with his unique Windows knowledge, and others could contribute to platforms of their own interest/ability.

Now the question (b) of which platforms should be supported seems slightly thornier. One significant advantage of having a CI in place is that then the onus of support for each platform is shifted to committers when submitting PRs, and off of maintainers. Ie, platform-specific issues are caught on each build/test triggered on every PR. The easiest strategy in my mind would be to set up CI with the platforms listed in #6616 (comment), and then if certain platforms prove to be too onerous then a discussion can be had about whether or not to allow failures for those targets in CI.

[phadej]

It sounds to me like there are two questions at play here: (a) How much can/should be put into CI? and (b) What platforms should be supported and at what "support tiers"?

Please send an email to [email protected] to discuss that. I have put way too much time (month of work in the beginning of this year) into current CI setup which works reliably enough to actually be useful but that might be my familiarity bias. GitHub Actions don't offer not-self-run ARM nor FreeBSD runners, so adding them in current setup is not possible. Changing CI is not what I want to be part of. In principle it should be easy, "run ./validate.hs".

I promise to stay out of that discussion.

[phadej]

To add

I'm not completely happy with GitHub actions setup either.

• You cannot restart individual jobs, only all jobs in a workflow at once.
  • Fortunately our tests aren't as fluky as it used to be
• The continuonsly updating base images of macOS and Windows.

But on the other hand

• It's free
• Setup is relatively simple
  • Accessible for modification to all Cabal contributors
• It's a lot faster than previous travis setup
  • This is something I'd like not to give up. I can trust jobs to complete in 1-2 hours since creation.
• There weren't hardware issues during half year we have used it
  • There were occasional software "events non delivered" outrages on GitHub, but those affected all CI providers.

[samuela]

Well ARM support could be accomplished through self-hosted runners, which is actually the same as what the GitLab CI does. We could probably scavenge a Raspberry Pi or two to run those. I personally don't care about FreeBSD, but that can be run in docker on an Ubuntu runner, so it's not out of the question. Should we just forge ahead with GitHub Actions then? The work done here can be migrated to use GitHub Actions?

[samuela]

The Raspberry Pi folks seem to be moving towards 64 bit builds, which makes armv7 less important in my mind. Of course aarch64 is still very much needed, and many devices will be running armv7 for a while to come.

[gbaz]

I think this is now done?

[phadej]

@gbaz, I wouldn't say so. https://github.com/haskell/cabal/blob/6a8c3f80173b71436386d9724b64eb67ccb1cf40/.gitlab/ci.sh doesn't use https://github.com/haskell/cabal/blob/6a8c3f80173b71436386d9724b64eb67ccb1cf40/release.py which is however used in https://github.com/haskell/cabal/blob/master/.github/workflows/artifacts.yml

So there is two ways to produce artifacts...

I'd consider this done when there is only one way to build release artifacts used.

[hasufell]

@gbaz, I wouldn't say so. https://github.com/haskell/cabal/blob/6a8c3f80173b71436386d9724b64eb67ccb1cf40/.gitlab/ci.sh doesn't use https://github.com/haskell/cabal/blob/6a8c3f80173b71436386d9724b64eb67ccb1cf40/release.py which is however used in https://github.com/haskell/cabal/blob/master/.github/workflows/artifacts.yml

So there is two ways to produce artifacts...

I'd consider this done when there is only one way to build release artifacts used.

Does release.py do anything special?

[phadej]

Does release.py do anything special?

Not really, I guess.

[Mikolaj]

Does release.py do anything special?

Not really, I guess.

Seriously, what kind of passive-aggressive conversation is that? @hasufell, release.py is a lot of, until very recently, actively developed code, of course it does a lot of special things. @phadej: could you please say something less witty, but more useful?

[phadej]

@Mikolaj release.py doesn't do anything special, honestly.

It takes some configuration via command line interface

cabal/release.py

Lines 282 to 290 in 6a8c3f8

	parser.add_argument('-w', '--with-compiler', type=str, default='ghc', help='path to GHC')
	parser.add_argument('-C', '--with-cabal', type=str, default='cabal', help='path to cabal-install')
	parser.add_argument('-i', '--index-state', type=str, default=DEFAULT_INDEXSTATE, help='index state of Hackage to use')
	parser.add_argument('--enable-static-executable', '--disable-static-executable', dest='static', nargs=0, default=False, action=EnableDisable, help='Statically link cabal executable')
	parser.add_argument('--enable-ofd-locking', '--disable-ofd-locking', dest='ofd_locking', nargs=0, default=True, action=EnableDisable, help='OFD locking (lukko)')
	parser.add_argument('--tarlib', dest='tarlib', required=True, metavar='LIBTAR', help='path to Cabal-version.tar.gz')
	parser.add_argument('--tarsolver', dest='tarsolver', required=True, metavar='SOLVERTAR', help='path to cabal-install-solver-version.tar.gz')
	parser.add_argument('--tarexe', dest='tarexe', required=True, metavar='EXETAR', help='path to cabal-install-version.tar.gz')
	parser.add_argument('--builddir', dest='builddir', type=str, default='_build', help='build directory')

, (tries it best to) setup clean environment, but at the end it's similar cabal v2-install as in gitlab setup; https://github.com/haskell/cabal/blob/6a8c3f80173b71436386d9724b64eb67ccb1cf40/.gitlab/ci.sh#L33-l41

one benefit of release.py is that it can be run on any machine with tools installed (the clean env bit), but you may not value that, and it's fine.

[Mikolaj]

@phadej: but it has some tested and benchmarked options that have been accumulated over time, right? E.g,

cabal/release.py

Line 169 in 6a8c3f8

ghc-options: -fexpose-all-unfoldings -fspecialise-aggressively

Or are such recommended options easily available from somewhere else than this script? Or is that the only example of special configuration that cabal install does not set up by default?

[phadej]

there is https://github.com/haskell/cabal/blob/master/cabal.project.release, which however should probably be renamed to cabal.project.bootstrap as it's used for that purpose only atm. And IMO it would be good to have a bootstrap specific cabal.project, as it may be slightly or a lot different then release / other projects, e.g. that optimization is probably not important for bootstrapping (in fact, i'm not sure we need to optimize Cabal at all in boostrapped cabal-install - it depends on what bootstrapped cabal-install is used for)

[jneira]

for better or worse release.py does not exist anymore, neither https://github.com/haskell/cabal/blob/master/.github/workflows/artifacts.yml.
gitlab was used for the last release and likely will be used for the next ones, with the config in https://github.com/haskell/cabal/blob/master/.gitlab-ci.yml and https://github.com/haskell/cabal/tree/master/.gitlab
so i think taking in account the original issue title and this comment from @phadej:

I'd consider this done when there is only one way to build release artifacts used.

, it would be done

[jneira]

@Mikolaj there were some pending cuestions like the missing flags you mentioned here: #6616 (comment)
do you think it would need a new issue to track it?

I really appreciate @phadej thoughts on ci exposed here, like

Please send an email to [email protected] to discuss that. I have put way too much time (month of work in the beginning of this year) into current CI setup which works reliably enough to actually be useful but that might be my familiarity bias. GitHub Actions don't offer not-self-run ARM nor FreeBSD runners, so adding them in current setup is not possible. Changing CI is not what I want to be part of. In principle it should be easy, "run ./validate.hs".

And there are other open questions about actual ci, like gitlab artifacts not being tested, but i think they would need new issues (if they do not exists already)

[Mikolaj]

@Mikolaj there were some pending cuestions like the missing flags you mentioned here: #6616 (comment)
do you think it would need a new issue to track it?

Let's let it rest. I was booed into closing a ticket about testing the benefit of these options last time I tried, and the partial test results were strange --- I couldn't apply the options to the actual compilation of the cabal executable despite including them in the used cabal.project. So that would require some extra time, which the collective maintainer is not inclined to donate (but we took a similarly large time arguing why the potential measurements can't show anything useful, if completed).

Regarding scavenging anything of value from validate.hs or release.py or other removed scripts or @phadej skull, that's very worthwhile IMHO, but not trivial. :)