Skip to content

Fix #9815: fix caching for quick-jobs CI (XDG, cache keys) #9845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 2, 2024
Merged

Fix #9815: fix caching for quick-jobs CI (XDG, cache keys) #9845

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e916cb5
Fix #9815: switch quick-jobs CI to XDG
andreasabel Mar 26, 2024
c209a82
CI quick-jobs: use preinstalled GHC and Cabal
andreasabel Mar 26, 2024
e600087
Makefile: remove dead target 'lexer', use '.PHONY' systematically
andreasabel Mar 26, 2024
56426e4
CI "Meta checks": correct cache key
andreasabel Mar 26, 2024
9a311bd
CI "Meta checks": print Haskell versions
andreasabel Mar 27, 2024
ba6f6ff
CI "Doctest Cabal": daily refresh of cache
andreasabel Mar 27, 2024
5949e3f
CI "Check Field Syntax Reference": correct cache key
andreasabel Mar 27, 2024
947860a
Update generated Cabal/src/Distribution/Simple/Build/Macros/Z.hs
andreasabel Mar 27, 2024
5aa8afd
CI quick-jobs: entirely wipe ghcup directory rights workaround
andreasabel Mar 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 75 additions & 80 deletions .github/workflows/quick-jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,130 +18,125 @@ jobs:
meta:
name: Meta checks
runs-on: ubuntu-latest
env:
cabal_build: >-
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, that's the next level... I'm always worried about putting too much intelligence in the action code. Could this be a Makefile target too? (I see you made some updates to the Makefile).

I know, I'm stretching it, but some specification of what's supposed to happen inside the "quick jobs" workflow may help in future maintenance... Maybe a comment at the top of the file if not a whole section in CONTRIBUTING.md... OTOH it's always impossible to keep it up to date with the code.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am aware that I am breaking the abstraction that the Makefile should offer, but I didn't see a way around it.
What I need here is a cache-key for some specific makefile goal(s), like e.g spdx templates. Putting this computation into the Makefile e.g. as spdx-templates-cache-hash would also be a bit awkward, and, I would need to invoke hashfiles from the Makefile (although this should be solvable).

I think we should document which goals of the Makefiles are in which workflows, so that these are changed in sync. E.g. the lexer goal was removed from the Makefile in #8980 but not from the workflow.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Future work then... Maybe a ticket.

cabal build --builddir=dist-newstyle-meta --project-file=cabal.project.meta
gen-cabal-macros
gen-paths-module
gen-spdx
gen-spdx-exc
# This job is not run in a container, any recent GHC should be fine
steps:
- name: Set PATH
# https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-commands-for-github-actions#adding-a-system-path
# SKIP installation of ghc and cabal and use the preinstalled versions.
# - name: ghcup
# run: |
# ghcup --version
# ghcup config set cache true
# ghcup install ghc recommended
# ghcup set ghc recommended
- name: Haskell versions
run: |
echo "$HOME/.cabal/bin" >> $GITHUB_PATH
- uses: actions/cache@v3
with:
path: ~/.cabal/store
key: linux-store-meta
# See https://github.com/haskell/cabal/pull/8739
- name: Sudo chmod to permit ghcup to update its cache
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo ls -lah /usr/local/.ghcup/cache
sudo mkdir -p /usr/local/.ghcup/cache
sudo ls -lah /usr/local/.ghcup/cache
sudo chown -R $USER /usr/local/.ghcup
sudo chmod -R 777 /usr/local/.ghcup
fi
- name: ghcup
run: |
ghcup --version
ghcup config set cache true
ghcup install ghc recommended
ghcup set ghc recommended
ghc --version
cabal --version
- name: Update Hackage index
run: cabal v2-update
- name: Install alex
run: cabal v2-install alex --constraint='alex ==3.2.7.3'
- uses: actions/checkout@v4
- name: Generate build plan for correct cache key
run: ${{ env.cabal_build }} --dry-run
- name: Restore cached dependencies
uses: actions/cache/restore@v4
id: cache
with:
path: ~/.local/state/cabal
key: linux-store-meta-${{ hashfiles('dist-newstyle-meta/cache/plan.json') }}
restore-keys: linux-store-meta-
- name: Build tools
run: ${{ env.cabal_build }}
- name: Regenerate files
run: |
make -B lexer
make -B spdx
make -B templates
- name: Check that diff is clean
run: |
git status > /dev/null
git diff-files -p --exit-code
- name: Cache dependencies
uses: actions/cache/save@v4
if: always() && steps.cache.outputs.cache-hit != 'true'
with:
path: ~/.local/state/cabal
key: ${{ steps.cache.outputs.cache-primary-key }}

doctest:
name: Doctest Cabal
runs-on: ubuntu-latest
steps:
- name: Set PATH
# It is complicated to get a proper cache key for the dependencies of a package
# (here: doctest) that we just `cabal install`.
# So, as a heuristics we update the cache once per day.
# Updating it with each run would be an alternative, but we a short of cache space,
# and this would generate too many new caches.
- name: Use date as cache key
run: |
echo "$HOME/.cabal/bin" >> $GITHUB_PATH
- uses: actions/cache@v3
echo "DATE=$(date +'%Y-%m-%d')" >> "${GITHUB_ENV}"
- name: Restore cached dependencies
uses: actions/cache/restore@v4
id: cache
with:
path: ~/.cabal/store
key: linux-store-doctest
# See https://github.com/haskell/cabal/pull/8739
- name: Sudo chmod to permit ghcup to update its cache
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo ls -lah /usr/local/.ghcup/cache
sudo mkdir -p /usr/local/.ghcup/cache
sudo ls -lah /usr/local/.ghcup/cache
sudo chown -R $USER /usr/local/.ghcup
sudo chmod -R 777 /usr/local/.ghcup
fi
- name: ghcup
run: |
ghcup --version
ghcup config set cache true
ghcup install ghc --set recommended
ghcup install cabal --set latest
path: ~/.local/state/cabal
key: linux-store-doctest-${{ env.DATE }}
restore-keys: linux-store-doctest
- name: Update Hackage index
run: cabal v2-update
- uses: actions/checkout@v4
- name: Install doctest
run: make doctest-install
- name: Doctest
run: make doctest
- name: Cache dependencies
if: always() && steps.cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
path: ~/.local/state/cabal
key: ${{ steps.cache.outputs.cache-primary-key }}

buildinfo:
name: Check Field Syntax Reference
runs-on: ubuntu-latest
env:
cabal_build: cabal build buildinfo-reference-generator
steps:
- name: Set PATH
run: |
echo "$HOME/.cabal/bin" >> $GITHUB_PATH
- uses: actions/cache@v3
with:
path: ~/.cabal/store
key: linux-store-buildinfo-doc-diff
# See https://github.com/haskell/cabal/pull/8739
- name: Sudo chmod to permit ghcup to update its cache
run: |
if [[ "${{ runner.os }}" == "Linux" ]]; then
sudo ls -lah /usr/local/.ghcup/cache
sudo mkdir -p /usr/local/.ghcup/cache
sudo ls -lah /usr/local/.ghcup/cache
sudo chown -R $USER /usr/local/.ghcup
sudo chmod -R 777 /usr/local/.ghcup
fi
- name: ghcup
run: |
ghcup --version
ghcup config set cache true
ghcup install ghc --set recommended
ghcup install cabal --set latest
- name: Update Hackage index
run: cabal v2-update
- uses: actions/checkout@v4
- name: Generate build plan for correct cache key
run: ${{ env.cabal_build }} --dry-run
- name: Restore cached dependencies
uses: actions/cache/restore@v4
id: cache
with:
path: ~/.local/state/cabal
key: linux-store-buildinfo-doc-diff-${{ hashfiles('dist-newstyle/cache/plan.json') }}
restore-keys: linux-store-buildinfo-doc-diff
- name: Build buildinfo-reference-generator
run: ${{ env.cabal_build }}
- name: Are buildinfo docs up to date?
run: make doc/buildinfo-fields-reference.rst
- name: Cache dependencies
uses: actions/cache/save@v4
if: always() && steps.cache.outputs.cache-hit != 'true'
with:
path: ~/.local/state/cabal
key: ${{ steps.cache.outputs.cache-primary-key }}

release-project:
name: Check Release Project
runs-on: ubuntu-latest
steps:
- name: Set PATH
run: |
echo "$HOME/.cabal/bin" >> $GITHUB_PATH
- name: ghcup
run: |
ghcup --version
ghcup config set cache true
ghcup install ghc --set recommended
ghcup install cabal --set latest
- name: Update Hackage Index
run: cabal v2-update
- uses: actions/checkout@v4
- name: Check Release with Pinned Hackage
run: cabal build all --dry-run --project-file=cabal.project.release
- name: Check Release with Latest Hackage
run: cabal build all --dry-run --project-file=cabal.project.release --index-state="hackage.haskell.org HEAD"

4 changes: 2 additions & 2 deletions Cabal/src/Distribution/Simple/Build/Macros/Z.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
module Distribution.Simple.Build.Macros.Z (render, Z(..), ZPackage (..), ZTool (..)) where
import Distribution.ZinzaPrelude
data Z
= Z {zPackages :: ([ZPackage]),
zTools :: ([ZTool]),
= Z {zPackages :: [ZPackage],
zTools :: [ZTool],
zPackageKey :: String,
zComponentId :: String,
zPackageVersion :: Version,
Expand Down
Loading