The GitHub Actions caches don't work for PRs

[seisman]

Description of the problem

See one of the recent PR for an example.

The caches are correctly downloaded, but the "cache" step still runs:

[weiji14]

The download is saving it to '/home/runner/.gmt/server/earth/earth_relief/earth_relief_01d_g.grd.2'? Why the .2?

[weiji14]

Ah, cause we're using wget, Maybe use wget -c so we don't repeat the download.

[seisman]

wget -c should work, but the problem is that the step should not be triggered at all.

[seisman]

This is from the master branch. PR should also work like this one.

[weiji14]

If we use wget -c, the step will run once when the PR is opened (to pull in the master cache) and a PR cache will be created. Subsequent runs in the PR will use that PR cache. See also my note at #475 (comment).

If we don't want to trigger the download step, then we would need to use a 'global' cache, which can be done by removing the -${{ github.ref }}- part and dropping the restore-keys: line here:

pygmt/.github/workflows/ci_tests.yaml

Lines 94 to 95 in f401f85

	key: cache-gmt-${{ runner.os }}-${{ github.ref }}-20200711
	restore-keys: cache-gmt-${{ runner.os }}-refs/heads/master-

[seisman]

So each PR first downloads the master cache, then create its own PR cache. It means that the PR cache can be different from the master cache as the PR cache is always newer.

It may cause problems, because a PR may work with the PR cache (newer) but not with the master cache (older). When the PR merges into master branch, it may break.

[weiji14]

It means that the PR cache can be different from the master cache as the PR cache is always newer.

Yes. We would then realize the cache needs updating on master and refresh the key accordingly. This actually means that PRs are unaffected (i.e. tests on PRs will always have a fresh cache). You can see it as a feature or a bug.

One thing we could look at is to add a date to the cache-key as outlined at https://github.com/actions/cache#creating-a-cache-key. Maybe do a cache that is refreshed 'monthly', but also have an environment key to do a manual bump if needed.

[seisman]

You can see it as a feature or a bug.

I can take it as a feature. I think we can first try if wget -c works.

[seisman]

wget -c doesn't work as we expected.

     Beginning with Wget 1.7, if you use -c on a file which is of equal size as the one on the server, Wget will refuse to download the file and print an
       explanatory message.  The same happens when the file is smaller on the server than locally (presumably because it was changed on the server since your
       last download attempt)---because "continuing" is not meaningful, no download occurs.

       On the other side of the coin, while using -c, any file that's bigger on the server than locally will be considered an incomplete download and only
       "(length(remote) - length(local))" bytes will be downloaded and tacked onto the end of the local file.  This behavior can be desirable in certain
       cases---for instance, you can use wget -c to download just the new portion that's been appended to a data collection or log file.

       However, if the file is bigger on the server because it's been changed, as opposed to just appended to, you'll end up with a garbled file.  Wget has no
       way of verifying that the local file is really a valid prefix of the remote file.  You need to be especially careful of this when using -c in
       conjunction with -r, since every file will be considered as an "incomplete download" candidate.

[weiji14]

Sigh, cache invalidation is hard. I'd like to use rsync but it's for Linux/macOS only. Is there a way to download only if the hash has changed, or do we need to go for a python solution like Requests or Pooch? Also what's the situation with the Europe mirror working but not Oceania?