Github Actions issue with `tox`

[chris-langfield]

As of this morning our tox build workflow is failing on Github Actions.

During attempted execution of the following command:

sh -c 'pip freeze | cut -d@ -f1 | cut -d= -f1 | xargs -n1 pip install -U'

Tox itself fails with the message:

py3.8-dev: failed with sh is not allowed, use allowlist_externals to allow it

(also happening for other environments)

https://github.com/ComputationalCryoEM/ASPIRE-Python/actions/runs/3766988114/jobs/6404083720
https://github.com/ComputationalCryoEM/ASPIRE-Python/actions/runs/3766921945

Test run including a naive addition to the check step that finds the sh installed on the runner:

https://github.com/ComputationalCryoEM/ASPIRE-Python/actions/runs/3766977486

[chris-langfield]

Okay, pretty sure we know what's going on:

From tox 3.x -> tox 4.x there have been changes regarding using commands not already in the tox environment, causing the error above.

We have not caught these until now because we have only been testing using tox 3.x:

• The last GA job to pass (https://github.com/ComputationalCryoEM/ASPIRE-Python/actions/runs/3760725742/jobs/6391860006) was using tox version 3.28. Some kind of change to GA envs in the last 24 hours or so has led the GA environment to stop using this cached version and instead actually use the latest tox, 4.0.16, exposing the error.

[chris-langfield]

What's interesting is that between the last passing GA workflow and the one that failed, the behavior of the Install dependencies step changed. In particular, pip used to pick up a tox==3.28 in addition to latest tox and ended up using that one for some reason:

Old:

Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages (22.3.1)
Collecting tox
  Downloading tox-4.0.16-py3-none-any.whl (143 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.9/143.9 kB 3.5 MB/s eta 0:00:00
Collecting tox-gh-actions
  Downloading tox_gh_actions-2.12.0-py2.py3-none-any.whl (11 kB)

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 12.9 MB/s eta 

[...]

Collecting tox
  Downloading tox-3.28.0-py2.py3-none-any.whl (86 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 26.1 MB/s eta 

[...]

Successfully installed ... tox-3.28.0 tox-gh-actions-2.12.0 ...

New:

Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages (22.3.1)
Collecting tox
  Downloading tox-4.0.16-py3-none-any.whl (143 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.9/143.9 kB 4.3 MB/s eta 0:00:00
Collecting tox-gh-actions
  Downloading tox_gh_actions-3.0.0-py2.py3-none-any.whl (9.7 kB)

[...]

Successfully installed ... tox-4.0.16 tox-gh-actions-3.0.0  ...

[chris-langfield]

Possible culprit: https://github.com/actions/setup-python/releases/tag/v4.4.0 released yesterday

[chris-langfield]

Still curious about what exactly happened there but the patch is in regardless