Cannot create directory ‘/opt/hostedtoolcache’ after recent release

[kenorb]

Description:

The following error start happening after the recent action release:

2022-07-12T14:12:47.7158848Z ##[group]Run actions/setup-python@v4
2022-07-12T14:12:47.7159230Z with:
2022-07-12T14:12:47.7159527Z   python-version: 3.x
2022-07-12T14:12:47.7160059Z   token: ***
2022-07-12T14:12:47.7160394Z   update-environment: true
2022-07-12T14:12:47.7162682Z ##[endgroup]
2022-07-12T14:12:48.0166854Z Version 3.x was not found in the local cache
2022-07-12T14:12:48.3072207Z Version 3.x is available for downloading
2022-07-12T14:12:48.3074130Z Download from "https://github.com/actions/python-versions/releases/download/3.10.5-2452607758/python-3.10.5-linux-20.04-x64.tar.gz"
2022-07-12T14:12:49.1279090Z Extract downloaded archive
2022-07-12T14:12:49.1380950Z [command]/usr/bin/tar xz --warning=no-unknown-keyword -C /actions-runner/_work/_temp/2d83bfeb-a252-4083-95bf-6f30cffc5507 -f /actions-runner/_work/_temp/c2619dbf-23ad-46f0-9b07-baf06258c9c8
2022-07-12T14:12:52.2679935Z Execute installation script
2022-07-12T14:12:52.2887302Z Check if Python hostedtoolcache folder exist...
2022-07-12T14:12:52.2888180Z Creating Python hostedtoolcache folder...
2022-07-12T14:12:52.2921018Z ##[error]mkdir:
2022-07-12T14:12:52.2937518Z ##[error]cannot create directory ‘/opt/hostedtoolcache’
2022-07-12T14:12:52.2939104Z ##[error]: Permission denied
2022-07-12T14:12:52.2941609Z ##[error]The process '/usr/bin/bash' failed with exit code 1
2022-07-12T14:12:52.3294942Z Post job cleanup.

Action version:

v4

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

• 3.10.5

Repro steps:

  setup-python:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: 3.x

Expected behavior:

Won't error.

Actual behavior:

It errors.

[IvanZosimov]

Hi, @kenorb 👋 Thank you for that issue, we will investigate it and get back to you with updates.

[dmitry-shibanov]

Hello @kenorb. Could you please take a look at this pull request. I think it can be related to your issue.

[techman83]

@dmitry-shibanov yes, it would be directly related. RUNNER_TOOL_CACHE on GitHub hosted vs Self Hosted is different.

Self Hosted

Run echo ${RUNNER_TOOL_CACHE}
  echo ${RUNNER_TOOL_CACHE}
  shell: /usr/bin/bash -e {0}
/home/actions/actions-runner/_work/_tool

Hosted

Run echo ${RUNNER_TOOL_CACHE}
  echo ${RUNNER_TOOL_CACHE}
  shell: /usr/bin/bash -e {0}
/opt/hostedtoolcache

This variable is a default, I'd expect whilst we might want to make sure it exists, replacing it with a fixed path might cause consumers of the action to experience unexpected behaviour.

We're experience similar issues, as the runner is intentionally not running as root, so will not be able to write to that path.

[dsame]

@techman83
The best solution is just to create /opt/hostedtoolcache with the write permissions on the self-hosted manually as it is said in README:

Alternatively, you can choose the installation directory to any location by setting AGENT_TOOLSDIRECTORY env. variable, but it should not be used because of:

[techman83]

@dsame yes, that is a work around. Why would an action choose to override this variable as opposed to being a consumer of it? If for instance there is another workflow step that uses this path before, and after the python actions setup, the behaviour would be unexpected.

I'm not sure I follow what the benefit of overriding this default is, and what is gained by trying to assume what GitHub would have set it to.

I see what you are saying here, that it relates to actions/python-versions. However reading through the source code, it uses RUNNER_TOOL_CACHE environment variable if AGENT_TOOLSDIRECTORY is not provided.

So, realistically there shouldn't be a requirement to override this variable. Rather when AGENT_TOOLSDIRECTORY is specified, use that path instead of RUNNER_TOOL_CACHE.

Along with that, actions/python-versions is likely to be run before actions/setup-python, so even allowing write to /opt/hostedtoolcache won't help, as the build output will actually be in /home/actions/actions-runner/_work/_tool.

[dsame]

@techman83
This is not workaround - this is the necessary host agent preparation step described in the README. The github hosted images already have the directory, but for self-images an user must create it manually.

The reason why the python must be installed there described in the README

The Python packages that are downloaded from actions/python-versions are originally compiled from source in /opt/hostedtoolcache/ with the [--enable-shared](https://github.com/actions/python-versions/blob/94f04ae6806c6633c82db94c6406a16e17decd5c/builders/ubuntu-python-builder.psm1#L35) flag, which makes them non-relocatable.

Unfortunatelly RUNNER_TOOL_CACHE on the self-hosted agent points to /home/actions/actions-runner/_work/_tool which leads to unpredictable issues and this is why we have to override it to point to the correct location /opt/hostedtoolcache

In the previous versions of setup-python it was necessary to assign AGENT_TOOLSDIRECTORY explicitly to /opt/hostedtoolcache but it seemed to be error prone redundant requirement and now AGENT_TOOLSDIRECTORY has default value /opt/hostedtoolcache and should never be changed.

But in some environments an user may have not ability to create /opt/hostedtoolcache directory and then, assuming he knows that he is doing, it is still possible to install python to other location.

Thank you for your question, feel free to ask more if you still have some problems - it really helps to improve the code.

[techman83]

@dsame - Apologies if I came across as blunt yesterday, still getting over covid and had already shaved many yaks. Having read through actions/python-versions this does not appear necessary. I have forked and done a quick hack to test my theory.

Log

Version 3.11.0-beta.4 was not found in the local cache
Version 3.11.0-beta.4 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.11.0-beta.4-266143546[8](https://github.com/Effortless-Networking/test-actions/runs/7332033650?check_suite_focus=true#step:4:9)/python-3.11.0-beta.4-linux-20.04-x64.tar.gz"
Extract downloaded archive
/usr/bin/tar xz --warning=no-unknown-keyword -C /home/actions/actions-runner/_work/_temp/625425b3-7372-47a5-[9](https://github.com/Effortless-Networking/test-actions/runs/7332033650?check_suite_focus=true#step:4:10)836-2d19931eb061 -f /home/actions/actions-runner/_work/_temp/56914020-7d5d-443e-81b9-ddae1e3bfff2
Execute installation script
Check if Python hostedtoolcache folder exist...
Creating Python hostedtoolcache folder...
Create Python 3.[11](https://github.com/Effortless-Networking/test-actions/runs/7332033650?check_suite_focus=true#step:4:12).0-beta.4 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading pip...
Looking in links: /tmp/tmpta_28ad7
Requirement already satisfied: setuptools in /home/actions/actions-runner/_work/_tool/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (58.1.0)
Requirement already satisfied: pip in /home/actions/actions-runner/_work/_tool/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (22.0.4)
Collecting pip
Downloading pip-[22](https://github.com/Effortless-Networking/test-actions/runs/7332033650?check_suite_focus=true#step:4:23).1.2-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 28.3 MB/s eta 0:00:00
Installing collected packages: pip
Successfully installed pip-22.1.2
Create complete file
Successfully set up CPython (3.11.0-beta.4)

Files in the local docker container connected to my test org:

actions@a34231b1bafa:~/actions-runner$ ls -lah _work/_tool/Python/3.11.0-beta.4/
total 12K
drwxr-xr-x 3 actions actions 4.0K Jul 14 01:15 .
drwxr-xr-x 3 actions actions 4.0K Jul 14 01:14 ..
drwxr-xr-x 6 actions actions 4.0K Jul 14 01:14 x64
-rw-r--r-- 1 actions actions    0 Jul 14 01:15 x64.complete

So to me it doesn't look necessary, and adds extra steps when configuring a self hosted runner, along with breaking existing workflows (though there is always that one guy). One of the benefits is normally that the runner is pretty self contained.

[techman83]

And for completeness to check hosted (I haven't checked self hosted Mac/Windows, I have not convenient access to either).

Linux Hosted:

Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages

Run techman83/setup-python@fix/tool_path
Version 3.11.0-beta.4 was not found in the local cache
Version 3.11.0-beta.4 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.11.0-beta.4-2661435468/python-3.11.0-beta.4-linux-20.04-x64.tar.gz"
Extract downloaded archive
/usr/bin/tar xz --warning=no-unknown-keyword -C /home/runner/work/_temp/bc6a529a-13[7](https://github.com/Effortless-Networking/test-actions/runs/7332530725?check_suite_focus=true#step:3:8)7-4477-bfe1-e2ac9bbec74a -f /home/runner/work/_temp/21a2a966-f272-471[8](https://github.com/Effortless-Networking/test-actions/runs/7332530725?check_suite_focus=true#step:3:9)-87d5-7104c13d[9](https://github.com/Effortless-Networking/test-actions/runs/7332530725?check_suite_focus=true#step:3:10)3cd
Execute installation script
Check if Python hostedtoolcache folder exist...
Create Python 3.[11](https://github.com/Effortless-Networking/test-actions/runs/7332530725?check_suite_focus=true#step:3:12).0-beta.4 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading pip...
Looking in links: /tmp/tmpb7m5gq7b
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (58.1.0)
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (22.0.4)
Collecting pip
Downloading pip-[22](https://github.com/Effortless-Networking/test-actions/runs/7332530725?check_suite_focus=true#step:3:23).1.2-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 84.9 MB/s eta 0:00:00
Installing collected packages: pip
Successfully installed pip-22.1.2
Create complete file
Successfully set up CPython (3.11.0-beta.4)

Windows Hosted:

Requirement already satisfied: setuptools in c:\hostedtoolcache\windows\python\3.11.0-beta.4\x64\lib\site-packages (58.1.0)

Run techman83/setup-python@fix/tool_path
Version 3.11.0-beta.4 was not found in the local cache
Version 3.11.0-beta.4 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.11.0-beta.4-2661435468/python-3.11.0-beta.4-win32-x64.zip"
Extract downloaded archive
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command "$ErrorActionPreference = 'Stop' ; try { Add-Type -AssemblyName System.IO.Compression.FileSystem } catch { } ; [System.IO.Compression.ZipFile]::ExtractToDirectory('D:\a\_temp\c501b4c7-c0e0-4a52-a1ec-178d361ae7f9', 'D:\a\_temp\a28e5e41-49e7-4003-bf2e-fcd102e395d0')"
Execute installation script
Check if Python hostedtoolcache folder exist...
Check if current Python version is installed...
No Python3.11.* found
Remove registry entries for Python 3.11(x64)...
Create Python 3.11.0-beta.4 folder in C:\hostedtoolcache\windows\Python
Copy Python binaries to C:\hostedtoolcache\windows\Python\3.11.0-beta.4\x64
Install Python 3.11.0-beta.4 in C:\hostedtoolcache\windows\Python...
Create python3 symlink
Directory: C:\hostedtoolcache\windows\Python\3.11.0-beta.4\x64
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---l         7/14/2022   2:19 AM              0 python3.exe
Install and upgrade Pip
Looking in links: c:\Users\RUNNER~1\AppData\Local\Temp\tmp4dibj_3k
Requirement already satisfied: setuptools in c:\hostedtoolcache\windows\python\3.11.0-beta.4\x64\lib\site-packages (58.1.0)
Requirement already satisfied: pip in c:\hostedtoolcache\windows\python\3.11.0-beta.4\x64\lib\site-packages (22.0.4)
Requirement already satisfied: pip in c:\hostedtoolcache\windows\python\3.11.0-beta.4\x64\lib\site-packages (22.0.4)
Collecting pip
Downloading pip-22.1.2-py3-none-any.whl (2.1 MB)
---------------------------------------- 2.1/2.1 MB 17.0 MB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Successfully uninstalled pip-22.0.4
Successfully installed pip-22.1.2
Create complete file
Successfully set up CPython (3.11.0-beta.4)

Mac Hosted:

Requirement already satisfied: pip in /Users/runner/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (22.0.4)

Run techman83/setup-python@fix/tool_path
  
Version 3.11.0-beta.4 was not found in the local cache
Version 3.11.0-beta.4 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.11.0-beta.4-2661435468/python-3.11.0-beta.4-darwin-x64.tar.gz"
Extract downloaded archive
/usr/bin/tar xz -C /Users/runner/work/_temp/d2c80313-3a62-466d-b5e4-8a101b949bb9 -f /Users/runner/work/_temp/b147eaa3-0341-46e6-b0ea-0c7322549c12
Execute installation script
Check if Python hostedtoolcache folder exist...
Create Python 3.11.0-beta.4 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading pip...
Looking in links: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpke_ijoy4
Requirement already satisfied: setuptools in /Users/runner/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (58.1.0)
Requirement already satisfied: pip in /Users/runner/hostedtoolcache/Python/3.11.0-beta.4/x64/lib/python3.11/site-packages (22.0.4)
Collecting pip
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Successfully installed pip-22.1.2
Create complete file
Successfully set up CPython (3.11.0-beta.4)

[dsame]

@techman83

As you already have noted, the python is installed in _work/_tool directory if AGENT_TOOLSDIRECTORY is not specified:

ls -lah _work/_tool/Python/3.11.0-beta.4/

This works in many cases, but the README warns:
compiled from source in /opt/hostedtoolcache/ with the [--enable-shared](https://github.com/actions/python-versions/blob/94f04ae6806c6633c82db94c6406a16e17decd5c/builders/ubuntu-python-builder.psm1#L35) flag, which makes them non-relocatable

Installing python in the directory other then /opt/hostedtoolcache/ works in many cases, but always there's a probability of unpredictable misbehaviour as soon as some shared libraries loaded. To avoid this the default installation forced to be /opt/hostedtoolcache/ although with AGENT_TOOLSDIRECTORY it is still possible to choose another installation directory.

Does it make sense now?

[techman83]

@dsame not really. I am aware of the warning, but this doesn't match the behaviour of the actions/python-versions that this change has been introduced to address. The *nix helper in python-versions is:

if [ -z ${AGENT_TOOLSDIRECTORY+x} ]; then
    # No AGENT_TOOLSDIRECTORY on GitHub images
    TOOLCACHE_ROOT=$RUNNER_TOOL_CACHE
else
    TOOLCACHE_ROOT=$AGENT_TOOLSDIRECTORY
fi

Internally they use a TOOLCACHE_ROOT variable, which they set to RUNNER_TOOL_CACHE when AGENT_TOOLSDIRECTORY is not provided. If the same is implemented here, actions-python will behave in the same manner, and respect the path that is configured as part of the self hosted runner options.

./config.sh \
  --url "https://github.com/${ORG}" \
  --token $(curl -s -X POST -H "authorization: token ${TOKEN}" "https://github.com/api/orgs/${ORG}/actions/runners/registration-token" | jq -r .token) \
  --name "${NAME}" \
  --unattended \
  --labels ${NAME} \
  --work _work

Part of the use case (not one we utilise) is for runners without internet access. You can see references to the documentation for this path. Other use cases maybe mounting a shared volume for all your runners, so they can access the same tool cache.

RUNNER_DIR
├── ...
└── _work
    ├── ...
    └── _tool
        └── node
            ├── 10.22.0
            │   └── ...
            └── 12.18.3
                └── ...

Overriding this path within an action is unexpected and doesn't match GitHub's documented use of that variable.

To be explicit, that documentation specifically calls out setup-python as a use case, which this is a breaking change for.

For example, if you need to use the setup-python action, you will need to populate the tool cache with the Python environment you want to use.