Skip to content

tests/functional: assert R_OK/W_OK for XDG dirs #16322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jul 22, 2024
Merged

tests/functional: assert R_OK/W_OK for XDG dirs #16322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
03667a3
tests/functional: assert R_OK/W_OK for XDG dirs
woodruffw Jul 22, 2024
b8959cb
tests/functional: assert R_OK/W_OK for XDG dirs
woodruffw Jul 22, 2024
76de855
Dockerfile: remove XDG settings
woodruffw Jul 22, 2024
10c0a90
tests/functional: ensure_exists for platformdirs
woodruffw Jul 22, 2024
ad88a62
tests/functional: lintage
woodruffw Jul 22, 2024
109c09e
config: ensure XDG dirs
woodruffw Jul 22, 2024
a15df01
requirements: make platformdirs a direct dep
woodruffw Jul 22, 2024
b85eb2b
config: lintage
woodruffw Jul 22, 2024
0582d76
config: make the sanity check stronger
woodruffw Jul 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,15 +225,6 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Warehouse runs as `nobody` with `/nonexistent` as `$HOME`, which causes
# significant shpilkes for libraries that expect to use XDG dirs.
# Placate everybody by making some directories for runtime use.
# Note that this happens right after `/tmp/*` is blown away, because we want
# to keep these.
RUN mkdir -p /tmp/share /tmp/cache
ENV XDG_DATA_HOME /tmp/share
ENV XDG_CACHE_HOME /tmp/cache

# Copy the directory into the container, this is done last so that changes to
# Warehouse itself require the least amount of layers being invalidated from
# the cache. This is most important in development, but it also useful for
Expand Down
1 change: 1 addition & 0 deletions requirements/main.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ packaging_legacy
paginate>=0.5.2
paginate_sqlalchemy
passlib>=1.6.4
platformdirs>=4.2
premailer
psycopg[c]
pycurl
Expand Down
4 changes: 3 additions & 1 deletion requirements/main.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1516,7 +1516,9 @@ plaster-pastedeploy==1.0.1 \
platformdirs==4.2.2 \
--hash=sha256:2d7a1657e36a80ea911db832a8a6ece5ee53d8de21edd5cc5879af6530b1bfee \
--hash=sha256:38b7b51f512eed9e84a22788b4bce1de17c0adb134d6becb09836e37d8654cd3
# via sigstore
# via
# -r requirements/main.in
# sigstore
premailer==3.10.0 \
--hash=sha256:021b8196364d7df96d04f9ade51b794d0b77bcc19e998321c515633a2273be1a \
--hash=sha256:d1875a8411f5dc92b53ef9f193db6c0f879dc378d618e0ad292723e388bfe4c2
Expand Down
13 changes: 9 additions & 4 deletions tests/functional/test_environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import os

import platformdirs


def test_xdg_environment():
# Backstop checks for Warehouse's (Dockerfile-configured) environment.
user_data = platformdirs.user_data_dir()
user_cache = platformdirs.user_cache_dir()
# NOTE: We don't check the exact paths here, because they vary between
# environments: they're in `$HOME` in local and CI builds, and in `/tmp`
# when deployed.
user_data = platformdirs.user_data_dir(ensure_exists=True)
user_cache = platformdirs.user_cache_dir(ensure_exists=True)
Comment on lines +23 to +24
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: With ensure_exists this is now partially a no-op test, at least in local/CI testing (since these dirs don't exist). However for Cabotage/deployment it'll end up testing something useful, per cabotage/cabotage-app#85

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just for clarity, we do not execute tests in the cabotage environment. should this be a runtime check?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, didn't realize that -- yeah, in that case it probably makes more sense as a runtime check. I'll look at putting it somewhere early in python -m warehouse's initialization 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've put this in warehouse/config.py, as part of the early initialization that happens there. But I'm not 100% sure if that's the best place 🙂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In smoke testing this, it looks like this doesn't correctly detect that things are working.

I rolled out a new release of test.pypi.org using this wrapper and you can see results in cabotage/cabotage-app#85 (comment)

platformdirs.user_data_dir(ensure_exists=True) succeeds, but platformdirs.user_data_dir('app', ensure_exists=True) fails, indicating that this wouldn't actually ensure success for xdg/platformdirs use later on.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appears this is due to the existing /tmp/{share, cache} files created by the Dockerfile on main :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validated this locally!

I still think it may be wise to change these to platformdirs.user_cache_dir(secrets.token_urlsafe(), ensure_exists=True) and similar to ensure not only that the dirs exist but that they're writable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appears this is due to the existing /tmp/{share, cache} files created by the Dockerfile on main :)

Whoops 🤦 -- I think merging this first should fix that then, since we won't be doing mkdir in the Dockerfile anymore.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think it may be wise to change these to platformdirs.user_cache_dir(secrets.token_urlsafe(), ensure_exists=True) and similar to ensure not only that the dirs exist but that they're writable.

Done!


assert user_data == "/tmp/share"
assert user_cache == "/tmp/cache"
assert os.access(user_data, os.R_OK | os.W_OK)
assert os.access(user_cache, os.R_OK | os.W_OK)
7 changes: 7 additions & 0 deletions warehouse/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@
import enum
import json
import os
import secrets
import shlex

from datetime import timedelta

import orjson
import platformdirs
import transaction

from pyramid import renderers
Expand Down Expand Up @@ -241,6 +243,11 @@ def from_base64_encoded_json(configuration):


def configure(settings=None):
# Sanity check: regardless of what we're configuring, some of Warehouse's
# application state depends on a handful of XDG directories existing.
platformdirs.user_data_dir(appname=secrets.token_urlsafe(), ensure_exists=True)
platformdirs.user_cache_dir(appname=secrets.token_urlsafe(), ensure_exists=True)

if settings is None:
settings = {}
settings["warehouse.forklift.legacy.MAX_FILESIZE_MIB"] = MAX_FILESIZE / ONE_MIB
Expand Down