move parts of our db initialization into db service #15549

ewdurbin · 2024-03-11T13:27:54Z

Move as much of our database initialization into the db container itself to improve ergonomics for both rstuf and warehouse.

This ensures the rstuf db is created before make initdb is run.

Running make initdb can now work without make serve first!

make serve and make tests are now both a one-shot! no need for a separate window for initdb

di · 2024-03-11T14:21:52Z

Here's what happens when I try to run make serve on this branch:

$ make serve
make .state/db-populated
make[1]: Entering directory '/home/di/git/pypi/warehouse'
docker compose up -d db
[+] Running 1/0
 ✔ Container warehouse-db-1  Running                                                                                                                  0.0s
docker compose exec db /usr/local/bin/wait-for-db
waiting for db to be prepared.
make runmigrations
make[2]: Entering directory '/home/di/git/pypi/warehouse'
docker compose run --rm web python -m warehouse db upgrade head
[+] Creating 4/0
 ✔ Container warehouse-elasticsearch-1  Running                                                                                                       0.0s
 ✔ Container warehouse-stripe-1         Running                                                                                                       0.0s
 ✔ Container warehouse-redis-1          Running                                                                                                       0.0s
 ✔ Container warehouse-db-1             Running                                                                                                       0.0s
dependency failed to start: container warehouse-db-run-be9f291a638d exited (2)
make[2]: *** [Makefile:127: runmigrations] Error 1
make[2]: Leaving directory '/home/di/git/pypi/warehouse'
make[1]: *** [Makefile:120: .state/db-migrated] Error 2
make[1]: Leaving directory '/home/di/git/pypi/warehouse'
make: *** [Makefile:58: serve] Error 2

ewdurbin · 2024-03-11T14:23:42Z

@di try make purge first.

di · 2024-03-11T14:26:57Z

Same outcome:

$ make purge
docker compose stop
rm -rf dev/*.sql
rm -rf .state dev/.coverage* dev/.mypy_cache dev/.pip-cache dev/.pip-tools-cache dev/.pytest_cache .state/db-populated .state/db-migrated
docker compose down -v
[+] Running 7/0
 ! Volume warehouse_packages          Resource is still in use                                                                                        0.0s
 ! Volume warehouse_policies          Resource is still in use                                                                                        0.0s
 ! Volume warehouse_caches            Resource is still in use                                                                                        0.0s
 ! Volume warehouse_packages-archive  Resource is still in use                                                                                        0.0s
 ! Volume warehouse_sponsorlogos      Resource is still in use                                                                                        0.0s
 ✔ Volume warehouse_rstuf-metadata    Removed                                                                                                         0.0s
 ! Volume warehouse_simple            Resource is still in use                                                                                        0.0s
docker compose rm --force
No stopped containers

$ make serve
# Build our base container for this project.
docker compose build --build-arg IPYTHON=no --force-rm base
[+] Building 2.1s (30/30) FINISHED                                                                                                          docker:default
 => [base internal] load build definition from Dockerfile                                                                                             0.1s
 => => transferring dockerfile: 10.00kB                                                                                                               0.0s
 => [base internal] load metadata for docker.io/library/python:3.11.8-slim-bookworm                                                                   0.4s
 => [base internal] load metadata for docker.io/library/node:20.11.1-bookworm                                                                         0.4s
 => [base internal] load .dockerignore                                                                                                                0.1s
 => => transferring context: 127B                                                                                                                     0.0s
 => [base internal] load build context                                                                                                                1.2s
 => => transferring context: 59.78MB                                                                                                                  1.2s
 => [base static-deps 1/4] FROM docker.io/library/node:20.11.1-bookworm@sha256:f3299f16246c71ab8b304d6745bb4059fa9283e8d025972e28436a9f9b36ed24       0.0s
 => [base build 1/9] FROM docker.io/library/python:3.11.8-slim-bookworm@sha256:ce81dc539f0aedc9114cae640f8352fad83d37461c24a3615b01f081d0c0583a       0.0s
 => CACHED [base stage-4 2/8] WORKDIR /opt/warehouse/src/                                                                                             0.0s
 => CACHED [base stage-4 3/8] RUN set -x     && mkdir -p /usr/share/man/man1     && mkdir -p /usr/share/man/man7                                      0.0s
 => CACHED [base stage-4 4/8] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=locked   0.0s
 => CACHED [base static-deps 2/4] WORKDIR /opt/warehouse/src/                                                                                         0.0s
 => CACHED [base static-deps 3/4] COPY package.json package-lock.json .babelrc /opt/warehouse/src/                                                    0.0s
 => CACHED [base static-deps 4/4] RUN --mount=type=cache,target=/root/.npm,sharing=locked     npm ci                                                  0.0s
 => CACHED [base static 1/4] COPY warehouse/static/ /opt/warehouse/src/warehouse/static/                                                              0.0s
 => CACHED [base static 2/4] COPY warehouse/admin/static/ /opt/warehouse/src/warehouse/admin/static/                                                  0.0s
 => CACHED [base static 3/4] COPY webpack.config.js /opt/warehouse/src/                                                                               0.0s
 => CACHED [base static 4/4] RUN NODE_ENV=production npm run build                                                                                    0.0s
 => CACHED [base stage-4 5/8] COPY --from=static /opt/warehouse/src/warehouse/static/dist/ /opt/warehouse/src/warehouse/static/dist/                  0.0s
 => CACHED [base stage-4 6/8] COPY --from=static /opt/warehouse/src/warehouse/admin/static/dist/ /opt/warehouse/src/warehouse/admin/static/dist/      0.0s
 => CACHED [base build 2/9] RUN set -eux;     rm -f /etc/apt/apt.conf.d/docker-clean;     echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";'   0.0s
 => CACHED [base build 3/9] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=locked     0.0s
 => CACHED [base build 4/9] RUN set -x     && python3 -m venv /opt/warehouse                                                                          0.0s
 => CACHED [base build 5/9] RUN pip --no-cache-dir --disable-pip-version-check install --upgrade pip setuptools wheel                                 0.0s
 => CACHED [base build 6/9] COPY requirements /tmp/requirements                                                                                       0.0s
 => CACHED [base build 7/9] RUN --mount=type=cache,target=/root/.cache/pip     set -x     && if [ "yes" = "yes" ]; then pip --disable-pip-version-ch  0.0s
 => CACHED [base build 8/9] RUN --mount=type=cache,target=/root/.cache/pip     set -x     && if [ "yes" = "yes" ] && [ "no" = "yes" ]; then pip --di  0.0s
 => CACHED [base build 9/9] RUN --mount=type=cache,target=/root/.cache/pip     set -x     && pip --disable-pip-version-check             install --n  0.0s
 => CACHED [base stage-4 7/8] COPY --from=build /opt/warehouse/ /opt/warehouse/                                                                       0.0s
 => CACHED [base stage-4 8/8] COPY . /opt/warehouse/src/                                                                                              0.0s
 => [base] exporting to image                                                                                                                         0.1s
 => => exporting layers                                                                                                                               0.0s
 => => writing image sha256:fb9d5f1a3372ba15b071ce4482594bfd922c69416d9d8f6699cfb48db456c1ed                                                          0.1s
 => => naming to docker.io/library/warehouse:docker-compose                                                                                           0.0s
# Mark the state so we don't rebuild this needlessly.
mkdir -p .state
touch .state/docker-build-base
# Build our static container for this project.
docker compose build --force-rm static
[+] Building 0.4s (9/9) FINISHED                                                                                                            docker:default
 => [static internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 10.00kB                                                                                                               0.0s
 => [static internal] load metadata for docker.io/library/node:20.11.1-bookworm                                                                       0.1s
 => [static internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 127B                                                                                                                     0.0s
 => [static static-deps 1/4] FROM docker.io/library/node:20.11.1-bookworm@sha256:f3299f16246c71ab8b304d6745bb4059fa9283e8d025972e28436a9f9b36ed24     0.0s
 => [static internal] load build context                                                                                                              0.1s
 => => transferring context: 99B                                                                                                                      0.1s
 => CACHED [static static-deps 2/4] WORKDIR /opt/warehouse/src/                                                                                       0.0s
 => CACHED [static static-deps 3/4] COPY package.json package-lock.json .babelrc /opt/warehouse/src/                                                  0.0s
 => CACHED [static static-deps 4/4] RUN --mount=type=cache,target=/root/.npm,sharing=locked     npm ci                                                0.0s
 => [static] exporting to image                                                                                                                       0.0s
 => => exporting layers                                                                                                                               0.0s
 => => writing image sha256:35630f9a46dfc69207ff7bc00171edd008938f09de14daa1a07190900f80f416                                                          0.0s
 => => naming to docker.io/library/warehouse:docker-compose-static                                                                                    0.0s
# Mark the state so we don't rebuild this needlessly.
mkdir -p .state
touch .state/docker-build-static
# Build the worker container for this project
docker compose build --build-arg  USER_ID=1001  --build-arg GROUP_ID=1002 --force-rm dev-docs
[+] Building 0.4s (16/16) FINISHED                                                                                                          docker:default
 => [dev-docs internal] load build definition from Dockerfile                                                                                         0.0s
 => => transferring dockerfile: 10.00kB                                                                                                               0.0s
 => [dev-docs internal] load metadata for docker.io/library/python:3.11.8-slim-bookworm                                                               0.2s
 => [dev-docs internal] load .dockerignore                                                                                                            0.0s
 => => transferring context: 127B                                                                                                                     0.0s
 => [dev-docs docs  1/11] FROM docker.io/library/python:3.11.8-slim-bookworm@sha256:ce81dc539f0aedc9114cae640f8352fad83d37461c24a3615b01f081d0c0583a  0.0s
 => [dev-docs internal] load build context                                                                                                            0.0s
 => => transferring context: 301.24kB                                                                                                                 0.0s
 => CACHED [dev-docs docs  2/11] RUN set -eux;     rm -f /etc/apt/apt.conf.d/docker-clean;     echo 'Binary::apt::APT::Keep-Downloaded-Packages "tru  0.0s
 => CACHED [dev-docs docs  3/11] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=lock  0.0s
 => CACHED [dev-docs docs  4/11] RUN set -x     && python3 -m venv /opt/warehouse                                                                     0.0s
 => CACHED [dev-docs docs  5/11] RUN pip --no-cache-dir --disable-pip-version-check install --upgrade pip setuptools wheel                            0.0s
 => CACHED [dev-docs docs  6/11] COPY requirements /tmp/requirements                                                                                  0.0s
 => CACHED [dev-docs docs  7/11] RUN --mount=type=cache,target=/root/.cache/pip     set -x     && pip --disable-pip-version-check             instal  0.0s
 => CACHED [dev-docs docs  8/11] WORKDIR /opt/warehouse/src/                                                                                          0.0s
 => CACHED [dev-docs docs  9/11] RUN groupadd -o -g 1002 -r docs                                                                                      0.0s
 => CACHED [dev-docs docs 10/11] RUN useradd -o -m -u 1001 -g 1002 docs                                                                               0.0s
 => CACHED [dev-docs docs 11/11] RUN chown docs /opt/warehouse/src                                                                                    0.0s
 => [dev-docs] exporting to image                                                                                                                     0.0s
 => => exporting layers                                                                                                                               0.0s
 => => writing image sha256:02cbf529eecf42cf23e8d7a37eba436527550d9ae56bbfad37cba54021afdd7b                                                          0.0s
 => => naming to docker.io/library/warehouse:docker-compose-docs                                                                                      0.0s
# Mark the state so we don't rebuild this needlessly.
mkdir -p .state
touch .state/docker-build-docs
# Build the worker container for this project
docker compose build --force-rm worker
# Mark the state so we don't rebuild this needlessly.
mkdir -p .state
touch .state/docker-build
make .state/db-populated
make[1]: Entering directory '/home/di/git/pypi/warehouse'
docker compose up -d db
[+] Running 1/3
 ⠦ Network warehouse_default          Created                                                                                                         0.6s
 ⠴ Volume "warehouse_rstuf-metadata"  Created                                                                                                         0.5s
 ✔ Container warehouse-db-1           Started                                                                                                         0.5s
docker compose exec db /usr/local/bin/wait-for-db
waiting for db to be prepared................................................
make runmigrations
make[2]: Entering directory '/home/di/git/pypi/warehouse'
docker compose run --rm web python -m warehouse db upgrade head
[+] Creating 4/4
 ✔ Container warehouse-stripe-1         Created                                                                                                       0.2s
 ✔ Container warehouse-redis-1          Created                                                                                                       0.2s
 ✔ Container warehouse-elasticsearch-1  Created                                                                                                       0.2s
 ✔ Container warehouse-db-1             Running                                                                                                       0.0s
[+] Running 3/3
 ✔ Container warehouse-stripe-1         Started                                                                                                       0.6s
 ✔ Container warehouse-redis-1          Started                                                                                                       0.6s
 ✔ Container warehouse-elasticsearch-1  Started                                                                                                       0.7s
dependency failed to start: container warehouse-db-run-be9f291a638d exited (2)
make[2]: *** [Makefile:127: runmigrations] Error 1
make[2]: Leaving directory '/home/di/git/pypi/warehouse'
make[1]: *** [Makefile:120: .state/db-migrated] Error 2
make[1]: Leaving directory '/home/di/git/pypi/warehouse'
make: *** [Makefile:58: serve] Error 2

ewdurbin · 2024-03-11T14:30:53Z

@di looks like you have an orphaned db from a run command somehow, pull 4fe3e33 and purge again.

Makefile

di

Works great!

move parts of our db initialization into db service

2b87ee7

ewdurbin requested a review from a team as a code owner March 11, 2024 13:27

ewdurbin mentioned this pull request Mar 11, 2024

PEP458: Bootstrap TUF metadata repository for development #15484

Merged

simplify dev experience further by complicating Makefile

c1f313c

remove orphans on purge

4fe3e33

di reviewed Mar 11, 2024

View reviewed changes

Makefile Show resolved Hide resolved

di approved these changes Mar 11, 2024

View reviewed changes

ewdurbin added 3 commits March 11, 2024 11:16

sush encoding warning on make tests

322538a

give more time in bin/tests for db-initialization

554daa1

pause web and workers during resetdb

ca071c9

ewdurbin merged commit fca2efa into main Mar 11, 2024

ewdurbin deleted the db-initification branch March 11, 2024 19:10

di mentioned this pull request Apr 3, 2024

Use docker initialization scripts for DB init #9993

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

move parts of our db initialization into db service #15549

move parts of our db initialization into db service #15549

Uh oh!

ewdurbin commented Mar 11, 2024 •

edited

Loading

Uh oh!

di commented Mar 11, 2024

Uh oh!

ewdurbin commented Mar 11, 2024

Uh oh!

di commented Mar 11, 2024

Uh oh!

ewdurbin commented Mar 11, 2024

Uh oh!

Uh oh!

di left a comment

Uh oh!

Uh oh!

move parts of our db initialization into db service #15549

move parts of our db initialization into db service #15549

Uh oh!

Conversation

ewdurbin commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

di commented Mar 11, 2024

Uh oh!

ewdurbin commented Mar 11, 2024

Uh oh!

di commented Mar 11, 2024

Uh oh!

ewdurbin commented Mar 11, 2024

Uh oh!

Uh oh!

di left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ewdurbin commented Mar 11, 2024 •

edited

Loading