Skip to content

PEP458: Add minimal RSTUF service configuration #15241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jan 26, 2024
Merged

PEP458: Add minimal RSTUF service configuration #15241

merged 5 commits into from
Jan 26, 2024

Conversation

kairoaraujo
Copy link
Contributor

@kairoaraujo kairoaraujo commented Jan 21, 2024 •
edited
Loading

Supersedes: #13943

This PR is part 1 in a series of PRs to integrate Repository Service for TUF (RSTUF) with Warehouse for PEP 458 adoption.

Background
RSTUF maintains a PEP 458 TUF metadata repository as a black box. It provides REST API, which accepts requests upon addition or removal of release files, and a Worker service, which performs the TUF metadata update and signing tasks asynchronously upon API requests.

Change details

  • Add rstuf-api service to docker-compose.yml

    • pull pinned image
    • configure redis

  • Add rstuf-worker service to docker-compose.yml

    • pull pinned image
    • configure storage backend for TUF metadata files
    • configure Postgres DB for TUF metadata
    • configure Redis, used as task message queue for communication between API and Worker, and between Workers

  • Include rstuf db initialization in initdb target in Makefile.

Preview of next PR
The next PR will include:

  • Signing configuration, to sign online metadata changes
  • Static root metadata, to bootstrap the RSTUF repository
  • A Makefile target, to trigger bootstrap
  • Instructions, to interact with the RSTUF REST API

@kairoaraujo
Copy link
Contributor Author

kairoaraujo commented Jan 21, 2024 •
edited
Loading

@lukpueh
@ewdurbin
@dstufft
🚀

@kairoaraujo kairoaraujo force-pushed the rstuf_dev_infra branch 2 times, most recently from 6045b4d to e5b0658 Compare January 25, 2024 08:56
@kairoaraujo kairoaraujo changed the title PEP458: Add RSTUF services in the Warehouse Infra PEP458: Add minimal RSTUF service configuration Jan 25, 2024
@kairoaraujo kairoaraujo marked this pull request as ready for review January 25, 2024 15:54
@kairoaraujo kairoaraujo requested a review from a team as a code owner January 25, 2024 15:54
ewdurbin
ewdurbin approved these changes Jan 25, 2024
View reviewed changes
Copy link
Member

@ewdurbin ewdurbin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall i appreciate how clean this is when treated as a dependent service.

i believe we could remove the vault service at this time since nothing integrates with it and it only was added in support of implementing TUF inside warehouse.

@kairoaraujo
remove vault container
7ed9a00 
The vault container was a previous TUF implementation
(TUF initialization pypi#7488)

The new integration with RSTUF does not require this container.

Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo
PEP 458: Add RSTUF services in the Warehouse Infra
2e182d3 
This commit adds the RSTUF services to the Warehouse infrastructure
for development and sets the minimum required to start RSTUF services.

It adds the RSTUF API, which is used later to integrate into Warehouse
and RSTUF Worker, which is responsible for computing the TUF metadata.

The RSTUF requires the Postgres and Redis.
Postgres stores the rstuf database used for TUF metadata computing.
Redis stores the task message queue between RSTUF API and Worker, task
backend result, and live settings between RSTUF services.

RSTUF shares the same Postgres and Redis in development environment
but has a specific setup to use its own Postgres database and Redis
database ID.

Postgresql URI
`RSTUF_SQL_SERVER=postgresql://postgres@db:5432/rstuf`

Redis DB Broker and Result is id 1
`RSTUF_BROKER_SERVER=redis://redis/1`
`RSTUF_REDIS_SERVER_DB_RESULT=1`

Redis DB for TUF repository settings is 2
`RSTUF_REDIS_SERVER_DB_REPO_SETTINGS=2`

This commit also includes TUF database creation in the Makefile
during the `make initdb`.

Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo
remove rstuf-worker unnecessary settings
6116b2e 
Remove settings from rstuf-worker in docker-compose.yml

Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo
Copy link
Contributor Author

i believe we could remove the vault service at this time since nothing integrates with it and it only was added in support of implementing TUF inside warehouse.

I added the vault removal as the first commit to keep a logical sequence in the PR commits.

@ewdurbin
Copy link
Member

Thanks @kairoaraujo! Looks great.

@ewdurbin ewdurbin enabled auto-merge (squash) January 25, 2024 19:39
@ewdurbin ewdurbin disabled auto-merge January 25, 2024 19:46
@ewdurbin
Copy link
Member

Unclear to me why CI is failing. Seems to be unrelated and consistent with other branches.

@kairoaraujo
Copy link
Contributor Author

kairoaraujo commented Jan 25, 2024 •
edited
Loading

Unclear to me why CI is failing. Seems to be unrelated and consistent with other branches.

I see this failure in two other opened PRs 🤔

https://github.com/pypi/warehouse/actions/runs/7655175103/job/20867003665#step:8:1

@kairoaraujo
Copy link
Contributor Author

Unclear to me why CI is failing. Seems to be unrelated and consistent with other branches.

I see this failure in two other opened PRs 🤔

https://github.com/pypi/warehouse/actions/runs/7655175103/job/20867003665#step:8:1

I did some investigation and I commented here:
#15276 (comment)

@ewdurbin ewdurbin merged commit 34dd5ef into pypi:main Jan 26, 2024
This was referenced Jan 31, 2024
Closed
Closed
Closed
Closed
@kairoaraujo kairoaraujo mentioned this pull request Feb 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lukpueh lukpueh lukpueh left review comments

@ewdurbin ewdurbin ewdurbin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kairoaraujo @ewdurbin @lukpueh