From 8fc84893ad49347174a67667f8b5051969934efd Mon Sep 17 00:00:00 2001 From: Marcelo Fernandes Date: Tue, 7 Apr 2026 08:31:26 +1200 Subject: [PATCH 1/2] Update README with information about partitioning --- README.md | 54 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 49 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e8ccda9..8965f07 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,16 @@ Psycopack can be used to perform the following operations without downtime: - Converting a primary key from int to bigint. - Clustering (or repacking) the table to address table and index bloat (similar to `pg_repack`). -- TODO: partition an existing table. +- Partition an existing table (only by date range at the moment). ## Basics Psycopack uses a shadow-table approach. This means creating a copy of the table (shadow table), synchronising its data -with the original table data, and swapping it with the original. +with the original table data, and swapping it with the original. It is also +possible to reverse the swap, so that you can verify the results before +deciding to commit to them. The Psycopack process achieves the above in 6 major steps: @@ -32,8 +34,26 @@ The Psycopack process achieves the above in 6 major steps: table. 5. **Swap**: Effectively swaps the original table by the copy table. The original (now old) table will be kept in sync via triggers, in case the swap - process needs to be reverted. -6. **Clean up**: Drops the old table. + process needs to be reverted. The swapping is done in a way that allows to + revert the process if something goes wrong, and without locking the table + for more than a few milliseconds. This is achieved via an operation called + **reverse swap**. +6. **Clean up**: Drops the old table. After this point, the process is + irreversible, so it's recommended to verify the results before running this + step. + +There are also two strategies for performing the process: + +- **Change log** (Recommended): The backfill is performed in batches, and + changes after the process kicks off are tracked via a change log table. This + allows the copy table to synchronise its schema with the original table + without using concurrent index creation, which can take a long time and can + be blocked by long-running transactions. +- **Triggers**: The backfill is performed in batches, and changes after the + process kicks off are immediately pushed to the copy table via triggers. This + allows the copy table to be up to date after the backfill process is + complete, the schema can be synchronised afterwards using operations that + only acquire weak locks. ## Python Interface @@ -62,6 +82,7 @@ with get_db_connection("postgresql://user:password@host:port/db") as conn: # Finish the process. psycopack.sync_schemas() + repack.post_sync_update() psycopack.swap() psycopack.clean_up() ``` @@ -89,7 +110,6 @@ with get_db_connection("postgresql://user:password@host:port/db") as conn: ### Example: Repacking (or clustering) ```python - from psycopack import psycopack, get_db_connection with get_db_connection("postgresql://user:password@host:port/db") as conn: @@ -106,6 +126,30 @@ with get_db_connection("postgresql://user:password@host:port/db") as conn: psycopack.full() ``` +### Example: Partitioning by date range + +```python +from psycopack import psycopack, get_db_connection + +with get_db_connection("postgresql://user:password@host:port/db") as conn: + with conn.cursor() as cur: + psycopack = Psycopack( + table="to_repack", + batch_size=1, + conn=connection, + cur=cur, + change_log_batch_size=10, + sync_strategy=SyncStrategy.CHANGE_LOG, + partition_config=_partition.PartitionConfig( + column="datetime_field", + num_of_extra_partitions_ahead=10, + strategy=_partition.DateRangeStrategy( + partition_by=_partition.PartitionInterval.DAY + ), + ), + ) +``` + ## Known Limitations The following types of tables aren't currently supported: From a68150891acbfd7d6fb884f3a5b889eb096d882d Mon Sep 17 00:00:00 2001 From: Marcelo Fernandes Date: Tue, 7 Apr 2026 09:04:37 +1200 Subject: [PATCH 2/2] Add CONTRIBUTING.md file --- CONTRIBUTING.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f9c2c65 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,51 @@ +# Contributing to Psycopack + +## Installing development dependencies + +```sh +make dev +``` + +## Running tests + +The entire test suite can be run with: + +```sh +make test +``` + +The test suite is supported by pystest. You can pass flags to pytest by running: + +```sh +PYTEST_FLAGS="tests/test_repack.py --pdb -vv" make test +``` + +To run tests against a specific database, set the `DATABASE_URL` environment variable: + +```sh +DATABASE_URL=postgres://marcelo.fernandes:postgres@localhost:5415/postgres make test +``` + +## Coverage + +We try to keep code coverage as high as possible. There are a few coverage commands: + +Run tests and produce a report in the terminal: + +```sh +make coverage +``` + +Run tests and produce a rich HTML report (more info than int the terminal command): + +```sh +make coverage_html +``` + +## Linter + +We use ruff and mypy for linting. To run both linters, run: + +```sh +make lint +```