feat: Disable wal_autocheckpoint #7121
Conversation
From https://www.sqlite.org/wal.html: > The default strategy is to allow successive write transactions to grow the WAL until the WAL becomes about 1000 pages in size, then to run a checkpoint operation for each subsequent COMMIT until the WAL is reset to be smaller than 1000 pages. By default, the checkpoint will be run automatically by the same thread that does the COMMIT that pushes the WAL over its size limit. This has the effect of causing most COMMIT operations to be very fast but an occasional COMMIT (those that trigger a checkpoint) to be much slower. And while autocheckpoint runs in the `PASSIVE` mode and thus doesn't block concurrent readers and writers, in our design it blocks writers because it's done under `write_mutex` locked and thus may cause the app to stuck for noticeable time. Let's disable autocheckpointing then, we can't rely on it anyway.
|
We should consider scheduling |
|
Also w/o WAL autocheckpointing there's another interesting option: we can disable running |
Some migrations want housekeeping to run. Also if housekeeping failed before, fixing the reason and restarting the program is the most natural way to retry it.
|
I am not sure how many pages we accumulate during the day. I'm afraid we will grow a huge unlimited WAL in some workloads, e.g. for a bot that receives a lot of messages and normally deletes them almost immediately. It seems correct way is to register our own WAL hook (https://sqlite.org/c3ref/wal_hook.html) that notifies a separate thread to do manual checkpointing via channel when WAL grows too large. And just have this thread/task constantly waiting in for notification and then taking all the connections and doing checkpointing just like it is currently done in housekeeping. It is possible to register a hook via https://docs.rs/rusqlite/0.37.0/rusqlite/struct.Connection.html#method.wal_hook, it also unregisters autocheckpointing. |
|
If a bot deletes messages or chats, |
No description provided.