Skip to content

Allow pausing IO scheduler from inside core #4138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Mar 19, 2023
Merged

Allow pausing IO scheduler from inside core #4138

merged 12 commits into from
Mar 19, 2023

Conversation

flub
Copy link
Contributor

@flub flub commented Mar 8, 2023

To handle backups the UIs have to make sure they do stop the IO
scheduler and also don't accidentally restart it while working on it.
Since they have to call start_io from a bunch of locations this can be
a bit difficult to manage.

This introduces a mechanism for the core to pause IO for some time,
which is used by the imex function. It interacts well with other
calls to dc_start_io() and dc_stop_io() making sure that when resumed
the scheduler will be running or not as the latest calls to them.

This was a little more invasive then hoped due to the scheduler. The
additional abstraction of the scheduler on the context seems a nice
improvement though.

@flub flub requested review from link2xt and r10s March 8, 2023 13:27
@flub
Copy link
Contributor Author

flub commented Mar 8, 2023

one question is also whether we should completely remove the .stop_io() call from the public api?

Copy link
Contributor

@r10s r10s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great stuff!

really looking forward to have that in! eg. UI typically call startIO on periodically every some minutes because of wakeups - or when the app comes to foreground. in the past, they have to take care not to do that under some circumstances - which is hard to track - and also was just not done in many cases.

however, although code looks good on a first glance (i esp. checked some boolean conditions) i cannot really judge the thing, i do know too less about the rust internals here.

@r10s
Copy link
Contributor

r10s commented Mar 8, 2023

one question is also whether we should completely remove the .stop_io() call from the public api?

isn't it still needed on shutdown? otherwise, yes, i'd do a stub as #define dc_stop_io((a)) to lower the barrier of things UI need to do, after all experierence, this makes things easier, eg. when switching core versions is needed.

@flub
Copy link
Contributor Author

flub commented Mar 9, 2023

one question is also whether we should completely remove the .stop_io() call from the public api?

isn't it still needed on shutdown? otherwise, yes, i'd do a stub as #define dc_stop_io((a)) to lower the barrier of things UI need to do, after all experierence, this makes things easier, eg. when switching core versions is needed.

oh, I was only going to remove it from the rust lib. not from deltachat-ffi which can keep it as a no-op.

@Simon-Laux
Copy link
Contributor

Simon-Laux commented Mar 9, 2023

one question is also whether we should completely remove the .stop_io() call from the public api?

desktop still uses all start and stop io calls (on manager and on account) to achieve the feature of only syncing the active account instead of all accounts in the background. But in the future it would be nicer if core could also manage this scenario. (desktop uses it over the jsonrpc api, so you could make the cffi a no-op probably?)

@link2xt
Copy link
Collaborator

link2xt commented Mar 9, 2023

to achieve the feature of only syncing the active account instead of all accounts in the background

Why is it needed? I thought it is better to have all accounts synchronized in the background so when you go to an account it is already updated.

@flub
Copy link
Contributor Author

flub commented Mar 9, 2023

one question is also whether we should completely remove the .stop_io() call from the public api?

desktop still uses all start and stop io calls (on manager and on account) to achieve the feature of only syncing the active account instead of all accounts in the background. But in the future it would be nicer if core could also manage this scenario. (desktop uses it over the jsonrpc api, so you could make the cffi a no-op probably?)

ok, let's not touch this for now. no harm leaving stop_io in.

@flub flub requested a review from link2xt March 9, 2023 14:59
@flub
Copy link
Contributor Author

flub commented Mar 11, 2023

@link2xt any remaining problems before this can be approved? would be nice to get merged

src/scheduler.rs Outdated
if self.done {
return;
}
let context = self.context.clone();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe emit an error! here and do nothing else? resume is better called explicitly in the async function so we don't have to spawn a task here. Spawning tasks from Drop is tricky, maybe everything around is in the process of being dropped. We normally don't use this, so this is just some untested dead code that may become unmaintained and not actually working when needed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested that when panic! is inserted here the tests still pass, then replaced it with error!.

flub added 7 commits March 19, 2023 09:36
To handle backups the UIs have to make sure they do stop the IO
scheduler and also don't accidentally restart it while working on it.
Since they have to call start_io from a bunch of locations this can be
a bit difficult to manage.

This introduces a mechanism for the core to pause IO for some time,
which is used by the imex function.  It interacts well with other
calls to dc_start_io() and dc_stop_io() making sure that when resumed
the scheduler will be running or not as the latest calls to them.

This was a little more invasive then hoped due to the scheduler.  The
additional abstraction of the scheduler on the context seems a nice
improvement though.
@link2xt
Copy link
Collaborator

link2xt commented Mar 19, 2023

I have rebased the PR

@link2xt
Copy link
Collaborator

link2xt commented Mar 19, 2023

I have resolved my own comments and will merge this manually with git merge --no-ff once the tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants