In Manager use multiprocessing.Pool not ProcessPoolExecutor

[JackKelly]

We probably want the worker processes to be demonic.

If that's not possible then maybe have Manager only send a single batch to each process at a time (so the workers will terminate faster)

noxdafox/pebble#76 suggests that ProcessPoolExecutor is not demonic, whilst multiprocessing.Pool is demonic

https://stackoverflow.com/questions/56237493/how-to-make-tasks-in-processpoolexecutor-behave-like-daemon-process suggests we should be using multiprocessing.Pool

[peterdudfield]

could we use 'futures.ThreadPoolExecutor' - I have admit I do get a bit confused with some multiprocess things

[JackKelly]

Hehe, me too!

So, I'm pretty sure we want to use processes not threads because the tasks are CPU bound, and python threads run on a single CPU core due to the python global interpreter lock.

Don't worry, I can have a look at this issue real soon now

[JackKelly]

I'll start working on this now...

[JackKelly]

OK, this is implemented in PR #453

This all makes me think that issue #321 would be good to do soon. But maybe not today 🙂

This doesn't seem to have sped things up... but I wasn't really expecting it to... Speeding things up is what I'll work on now (in a separate PR)...