Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Run get_sample() for all 3 DataSources in a ThreadPoolExecutor #38

Closed
JackKelly opened this issue Jul 2, 2021 · 0 comments
Closed

Run get_sample() for all 3 DataSources in a ThreadPoolExecutor #38

JackKelly opened this issue Jul 2, 2021 · 0 comments

Comments

@JackKelly
Copy link
Member

JackKelly commented Jul 2, 2021 •
edited
Loading

Should be easy because threads can create other threads. So just have nested threads.
@JackKelly JackKelly changed the title Run all get_sample for all 3 DataSources in a ThreadPoolExecutor Run get_sample() for all 3 DataSources in a ThreadPoolExecutor Jul 4, 2021
JackKelly added a commit that referenced this issue Jul 6, 2021
@JackKelly
Refactor: Now each DataSource can have its own get_batch() logic.
2f71b7d 
NWPDataSource lazily assembles its batch and then dask.compute() can
optimise the loading.

SatelliteDataSource doesn't use dask.  Instead it uses
ThreadPoolExecutor.

PVDataSource doesn't use dask or threads!  (Because the entire dataset
easily fits into memory!)

Tests pass.

Issues #38 #42 #20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JackKelly