Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Implement NWPDataSource #20

Closed
JackKelly opened this issue Jun 18, 2021 · 0 comments
Closed

Implement NWPDataSource #20

JackKelly opened this issue Jun 18, 2021 · 0 comments
Assignees
@JackKelly

Comments

@JackKelly
Copy link
Member

See notebooks/design.ipynb for ideas of interface.

Almost certainly will also have to re-create NWP Zarr (#11) so we load smaller files. e.g. each chunk should be a single init_time and a single step.
@JackKelly JackKelly self-assigned this Jun 18, 2021
JackKelly added a commit that referenced this issue Jul 5, 2021
@JackKelly
Implement NWPDataSource! #20 #3
dfc18ed
JackKelly added a commit that referenced this issue Jul 6, 2021
@JackKelly
Refactor: Now each DataSource can have its own get_batch() logic.
2f71b7d 
NWPDataSource lazily assembles its batch and then dask.compute() can
optimise the loading.

SatelliteDataSource doesn't use dask.  Instead it uses
ThreadPoolExecutor.

PVDataSource doesn't use dask or threads!  (Because the entire dataset
easily fits into memory!)

Tests pass.

Issues #38 #42 #20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JackKelly JackKelly

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JackKelly