Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Try with chunks representing quarter of the spatial extent #22

Closed
JackKelly opened this issue Jun 19, 2021 · 5 comments
Closed

Try with chunks representing quarter of the spatial extent #22

JackKelly opened this issue Jun 19, 2021 · 5 comments
Assignees
@JackKelly

Comments

@JackKelly
Copy link
Member

Over the weekend, I (hopefully) created all_zarr_int16_single_timestep_quarter_spatial.zarr
@JackKelly
Copy link
Member Author

Makes loading slower because dask doesn't like having too many chunks to choose from! See #23

@JackKelly JackKelly self-assigned this Jun 21, 2021
@JackKelly JackKelly reopened this Jun 21, 2021
@JackKelly
Copy link
Member Author

re-opening because the fix for #23 possibly makes this relevant again

@JackKelly
Copy link
Member Author

JackKelly commented Jun 22, 2021 •
edited
Loading

Maybe it is a bit faster: 35 it/s after 30,000 it (with over 50% utilisation of GPU; and about 150 MB/s read speed)

@JackKelly
Copy link
Member Author

Even if dividing spatial extent isn't super-useful when reading a single satellite channel, dividing the spatial extent may become more important if / when using multiple satellite channels.

@JackKelly
Copy link
Member Author

Using full-spatial-extent, one-timestep-per-chunk, consolidated, we get 32.7 it/s. But at more than twice the network bandwidth: hovering around 350 - 390 MB/s.

JackKelly added a commit that referenced this issue Jun 22, 2021
@JackKelly
using quarter geospatial data provides a slight training speedup, whi…
17808a7 
…lst using less than half of the disk bandwidth! #22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JackKelly JackKelly

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JackKelly