-
-
Notifications
You must be signed in to change notification settings - Fork 6
Re-create NWP Zarr with one chunk per init_time and step and small bit depth #30
Comments
Each of the four source Zarr stores have similar chunk structures:
|
ok, it's working with the first 1,000 init_times. I think the problem might be that each of the four individual Zarr stores ends with an init_time chunk of size 1; when all the other init_time chunks are of size 2. And that frightens rechunker. Perhaps I should bite the bullet and re-create from the original Met Office grib files. Need to do that some time. |
huh. Using float16 sometimes causes NaN losses (presumably because some of the data values are NaN???). Maybe use int16 (after scaling the original values to, say, [0, 1023].) |
Going from 1/4 spatial to full spatial extent actually increases speed to over 7.3 it/s (from 6.7 ish) |
Must figure out how to merge all 3 of James' NWP Zarrs (or re-create from the original files). Currently training on 6 months of data, which isn't great! |
Yay - it works if we load the 4 So we now have a single re-chunked Zarr which spans 2018-01 to 2019-12 with chunk sizes:
Filename: |
Still TODO: convert NWP to 16-bit ints, using 10 bits? |
This won't be necessary if #90 works :) |
Uh oh!
There was an error while loading. Please reload this page.
following on from #8
TODO:
The text was updated successfully, but these errors were encountered: