Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Experiment with ingesting NWP data from raw files, including relevant params at cloud heights #122

Closed
Tracked by #90
JackKelly opened this issue Sep 16, 2021 · 1 comment
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request

Comments

@JackKelly
Copy link
Member

JackKelly commented Sep 16, 2021

Try ingesting directly from the NetCDF files from the Met Office. If that doesn't work then create a new script for generating an intermediate Zarr file of NWP data.

All of "file 3" listed in this Met Office PDF look useful for helping to predict the evolution of clouds.

The "dream" would be that the user could request a list of NWP parameters when nowcasting_dataset runs, and the code will figure out which Met Office GRIB file to load from.

Related issues:

@JackKelly JackKelly added enhancement New feature or request data New data source or feature; or modification of existing data source labels Sep 16, 2021
@JackKelly JackKelly added this to the WP1 essential tasks milestone Sep 16, 2021
@peterdudfield peterdudfield removed this from the WP1 essential tasks milestone Sep 24, 2021
@flowirtz flowirtz moved this to Todo in Nowcasting Oct 15, 2021
@JackKelly JackKelly changed the title Get nowcasting_dataset to ingest NWP data from raw files, including relevant params at cloud heights Experiment with ingesting NWP data from raw files, including relevant params at cloud heights Oct 20, 2021
@JackKelly
Copy link
Member Author

Loading data from raw NWPs will be way too slow. It takes 1.8 seconds to open each grib file!

Here's the notebook: https://github.com/openclimatefix/nowcasting_dataset/blob/jack/experiment-reading-nwps/notebooks/load_raw_nwps_benchmark.ipynb

I'll start a new issue about converting NWP grib files to Zarr...

Repository owner moved this from Todo to Done in Nowcasting Nov 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
data New data source or feature; or modification of existing data source enhancement New feature or request
Projects
No open projects
Archived in project
Development

No branches or pull requests

2 participants