Ingest numerical weather prediction data (NWP)

[JackKelly]

Use temperature at surface, precipitation, irradiance, cloud fraction, accumulated snow cover.

• Finish NWPDataSource
• Resample to 5-minutely
• Standardise
• Convert to float32
• Plot timeseries data just before data goes into ML model

[JackKelly]

todo: finish get_nwp_for_datetime() (just under PyTorch dataset heading)

[JackKelly]

Largely done now. Need to finish off NWPDataSource.

If it's too slow to load then maybe just pre-load the temperature data for each PV system. But that's not very flexible. Probably better to find a good way to load in a separate thread

[JackKelly]

Resample to 5-minutely and interpolate, whilst making sure the resulting target_datetimes are correct. Need to make sure there's always data available at the start and end for the interpolation.

Maybe try resampling to 5 minutely as soon as we load the data. But prob won't work. So prob need to resample after creating a single target_datetime index for the NWPs.

Also need to throw an error if there isn't data for a particular datetime. And/or filter datetimes before training.

[JackKelly]

Done with resampling just using xarray's built-in resampling :)

[JackKelly]

Actually, no, using the 'simple approach' is an order of magnitude slower (1.8 seconds vs about 180 ms) and produces discontinuities between different NWP inits. I'll interpolate manually...

[JackKelly]

New approach for getting NWPs is way faster (2 ms)!

[JackKelly]

If the pesky for batch in dataloader: thing still doesn't work after restarting the kernel, probably need each worker to load the NWP Zarr itself.

[JackKelly]

OK! The code runs past for batch in dataloader now. But now has a problem in get_nwp_example: nwp_selected = nwp.sel(init_time=init_time_indexer, step=step_indexer) where "not all values found in index 'step'". So probably need to pre-compute the correct datetimes; or handle the KeyError.

[JackKelly]

Code appears to be running now; but is super-slow. (with 1 worker, it's now 15 seconds per iteration. It used to be more like 40 iterations per second!)

Still need to standardise and convert to float32

Ideas for speeding up in openclimatefix/predict_pv_yield_2#18