Speed up GSPDataSource.get_locations()

[JackKelly]

If a constant number of GSPs are available across all timesteps then we don't need to go round a slow for loop. Instead we can just randomly pick GSPs in one go using something like np.random.choice(self.metadata["gsp_id"], size=len(t0_datetimes))

[peterdudfield]

Don't we need to make sure there are no nan's for that timeperiod. i.e for each t0 only pick GSP where there is data.
Perhaps ive missed something though ....

[JackKelly]

yeah, that's right, but we could do that as a one-off bit of analysis... i.e. manually check there are no NaNs in GSP data, then we can vastly speed up GSPDataSource.get_locations

[JackKelly]

I think this should come before #304 tbh, because this could be a simple win, with minimal code to modify

[JackKelly]

In practice, unfortunately it looks like there are NaNs in the GSP data, so we still go round the big (slow) loop:

2021-11-16 15:13:06,479 WARNING There are some nans in the gsp data, so to get x,y locations we have to do a big loop at /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/gsp/gsp_data_source.py#L142

Some thoughts on next steps:

1. Investigate the GSP data on disk by hand (e.g. in a Jupyter notebook). How are the NaNs distributed? Specifically:
   • Are there a small number of timesteps when all GSPs are NaN? If so, can we just throw away those timesteps?
   • Are there some GSPs which are always NaN? If so, can we throw them away?

[peterdudfield]

Thats a shame, what about #304 ?

[JackKelly]

I'd be keen to find an elegant solution to #305 (this issue) if we can.... parallelism can be complex to think about and debug 🙂

[JackKelly]

Hehe.. good news, @peterdudfield! Your speedy code for GSPDataSource.get_locations() is now being used, after PR #497 drops GSPs with zero PV (and I assume those GSPs also had NaNs!)

[peterdudfield]

Hehe.. good news, @peterdudfield! Your speedy code for GSPDataSource.get_locations() is now being used, after PR #497 drops GSPs with zero PV (and I assume those GSPs also had NaNs!)

thats good news - close issue?

[JackKelly]

yeah, let's close this for now! Great work!