Dataset Validation script

[JackKelly]

Detailed Description

A script which goes through all the pre-prepared batches and checks:

• That there's no overlap between train, test, and validation sets :)
• The every batch contains the fields we'd expect
• That every value is within the range we'd expect
• That the duration of each example is what we'd expect

Context

Hopefully our unit-tests will catch these bugs. But, just to be super-careful, it might be nice to also have a script which goes through the entire pre-prepared dataset and checks for these issues!

[peterdudfield]

I'll give this ago

[JackKelly]

Awesome, thank you!

[peterdudfield]

For every value being in the range we expect,
for pv and gsp data, would expect between 0 and 1, as we have normalized them
Should we come up with a list of criteria for the others too.

Maybe a later issue, would be to create a slightl better object that a Typed Dict (I'm a fan of pydantic classes), that could do this validation on the fly