Download additional UK Met Office UKV NWPs

[JackKelly]

Detailed Description

We already have several years of UKV NWPs. But we should grab the most recent data, so it covers the same timespan as the ESO PV forecasts.

Possible Implementation

Use FileZilla to FTP from CEDA to leonardo. (I won't install FileZilla on leonardo because that will also install xorg etc. Instead I'll run FileZilla on my NUC, and mount leonardo on the NUC. If this doesn't work, consider using LFTP running on leonardo)

[JackKelly]

This seems to work in /etc/fstab to mount leonardo on my NUC:

[email protected]:/    /mnt/leonardo    fuse.sshfs    rw,user,exec,allow_other    0    0

I'm not sure if rw,user,exec are necessary. allow_other is definitely necessary.

[JackKelly]

Not using FileZilla. Instead, using wget running on leonardo (in a screen session):

The command below ignores:

• Wholesale5 files (which are just static orthographic files)
• Files initalisated at 03, 09, 15, and 21 hours.
• T120 files.

Run this in the local directory for 2021 data:

wget --user=<username> --password=<password> 
--recursive -nH --cut-dirs=5 --no-clobber 
--reject-regex "[[:digit:]]{8}(03|09|15|21)00.*\.grib$" 
--reject-regex "T120\.grib$" 
--reject-regex "Wholesale5.*\.grib$"
ftp://ftp.ceda.ac.uk/badc/ukmo-nwp/data/ukv-grib/2021

[JackKelly]

All UKV NWP data should be about 24 TBytes for:

• 2016 to 2021
• 4 init times per day (00, 06, 12, 18)
• Wholesale1, 2, 3, and 4
• Ignoring the T120 files

[JackKelly]

leonardo has been downloading UKV NWPs for about a day now. It looks like it can download about 6 months of data per day. I'm starting with 2021, then 2020, then 2019, etc...

[JackKelly]

Finished downloading raw NWP data for 2016 to 2021. Folder sizes:

jack@leonardo:/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/UK_Met_Office/UKV/native$ du -h --max-depth=1 .
2.0T    ./2016
3.5T    ./2017
4.0T    ./2018
4.0T    ./2019
3.9T    ./2020
3.0T    ./2021
21T     .