fluview & fluview_clinical data stale?

[ian-bury-kinsa]

Is there a known data outage for these two data sets? Per https://delphi.cmu.edu/epidata/fluview_meta/, fluview data has not been updated since 2022-09-09.

[krivard]

The ILINet metadata claims the most recent epiweek available is 2022w38, which included the week from 2022-09-18--2022-09-24.

Our logs show that we pulled that very epiweek in as recently as this afternoon, but added no rows to the database as a result (which seems wrong) or possibly added negative rows (which seems worse).

I suspect something funky with the change in database platforms -- @neul3 @korlaxxalrok, can you combine your powers and dig into this? The FluView pipeline predates the pandemic so I'm afraid none of us are experts on the code, but I can give you what I've managed to unearth so far:

The relevant logs from this morning's run are on primary at

$ sed -n '/Starting step .id=157480/,/Success/p' ~automation/automation_20220914_125347.log

The relevant acquisition pipeline starts and ends in fluview_update.py, but pulls its meta data using routines in fluview.py. There's also a rough overview in the directory README.

It stores its intermediate files on primary at ~automation/driver/flu_data (careful, that directory has thousands of files in it). There are .zip files from today, which acquisition seems to be able to read without unzipping them first.

The tables you want are epidata.fluview* but there's not much there that we don't already know: last touched on Sept 9.

mysql> select max(issue) from fluview;
+------------+
| max(issue) |
+------------+
|     202235 |
+------------+
1 row in set (0.01 sec)

mysql> select max(release_date) from fluview;
+-------------------+
| max(release_date) |
+-------------------+
| 2022-09-09        |
+-------------------+
1 row in set (0.00 sec)

[brookslogan]

Note that this outage also affects ILI-Nearby (nowcast), which should be one week ahead of fluview. It also likely affects the fluview_public data (although I'm not sure this was ever surfaced as an API endpoint).

[dmytrotsko]

@ian-bury-kinsa Hey, this issue has been resolved. Thank you for reporting once again!