wrong value of daysSinceStartOfSim written out on restart by globalStats am

[xylar]

In ACME, (but I was not able to reproduce this in standalone MPAS-O so far) if globalStats is set to write out results on startup, it writes out an incorrect value. Here is an example taken from @mark-petersen's G-compset run in the following folder on LANL IC:

$ pwd
/lustre/scratch1/turquoise/mpeterse/ACME/cases/a11o/run
$ ncdump -v daysSinceStartOfSim am.mpas-o.globalStats.0001-08-01.nc
...
 daysSinceStartOfSim = 0.00245346257716049, 213, 214, 215, 216, 217, 218, 
    219, 220, 221, 222, 223, 224, 225 ;

I have tried to reproduce this problem with both intel and gnu compilers, and both on my laptop and on wolf, and have not seen the problem. @douglasjacobsen attempted to fix the problem with #1090 but I determined that this would not be expected to fix the problem (daysSinceStartOfSim was not being used before it was computed, as that PR supposed) and indeed that PR introduced a bug where simulationStartTime was reset to the current time at each restart.

[xylar]

@mark-petersen, perhaps you can at least post here what ACME branch you're using, what compilers, etc. so I can try to reproduce this bug and hopefully fix it.

[mark-petersen]

Thanks, @xylar. I've seen this bug consistently the last few months, but this particular example is with:

ACME commit 186b887 (about alpha 7)
in /turquoise/usr/projects/climate/mpeterse/acme_repo/EC60to30km_with_land_ice_160926
run created with:
create_newcase -case $CASE_ROOT/$ACME_CASE -compset GMPAS_IAF -mach wolf -res T62_oEC60to30wLI -compiler gnu -mpilib openmpi -project s11_climateacme

I will try to look at this, as am running ACME regularly.

[xylar]

@mark-petersen, I have been working on MPAS analysis. We sometimes use time_avg_daysSinceStartOfSim, which has been contaminated with this same bug. While we are trying to move to using xtime_start and xtime_end, it would be helpful for analysis if daysSinceStartOfSim were correct. The error amounts to differences on the order of 5 days over a 10-year run, which really can affect the analysis and also makes debugging more challenging.

All that is to say, we should look into this bug further when we have time.

[mark-petersen]

@xylar that's good to know. I'm afraid I need to work on removing land-locked cells this week. I'll try to do it next week, I agree that it is important.