Skip to content

Update date type handling in mpas_xarray #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 8, 2016
Merged

Update date type handling in mpas_xarray #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
804e53c
Update date type handling in mpas_xarray
xylar Nov 25, 2016
00ee576
Add unit testing for mpas_xarray
xylar Dec 6, 2016
bb35611
Add dask to CI
xylar Dec 6, 2016
29e2e84
Checks selection for mpas_xarray
pwolfram Dec 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/requirements-py27.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ dependencies:
- lxml
- xarray
- matplotlib
- dask
18 changes: 9 additions & 9 deletions mpas_analysis/ocean/ocean_modelvsobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@ def ocn_modelvsobs(config, field):
if field.lower() == 'mld':
obs_filename = "%s/holtetalley_mld_climatology.nc" % obsdir

ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True,
timestr='time_avg_daysSinceStartOfSim',
onlyvars=['time_avg_dThreshMLD']))
ds = xr.open_mfdataset(infiles, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset,
timestr=['xtime_start', 'xtime_end'],
onlyvars=['time_avg_dThreshMLD']))
ds = remove_repeated_time_index(ds)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize that splitting the lambda x: and process_mpas(... lines here and elsewhere is not optimal. I have not found a more elegant solution that is PEP8 compliant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to find one either @xylar. The way you have done this is much clearer so this is great forward progress we all should be very happy about. Thanks for taking another look.

ds.rename({'time_avg_dThreshMLD':'mpasData'}, inplace = True)

Expand All @@ -93,11 +93,11 @@ def ocn_modelvsobs(config, field):

elif field.lower() == 'sst':

ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True,
timestr='time_avg_daysSinceStartOfSim',
onlyvars=['time_avg_activeTracers_temperature'],
selvals={'nVertLevels':1}))
ds = xr.open_mfdataset(infiles, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset,
timestr=['xtime_start', 'xtime_end'],
onlyvars=['time_avg_activeTracers_temperature'],
selvals={'nVertLevels':1}))
ds = remove_repeated_time_index(ds)
ds.rename({'time_avg_activeTracers_temperature':'mpasData'}, inplace = True)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make things backward compatible, it seems like we need a list of options for the timestr variable as well: did you want to handle this in the specific PR that will address #20?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milenaveneziani, yes that's exactly right. What I realized in the process of making this PR is that timestr will need to be a special case in the PR for addressing #20 because we want to support both the possibility of a single variable (e.g. xtime or daysSinceStartOfSim) and that of a pair of variable names like in this case. That will be a bit trickier to implement, and may require tweaking the config reader to handle a list of lists (if it doesn't already) but it should be doable. Again, this is a good case to run into now so we build in some more sophisticated functionality that will hopefully be useful for the future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xylar: this is what I was referring to in my comments to #48. But I guess I was wrong, because xtime_start and xtime_end are already supported in alpha.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Yes, this is backwards compatible at least back to alpha7.

Expand Down
34 changes: 22 additions & 12 deletions mpas_analysis/ocean/ohc_timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def ohc_timeseries(config):
Performs analysis of ocean heat content (OHC) from time-series output.

Author: Xylar Asay-Davis, Milena Veneziani
Last Modified: 10/26/2016
Last Modified: 11/25/2016
"""

# read parameters from config file
Expand Down Expand Up @@ -50,6 +50,8 @@ def ohc_timeseries(config):
plots_dir = config.get('paths','plots_dir')

yr_offset = config.getint('time','yr_offset')
timeseries_yr1 = yr_offset + config.getint('time', 'timeseries_yr1')
timeseries_yr2 = yr_offset + config.getint('time', 'timeseries_yr2')

N_movavg = config.getint('ohc_timeseries','N_movavg')

Expand All @@ -75,20 +77,27 @@ def ohc_timeseries(config):

# Load data
print " Load ocean data..."
ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True,
timestr='time_avg_daysSinceStartOfSim',
onlyvars=['time_avg_avgValueWithinOceanLayerRegion_avgLayerTemperature',
'time_avg_avgValueWithinOceanLayerRegion_sumLayerMaskValue',
'time_avg_avgValueWithinOceanLayerRegion_avgLayerArea',
'time_avg_avgValueWithinOceanLayerRegion_avgLayerThickness']))

varList = ['time_avg_avgValueWithinOceanLayerRegion_avgLayerTemperature',
'time_avg_avgValueWithinOceanLayerRegion_sumLayerMaskValue',
'time_avg_avgValueWithinOceanLayerRegion_avgLayerArea',
'time_avg_avgValueWithinOceanLayerRegion_avgLayerThickness']
ds = xr.open_mfdataset(infiles,
preprocess=lambda x:
preprocess_mpas(x,
yearoffset=yr_offset,
timestr=['xtime_start','xtime_end'],
onlyvars=varList))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice cleaning up!


ds = remove_repeated_time_index(ds)

# select only the data in the specified range of years
# time_start = datetime.datetime(timeseries_yr1, 1, 1)
# time_end = datetime.datetime(timeseries_yr2, 12, 31)
# ds = ds.sel(Time=slice(time_start, time_end))

# Select year-1 data and average it (for later computing anomalies)
time_start = datetime.datetime(yr_offset+1,1,1)
time_end = datetime.datetime(yr_offset+1,12,31)
time_start = datetime.datetime(timeseries_yr1, 1, 1)
time_end = datetime.datetime(timeseries_yr1, 12, 31)
ds_yr1 = ds.sel(Time=slice(time_start,time_end))
mean_yr1 = ds_yr1.mean('Time')

Expand All @@ -107,7 +116,8 @@ def ohc_timeseries(config):
if ref_casename_v0 != "None":
print " Load in OHC for ACMEv0 case..."
infiles_v0data = "".join([indir_v0data,'/OHC.',ref_casename_v0,'.year*.nc'])
ds_v0 = xr.open_mfdataset(infiles_v0data,preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = xr.open_mfdataset(infiles_v0data, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = remove_repeated_time_index(ds_v0)
ds_v0_tslice = ds_v0.sel(Time=slice(time_start,time_end))

Expand Down
10 changes: 6 additions & 4 deletions mpas_analysis/ocean/sst_timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,10 @@ def sst_timeseries(config):
iregions = config.getlist('sst_timeseries','regionIndicesToPlot',listType=int)

# Load data:
ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True, timestr='time_avg_daysSinceStartOfSim',
onlyvars=['time_avg_avgValueWithinOceanRegion_avgSurfaceTemperature']))
ds = xr.open_mfdataset(infiles, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset,
timestr=['xtime_start', 'xtime_end'],
onlyvars=['time_avg_avgValueWithinOceanRegion_avgSurfaceTemperature']))
ds = remove_repeated_time_index(ds)

SSTregions = ds.time_avg_avgValueWithinOceanRegion_avgSurfaceTemperature
Expand All @@ -65,7 +66,8 @@ def sst_timeseries(config):
if ref_casename_v0 != "None":
print " Load in SST for ACMEv0 case..."
infiles_v0data = "".join([indir_v0data,'/SST.',ref_casename_v0,'.year*.nc'])
ds_v0 = xr.open_mfdataset(infiles_v0data,preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = xr.open_mfdataset(infiles_v0data,preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = remove_repeated_time_index(ds_v0)
ds_v0_tslice = ds_v0.sel(Time=slice(time_start,time_end))

Expand Down
10 changes: 5 additions & 5 deletions mpas_analysis/sea_ice/modelvsobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ def seaice_modelvsobs(config):

# Load data
print " Load sea-ice data..."
ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True,
timestr='timeSeriesStatsMonthly_avg_daysSinceStartOfSim_1',
onlyvars=['timeSeriesStatsMonthly_avg_iceAreaCell_1',
'timeSeriesStatsMonthly_avg_iceVolumeCell_1']))
ds = xr.open_mfdataset(infiles, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset,
timestr='timeSeriesStatsMonthly_avg_daysSinceStartOfSim_1',
onlyvars=['timeSeriesStatsMonthly_avg_iceAreaCell_1',
'timeSeriesStatsMonthly_avg_iceVolumeCell_1']))
ds = remove_repeated_time_index(ds)

# Compute climatologies (first motnhly and then seasonally)
Expand Down
34 changes: 17 additions & 17 deletions mpas_analysis/sea_ice/timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,11 @@ def seaice_timeseries(config):
dsmesh = xr.open_dataset(meshfile)

# Load data
ds = xr.open_mfdataset(infiles, preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset,
timeSeriesStats=True,
timestr='timeSeriesStatsMonthly_avg_daysSinceStartOfSim_1',
onlyvars=['timeSeriesStatsMonthly_avg_iceAreaCell_1',
'timeSeriesStatsMonthly_avg_iceVolumeCell_1']))
ds = xr.open_mfdataset(infiles, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset,
timestr='timeSeriesStatsMonthly_avg_daysSinceStartOfSim_1',
onlyvars=['timeSeriesStatsMonthly_avg_iceAreaCell_1',
'timeSeriesStatsMonthly_avg_iceVolumeCell_1']))
ds = remove_repeated_time_index(ds)

ds = ds.merge(dsmesh)
Expand All @@ -88,8 +88,8 @@ def seaice_timeseries(config):

if ref_casename_v0 != "None":
infiles_v0data = "".join([indir_v0data,'/icevol.',ref_casename_v0,'.year*.nc'])
ds_v0 = xr.open_mfdataset(infiles_v0data,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = xr.open_mfdataset(infiles_v0data, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_v0_tslice = ds_v0.sel(Time=slice(time_start,time_end))

# Make Northern and Southern Hemisphere partition:
Expand Down Expand Up @@ -165,31 +165,31 @@ def seaice_timeseries(config):
if varname == "iceAreaCell":

if compare_with_obs:
ds_obs = xr.open_mfdataset(obs_filenameNH,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = xr.open_mfdataset(obs_filenameNH, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = remove_repeated_time_index(ds_obs)
var_nh_obs = ds_obs.IceArea
var_nh_obs = replicate_cycle(var_nh,var_nh_obs)

ds_obs = xr.open_mfdataset(obs_filenameSH,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = xr.open_mfdataset(obs_filenameSH, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = remove_repeated_time_index(ds_obs)
var_sh_obs = ds_obs.IceArea
var_sh_obs = replicate_cycle(var_sh,var_sh_obs)

if ref_casename_v0 != "None":
infiles_v0data = "".join([indir_v0data,'/icearea.',ref_casename_v0,'.year*.nc'])
ds_v0 = xr.open_mfdataset(infiles_v0data,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = xr.open_mfdataset(infiles_v0data, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_v0_tslice = ds_v0.sel(Time=slice(time_start,time_end))
var_nh_v0 = ds_v0_tslice.icearea_nh
var_sh_v0 = ds_v0_tslice.icearea_sh

elif varname == "iceVolumeCell":

if compare_with_obs:
ds_obs = xr.open_mfdataset(obs_filenameNH,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = xr.open_mfdataset(obs_filenameNH, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_obs = remove_repeated_time_index(ds_obs)
var_nh_obs = ds_obs.IceVol
var_nh_obs = replicate_cycle(var_nh,var_nh_obs)
Expand All @@ -198,8 +198,8 @@ def seaice_timeseries(config):

if ref_casename_v0 != "None":
infiles_v0data = "".join([indir_v0data,'/icevol.',ref_casename_v0,'.year*.nc'])
ds_v0 = xr.open_mfdataset(infiles_v0data,
preprocess=lambda x: preprocess_mpas(x, yearoffset=yr_offset))
ds_v0 = xr.open_mfdataset(infiles_v0data, preprocess=lambda x:
preprocess_mpas(x, yearoffset=yr_offset))
ds_v0_tslice = ds_v0.sel(Time=slice(time_start,time_end))
var_nh_v0 = ds_v0_tslice.icevolume_nh
var_sh_v0 = ds_v0_tslice.icevolume_sh
Expand Down
Loading