Skip to content

Consolidate base directory for diagnostics (obs, mapping and mask files) into one config option #500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 10, 2019
Merged

Consolidate base directory for diagnostics (obs, mapping and mask files) into one config option #500

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
11e638f
Add diagnostics config section
xylar Dec 2, 2018
9c796c4
Update example config file with new diagnostics section
xylar Dec 2, 2018
6f827b4
Add/fix dignostics sections in test config files
xylar Dec 2, 2018
06d559e
Make config.example and config.default more verbose
xylar Dec 4, 2018
f2b710a
Fix PEP8 errors
xylar Jan 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ used those components.
MPAS-Analysis is available as an anaconda package via the `e3sm` channel:

``` bash
conda install -c conda-forge -c e3sm mpas_analysis
conda create -n mpas_analysis -c conda-forge -c e3sm mpas_analysis
```

To use the latest version for developers, you will need to set up a conda
Expand All @@ -45,9 +45,9 @@ environment with the following packages:

These can be installed via the conda command:
``` bash
conda install -c conda-forge numpy scipy matplotlib netCDF4 xarray dask \
bottleneck basemap lxml nco pyproj pillow cmocean progressbar2 requests \
setuptools shapely
conda create -n mpas_analysis -c conda-forge numpy scipy matplotlib netCDF4 \
xarray dask bottleneck basemap lxml nco pyproj pillow cmocean \
progressbar2 requests setuptools shapely
```

Then, get the code from:
Expand All @@ -60,16 +60,16 @@ If you installed the `mpas_analysis` package, download the data that is
necessary to MPAS-Analysis by running:

``` bash
download_analysis_data -o /path/to/output/directory
download_analysis_data -o /path/to/mpas_analysis/diagnostics
```

If you are using the git repository, run:

``` bash
./download_analysis_data.py -o /path/to/output/directory
./download_analysis_data.py -o /path/to/mpas_analysis/diagnostics
```

where `/path/to/output/directory` is the main folder that will contain
where `/path/to/mpas_analysis/diagnostics` is the main folder that will contain
two subdirectories:

* `mpas_analysis`, which includes mapping and region mask files for
Expand All @@ -78,6 +78,10 @@ two subdirectories:
[Observations table](http://mpas-analysis.readthedocs.io/en/latest/observations.html)
and used to evaluate the model results

Once you have downloaded the analysis data, you will point to its location
(your equivalent of `path/to/mpas_analysis/diagnostics` above) in the config
option `baseDirectory` in the `[diagnostics]` section.

## List Analysis

If you installed the `mpas_analysis` package, list the available analysis tasks
Expand Down
55 changes: 41 additions & 14 deletions config.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ preprocessedReferenceRunName = None
# performed.
# mainRunConfigFile = /path/to/config/file


[execute]
## options related to executing parallel tasks

Expand All @@ -65,6 +66,29 @@ parallelTaskCount = 1
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = serial


[diagnostics]
## config options related to observations, mapping files and region files used
## by MPAS-Analysis in diagnostics computations.

# The base path to the diagnostics directory. Typically, this will be a shared
# directory on each E3SM supported machine (see the example config files for
# its location). For other machines, this would be the directory pointed to
# when running "download_analysis_data.py" to get the public observations,
# mapping files and region files.
baseDirectory = /path/to/diagnostics

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory. The user can supply an absolute
# path here to point to a path that is not within the baseDirectory above.
mappingSubdirectory = mpas_analysis/maps

# Directory for region mask files. The user can supply an absolute path here to
# point to a path that is not within the baseDirectory above.
regionMaskSubdirectory = mpas_analysis/region_masks


[input]
## options related to reading in the results to be analyzed

Expand Down Expand Up @@ -93,11 +117,6 @@ seaIceStreamsFileName = streams.seaice
# names of ocean and sea ice meshes (e.g. oEC60to30, oQU240, oRRS30to10, etc.)
mpasMeshName = mesh

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory
# mappingDirectory = /dir/for/mapping/files


[output]
## options related to writing out plots, intermediate cached data sets, logs,
Expand Down Expand Up @@ -175,10 +194,13 @@ startYear = 1
endYear = 9999

[oceanObservations]
## options related to ocean observations with which the results will be compared
## options related to ocean observations with which the results will be
## compared

# directory where ocean observations are stored
baseDirectory = /dir/to/ocean/observations
# subdirectory within [diagnostics]/baseDirectory where ocean observations are
# stored. The user can supply an absolute path here to point to a path that is
# not within [diagnostics]/baseDirectory.
obsSubdirectory = observations/Ocean

[oceanPreprocessedReference]
## options related to preprocessed ocean reference run with which the results
Expand All @@ -191,8 +213,10 @@ baseDirectory = /dir/to/ocean/reference
## options related to sea ice observations with which the results will be
## compared

# directory where sea ice observations are stored
baseDirectory = /dir/to/seaice/observations
# subdirectory within [diagnostics]/baseDirectory where sea ice observations
# are stored. The user can supply an absolute path here to point to a path
# that is not within [diagnostics]/baseDirectory.
obsSubdirectory = observations/SeaIce

[seaIcePreprocessedReference]
## options related to preprocessed sea ice reference run with which the results
Expand All @@ -201,8 +225,11 @@ baseDirectory = /dir/to/seaice/observations
# directory where ocean reference simulation results are stored
baseDirectory = /dir/to/seaice/reference

[regions]
## options related to ocean regions used in several analysis modules
[icebergObservations]
## options related to iceberg observations with which the results will be
## compared

# Directory for region mask files
regionMaskDirectory = /path/to/masks/
# subdirectory within [diagnostics]/baseDirectory where iceberg observations
# are stored. The user can supply an absolute path here to point to a path
# that is not within [diagnostics]/baseDirectory.
obsSubdirectory = observations/Icebergs
35 changes: 11 additions & 24 deletions configs/acme1/config.20180129.DECKv1b_piControl.ne30_oEC.edison
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,17 @@ parallelTaskCount = 4
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = bck

[diagnostics]
## config options related to observations, mapping files and region files used
## by MPAS-Analysis in diagnostics computations.

# The base path to the diagnostics directory. Typically, this will be a shared
# directory on each E3SM supported machine (see the example config files for
# its location). For other machines, this would be the directory pointed to
# when running "download_analysis_data.py" to get the public observations,
# mapping files and region files.
baseDirectory = /space2/diagnostics

[input]
## options related to reading in the results to be analyzed

Expand All @@ -46,11 +57,6 @@ seaIceHistorySubdirectory = archive/ice/hist
# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oEC60to30v3

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory
mappingDirectory = /space2/diagnostics/mpas_analysis/maps

# names of namelist and streams files, either a path relative to baseDirectory
# or an absolute path.
oceanNamelistFileName = run/mpas-o_in
Expand Down Expand Up @@ -126,25 +132,6 @@ endYear = 10
startYear = 1
endYear = 10

[oceanObservations]
## options related to ocean observations with which the results will be compared

# directory where ocean observations are stored
baseDirectory = /space2/diagnostics/observations/Ocean/

[seaIceObservations]
## options related to sea ice observations with which the results will be
## compared

# directory where sea ice observations are stored
baseDirectory = /space2/diagnostics/observations/SeaIce

[regions]
## options related to ocean regions used in several analysis modules

# Directory for region mask files
regionMaskDirectory = /space2/diagnostics/mpas_analysis/region_masks

[streamfunctionMOC]
## options related to plotting the streamfunction of the meridional overturning
## circulation (MOC)
Expand Down
71 changes: 12 additions & 59 deletions configs/alcf/config.20171031.tenYearTest.GMPAS-IAF.T62_oEC60to30v3wLI.60layer.theta
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,17 @@ parallelTaskCount = 4
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = bck

[diagnostics]
## config options related to observations, mapping files and region files used
## by MPAS-Analysis in diagnostics computations.

# The base path to the diagnostics directory. Typically, this will be a shared
# directory on each E3SM supported machine (see the example config files for
# its location). For other machines, this would be the directory pointed to
# when running "download_analysis_data.py" to get the public observations,
# mapping files and region files.
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics

[input]
## options related to reading in the results to be analyzed

Expand All @@ -39,11 +50,6 @@ baseDirectory = /lus/theta-fs0/projects/OceanClimate_2/mpeterse/20171031.tenYear
# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oEC60to30v3wLI

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory
mappingDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/mpas_analysis/maps

# names of namelist and streams files, either a path relative to baseDirectory
# or an absolute path.
oceanNamelistFileName = mpas-o_in
Expand Down Expand Up @@ -112,29 +118,6 @@ endYear = 9999
startYear = 1
endYear = 9999

[oceanObservations]
## options related to ocean observations with which the results will be compared

# directory where ocean observations are stored
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/observations/Ocean

[seaIceObservations]
## options related to sea ice observations with which the results will be
## compared

# directory where sea ice observations are stored
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/observations/SeaIce

[timeSeriesSeaIceAreaVol]
## options related to plotting time series of sea ice area and volume

# plot on polar plot
polarPlot = False

[regions]
# Directory containing mask files for ocean basins and ice shelves
regionMaskDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/mpas_analysis/region_masks

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the section below removed from this config file (same goes for other sections in other config files)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the whole point of this PR. People shouldn't need to point to these directories manually now that the directory structure is standardized.

Are you thinking that people would need to make their own region masks and point to those directories? I guess that's conceivable, particularly for non-standard grids.

My thought was that we want the example config files to be relatively simple. I guess you prefer to keep them relatively verbose and to point out to users that they could point the mappingSubdirectory and/or regionMaskSubdirectory as an absolute path?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, in this case I only meant the section below that has other (unrelated) things in it.
I would only keep config.example more verbose.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. The way comments work on GitHub, all I see are the previous 3 lines to your comment so that wasn't clear. It could be that i need to put the MOC section back in but I don't think the rest of this is helpful. It just makes it a lot harder to change default options because all of the example config files have to be changed to match. I think the examples should be relatively lean.

Copy link
Collaborator

@milenaveneziani milenaveneziani Dec 4, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, agree that only the MOC section should be put back in for this specific config files, while the others are redundant/arbitrary.

[streamfunctionMOC]
## options related to plotting the streamfunction of the meridional overturning
## circulation (MOC)
Expand All @@ -145,34 +128,4 @@ regionMaskDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/mpas_a
# in its calculation, whereas the postprocessing script does.
# NOTE: this is a temporary option that will be removed once the online
# MOC takes into account the bolus velocity when GM is on.
usePostprocessingScript = True
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will put the MOC section back. That was removed by mistake.


[climatologyMapSoseTemperature]
## options related to plotting climatology maps of Antarctic
## potential temperature at various levels, including the sea floor against
## reference model results and SOSE reanalysis data

# Times for comparison times (Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct,
# Nov, Dec, JFM, AMJ, JAS, OND, ANN)
seasons = ['JFM', 'JAS', 'ANN']

# list of depths in meters (positive up) at which to analyze, 'top' for the
# sea surface, 'bot' for the sea floor
depths = ['top', -200, -400, -600, -800, 'bot']

[climatologyMapSoseSalinity]
## options related to plotting climatology maps of Antarctic
## salinity at various levels, including the sea floor against
## reference model results and SOSE reanalysis data

# Times for comparison times (Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct,
# Nov, Dec, JFM, AMJ, JAS, OND, ANN)
seasons = ['JFM', 'JAS', 'ANN']

# list of depths in meters (positive up) at which to analyze, 'top' for the
# sea surface, 'bot' for the sea floor
depths = ['top', -200, -400, -600, -800, 'bot']

[timeSeriesAntarcticMelt]
# a list of ice shelves to plot
#iceShelvesToPlot = ['Antarctica', 'Peninsula', 'West Antarctica', 'East Antarctica', 'Larsen_C', 'Filchner', 'Ronne', 'Filchner-Ronne', 'Brunt_Stancomb', 'Fimbul', 'Amery', 'Totten', 'Ross_West', 'Ross_East', 'Ross', 'Getz', 'Thwaites', 'Pine_Island', 'Abbot', 'George_VI']
usePostprocessingScript = True
65 changes: 11 additions & 54 deletions configs/alcf/config.20180410.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,17 @@ parallelTaskCount = 1
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = serial

[diagnostics]
## config options related to observations, mapping files and region files used
## by MPAS-Analysis in diagnostics computations.

# The base path to the diagnostics directory. Typically, this will be a shared
# directory on each E3SM supported machine (see the example config files for
# its location). For other machines, this would be the directory pointed to
# when running "download_analysis_data.py" to get the public observations,
# mapping files and region files.
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics

[input]
## options related to reading in the results to be analyzed

Expand All @@ -39,11 +50,6 @@ baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/azamatm/E3SM_simulations
# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oRRS18to6v3

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory
mappingDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/mpas_analysis/maps

# names of namelist and streams files, either a path relative to baseDirectory
# or an absolute path.
oceanNamelistFileName = mpas-o_in
Expand Down Expand Up @@ -111,52 +117,3 @@ endYear = 10
# For valid statistics, index times should include at least 30 years
startYear = 1
endYear = 9999

[oceanObservations]
## options related to ocean observations with which the results will be compared

# directory where ocean observations are stored
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/observations/Ocean

[seaIceObservations]
## options related to sea ice observations with which the results will be
## compared

# directory where sea ice observations are stored
baseDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/observations/SeaIce

[timeSeriesSeaIceAreaVol]
## options related to plotting time series of sea ice area and volume

# plot on polar plot
polarPlot = False

[regions]
# Directory containing mask files for ocean basins and ice shelves
regionMaskDirectory = /lus/theta-fs0/projects/ClimateEnergy_2/diagnostics/mpas_analysis/region_masks

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is another example of section (below this comment) that seems unrelated to this PR and that is being removed. Not sure if it was intentional or not.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really think we don't want this stuff in the example config files, so, yes, it was intentional.

[climatologyMapSoseTemperature]
## options related to plotting climatology maps of Antarctic
## potential temperature at various levels, including the sea floor against
## reference model results and SOSE reanalysis data

# Times for comparison times (Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct,
# Nov, Dec, JFM, AMJ, JAS, OND, ANN)
seasons = ['JFM', 'JAS', 'ANN']

# list of depths in meters (positive up) at which to analyze, 'top' for the
# sea surface, 'bot' for the sea floor
depths = ['top', -200, -400, -600, -800, 'bot']

[climatologyMapSoseSalinity]
## options related to plotting climatology maps of Antarctic
## salinity at various levels, including the sea floor against
## reference model results and SOSE reanalysis data

# Times for comparison times (Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct,
# Nov, Dec, JFM, AMJ, JAS, OND, ANN)
seasons = ['JFM', 'JAS', 'ANN']

# list of depths in meters (positive up) at which to analyze, 'top' for the
# sea surface, 'bot' for the sea floor
depths = ['top', -200, -400, -600, -800, 'bot']
Loading