Skip to content

Truncate long lines in repr of Dataset.attrs #1319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Zac-HD opened this issue Mar 23, 2017 · 5 comments · Fixed by #1322
Closed

Truncate long lines in repr of Dataset.attrs #1319

Zac-HD opened this issue Mar 23, 2017 · 5 comments · Fixed by #1322

Comments

@Zac-HD
Copy link
Contributor

Zac-HD commented Mar 23, 2017

When loading from NetCDF, Dataset.attrs often has a few long strings, which may even have embedded newlines (eg a multi-paragraph summary or references section). It's lovely that these are available, but they tend to make the repr very long and poorly formatted - to the point that many Jupyter notebooks begin by discarding the attrs, which makes it rather pointless to store or display metadata at all!

Given that these values are already truncated at 500 characters (including the indicative ..., but not the start point), I propose that they should instead be truncated to 80 characters including the indentation and key (as values are). For the sake of pretty-printing, this should also replace newlines or tabs with spaces and truncate early if an empty line is encountered.

Another solution would be add appropriate indentation following newlines or wrapping, so that the structure remains clear. However, I think that it is better to print a fairly minimal representation of the metadata by default.

>>> xr.open_dataset('http://dapds00.nci.org.au/thredds/dodsC/uc0/rs0_dev/20170215-stacked_sample/LS7_ETM_NBART_3577_15_-40.ncml')

<xarray.Dataset>
Dimensions:  (time: 246, x: 4000, y: 4000)
Coordinates:
  * y        (y) float64 -3.9e+06 -3.9e+06 -3.9e+06 -3.9e+06 -3.9e+06 ...
  * x        (x) float64 1.5e+06 1.5e+06 1.5e+06 1.5e+06 1.5e+06 1.5e+06 ...
  * time     (time) datetime64[ns] 1999-07-16T23:49:39 1999-07-25T23:43:07 ...
Data variables:
    crs      int32 ...
    blue     (time, y, x) float64 ...
    green    (time, y, x) float64 ...
    red      (time, y, x) float64 ...
    nir      (time, y, x) float64 ...
    swir1    (time, y, x) float64 ...
    swir2    (time, y, x) float64 ...
Attributes:
    date_created: 2017-03-07T11:57:26.511217
    Conventions: CF-1.6, ACDD-1.3
    history: 2017-03-07T11:57:26.511307+11:00 adh547 datacube-ncml (1.2.2+23.gd1f3512.dirty) ls7_nbart_albers.yaml, 1.0.6a, /short/v10/datacube/002/LS7_ETM_NBART/LS7_ETM_NBART_3577_15_-40.ncml, (15, -40)  # Created NCML file to aggregate multiple NetCDF files along the time dimension
    geospatial_bounds: POLYGON ((148.49626113888138 -34.828378308133452,148.638689676063308 -35.720318326735864,149.734176111491877 -35.599556747691196,149.582601578289143 -34.708911907843387,148.49626113888138 -34.828378308133452))
    geospatial_bounds_crs: EPSG:4326
    geospatial_lat_min: -35.7203183267
    geospatial_lat_max: -34.7089119078
    geospatial_lat_units: degrees_north
    geospatial_lon_min: 148.496261139
    geospatial_lon_max: 149.734176111
    geospatial_lon_units: degrees_east
    comment: -	Ground Control Points (GCP): new GCP chips released by USGS in Dec 2015 are used for re-processing
-	Geometric QA: each product undergoes geometric assessment and the assessment result will be recorded within v2 AGDC for filtering/masking purposes.
-	Processing parameter settings: the minimum number of GCPs for Ortho-rectified product generation has been reduced from 30 to 10.
-	DEM: 1 second SRTM DSM is used for Ortho-rectification.
-	Updated Calibration Parameter File (CPF): the latest/cu...
    product_suite: Surface Reflectance NBAR+T 25m
    publisher_email: [email protected]
    keywords_vocabulary: GCMD
    product_version: 2
    cdm_data_type: Grid
    references: -	Berk, A., Anderson, G.P., Acharya, P.K., Hoke, M.L., Chetwynd, J.H., Bernstein, L.S., Shettle, E.P., Matthew, M.W., and Adler-Golden, S.M. (2003) Modtran 4 Version 3 Revision 1 User s manual. Airforce Research Laboratory, Hanscom, MA, USA.
-	Chander, G., Markham, B.L., and Helder, D.L. (2009) Summary of current radiometric calibration coefficients for Landsat MSS, TM, ETM+, and EO-1 ALI sensors. Remote Sensing of Environment 113, 893-903.
-	Edberg, R., and Oliver, S. (2013) Projection-Indep...
    platform: LANDSAT-7
    keywords: AU/GA,NASA/GSFC/SED/ESD/LANDSAT,REFLECTANCE,ETM+,TM,OLI,EARTH SCIENCE
    publisher_name: Section Leader, Operations Section, NEMO, Geoscience Australia
    institution: Commonwealth of Australia (Geoscience Australia)
    acknowledgment: Landsat data is provided by the United States Geological Survey (USGS) through direct reception of the data at Geoscience Australias satellite reception facility or download.
    license: CC BY Attribution 4.0 International License
    title: Surface Reflectance NBAR+T 25 v2
    summary: Surface Reflectance (SR) is a suite of Earth Observation (EO) products from GA. The SR product suite provides standardised optical surface reflectance datasets using robust 
physical models to correct for variations in image radiance values due to atmospheric properties, and sun and sensor geometry. The resulting stack of surface reflectance
grids are consistent over space and time which is instrumental in identifying and quantifying environmental change. SR is based on radiance data from the...
    instrument: ETM
    source: LANDSAT 7 ETM+ surface observation
    publisher_url: http://www.ga.gov.au
@shoyer
Copy link
Member

shoyer commented Mar 23, 2017

OK, I'm not opposed to this. Truncating at 500 characters was basically a minimal sanity check -- we could certainly go further than this.

Anyone else have opinions here about sane default behavior? I don't work with files with lots of long attributes.

@pwolfram
Copy link
Contributor

As long as we can explicitly obtain the attrs data if necessary truncating the data for the repr makes sense. Note that appending a set of characters like ... to the end of it would be useful to indicate that the string continues but is only partially displayed.

@fmaussion
Copy link
Member

I am in favor of truncating the attrs too.

@shoyer
Copy link
Member

shoyer commented Mar 23, 2017

Sounds like there is support here.

@Zac-HD Any interest in putting together a pull request? See here for the existing logic:

xarray/xarray/core/formatting.py

Line 255 in b3fc6c4

def summarize_attr(key, value, col_width=None):

@Zac-HD
Copy link
Contributor Author

Zac-HD commented Mar 24, 2017

Sure, I'd be happy to. The above example will look much nicer, especially in wrapping environments:

<xarray.Dataset>
Dimensions:  (time: 246, x: 4000, y: 4000)
Coordinates:
  * y        (y) float64 -3.9e+06 -3.9e+06 -3.9e+06 -3.9e+06 -3.9e+06 ...
  * x        (x) float64 1.5e+06 1.5e+06 1.5e+06 1.5e+06 1.5e+06 1.5e+06 ...
  * time     (time) datetime64[ns] 1999-07-16T23:49:39 1999-07-25T23:43:07 ...
Data variables:
    crs      int32 ...
    blue     (time, y, x) float64 ...
    green    (time, y, x) float64 ...
    red      (time, y, x) float64 ...
    nir      (time, y, x) float64 ...
    swir1    (time, y, x) float64 ...
    swir2    (time, y, x) float64 ...
Attributes:
    date_created:           2017-03-07T11:57:26.511217
    Conventions:            CF-1.6, ACDD-1.3
    history:                2017-03-07T11:57:26.511307+11:00 adh547 datacube...
    geospatial_bounds:      POLYGON ((148.49626113888138 -34.828378308133452...
    geospatial_bounds_crs:  EPSG:4326
    geospatial_lat_min:     -35.7203183267
    geospatial_lat_max:     -34.7089119078
    geospatial_lat_units:   degrees_north
    geospatial_lon_min:     148.496261139
    geospatial_lon_max:     149.734176111
    geospatial_lon_units:   degrees_east
    comment:                -    Ground Control Points (GCP): new GCP chips ...
    product_suite:          Surface Reflectance NBAR+T 25m
    publisher_email:        [email protected]
    keywords_vocabulary:    GCMD
    product_version:        2
    cdm_data_type:          Grid
    references:             -    Berk, A., Anderson, G.P., Acharya, P.K., Ho...
    platform:               LANDSAT-7
    keywords:               AU/GA,NASA/GSFC/SED/ESD/LANDSAT,REFLECTANCE,ETM+...
    publisher_name:         Section Leader, Operations Section, NEMO, Geosci...
    institution:            Commonwealth of Australia (Geoscience Australia)
    acknowledgment:         Landsat data is provided by the United States Ge...
    license:                CC BY Attribution 4.0 International License
    title:                  Surface Reflectance NBAR+T 25 v2
    summary:                Surface Reflectance (SR) is a suite of Earth Obs...
    instrument:             ETM
    source:                 LANDSAT 7 ETM+ surface observation
    publisher_url:          http://www.ga.gov.au

@Zac-HD Zac-HD mentioned this issue Mar 24, 2017
Merged
4 tasks
@max-sixty max-sixty mentioned this issue Feb 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Shorter repr for attributes Zac-HD/xarray
4 participants
@shoyer @pwolfram @fmaussion @Zac-HD