Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 #2757

pletchm · 2019-02-08T21:59:36Z

This pull request enhances the expand_dims method for both Dataset and DataArray objects to support inserting/broadcasting dimensions with size > 1. It corresponds to this issue #2710.

Changes:

dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions
dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions
Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg
Add expand_dims enhancement from issue 2710 to whats-new.rst

Included:

Tests added
Fully documented, including whats-new.rst for all changes and api.rst for new API

What's new:

All of the old functionality is still there, so it shouldn't break anyone's existing code that uses it.

You can now pass a dim as a dict, where the keys are the new dimensions and the values are either integers (giving the length of the new dimensions) or iterables (giving the coordinates of the new dimensions).

import numpy as np
import xarray as xr

>>> original = xr.Dataset({'x': ('a', np.random.randn(3)),
                                  'y': (['b', 'a'], np.random.randn(4, 3))},
                                  coords={'a': np.linspace(0, 1, 3),
                                                'b': np.linspace(0, 1, 4),
                                                'c': np.linspace(0, 1, 5)},
                                  attrs={'key': 'entry'})
>>> original
<xarray.Dataset>
Dimensions:  (a: 3, b: 4, c: 5)
Coordinates:
  * a        (a) float64 0.0 0.5 1.0
  * b        (b) float64 0.0 0.3333 0.6667 1.0
  * c        (c) float64 0.0 0.25 0.5 0.75 1.0
Data variables:
    x        (a) float64 -1.556 0.2178 0.6319
    y        (b, a) float64 0.5273 0.6652 0.3418 1.858 ... -0.3519 0.8088 0.8753
Attributes:
    key:      entry
>>> original.expand_dims({"d": 4, "e": ["l", "m", "n"]})
<xarray.Dataset>
Dimensions:  (a: 3, b: 4, c: 5, d: 4, e: 3)
Coordinates:
  * e        (e) <U1 'l' 'm' 'n'
  * a        (a) float64 0.0 0.5 1.0
  * b        (b) float64 0.0 0.3333 0.6667 1.0
  * c        (c) float64 0.0 0.25 0.5 0.75 1.0
Dimensions without coordinates: d
Data variables:
    x        (d, e, a) float64 -1.556 0.2178 0.6319 ... -1.556 0.2178 0.6319
    y        (d, e, b, a) float64 0.5273 0.6652 0.3418 ... -0.3519 0.8088 0.8753
Attributes:
    key:      entry

Or, equivalently, you can pass the new dimensions as kwargs instead of a dictionary.

original.expand_dims(d=4, e=["l", "m", "n"])

pep8speaks · 2019-02-08T21:59:42Z

Hello @pletchm! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-03-26 01:00:29 UTC

fujiisoup

Thanks for sending a PR. I also think it is a nice feature.
Some minor comments

xarray/core/dataset.py

xarray/tests/test_dataset.py

xarray/core/dataset.py

xarray/tests/test_dataarray.py

xarray/core/dataset.py

xarray/tests/test_dataarray.py

…ns with size>1 (pydata#2757) * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

pletchm · 2019-02-26T17:46:00Z

Hi @shoyer and @fujiisoup, I believe I've made all of the updates that you suggested in your comments. How is the PR looking now?
Thank you!

xarray/core/dataarray.py

…ns with size>1 (pydata#2757) * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

…a#2721) * Quarter offset implemented (base is now latest pydata-master). * Fixed issues raised in review (pydata#2721 (review)) * Updated whats-new.rst with info on quarter offset support. * Updated whats-new.rst with info on quarter offset support. * Update doc/whats-new.rst Co-Authored-By: jwenfai <[email protected]> * Added support for quarter frequencies when resampling CFTimeIndex. Less redundancy in CFTimeIndex resampling tests. * Removed normalization code (unnecessary for cftime_range) in cftime_offsets.py. Removed redundant lines in whats-new.rst. * Removed invalid option from _get_day_of_month docstring. Added tests back in that raises ValueError when resampling (base=24 when resampling to daily freq, e.g., '8D'). * Minor edits to docstrings/comments * lint

* ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation

* Added tests of desired name inferring behaviour * Infers names * updated what's new

It got deprecated in numpy 1.16 and throws a ton of warnings due to that. All the function does is returning .item() anyway, which is why it got deprecated.

* Friendlier io title. * Fix lists. * Fix *args, **kwargs "inline emphasis..." * misc * Reference xarray_extras for csv writing. Closes pydata#2289 * Add metpy accessor. Closes pydata#461 * fix transpose docstring. Closes pydata#2576 * Revert "Fix lists." This reverts commit 39983a5. * Revert "Fix *args, **kwargs" This reverts commit 1b9da35. * Add MetPy to related projects. * Add Weather and Climate specific page. * Add hvplot. * Note open_dataset, mfdataset open files as read-only (closes pydata#2345). * Update metpy 1 Co-Authored-By: dcherian <[email protected]> * Update doc/weather-climate.rst Co-Authored-By: dcherian <[email protected]>

…data#2810) These aren't valid for netCDF files. Fixes GH2803

0.12 will already have a big change in dropping Python 2.7 support. I'd rather wait a bit longer to finalize these deprecations to minimize the impact on users.

* attempt at loading remote hdf5 * added a couple tests * rewind bytes after reading header * addressed comments for tests and error message * fixed pep8 formatting * created _get_engine_from_magic_number function, new tests * added description in whats-new * fixed test failure on windows * same error on windows and nix

…ns with size>1 (pydata#2757) * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

pletchm · 2019-03-25T15:32:53Z

@shoyer, dim_kwargs is no longer allowed to be used for python 3.5. Does anything else need to change in this PR?

shoyer

One small note, otherwise looks good to me!

xarray/core/dataset.py

…ns with size>1 (pydata#2757) * use .size attribute to determine the size of a dimension, rather than converting to a list, which can be slow for large iterables * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

…etchm/xarray into feature/expand-dims-broadcast

doc/whats-new.rst

…ns with size>1 (pydata#2757) * Move enhancement description up to 0.12.1 * use .size attribute to determine the size of a dimension, rather than converting to a list, which can be slow for large iterables * Make using dim_kwargs for python 3.5 illegal -- a ValueError is thrown * dataset.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * dataarray.expand_dims() method take dict like object where values represent length of dimensions or coordinates of dimesnsions * Add alternative option to passing a dict to the dim argument, which is now an optional kwarg, passing in each new dimension as its own kwarg * Add expand_dims enhancement from issue 2710 to whats-new.rst * Fix test_dataarray.TestDataArray.test_expand_dims_with_greater_dim_size tests to pass in python 3.5 using ordered dicts instead of regular dicts. This was needed because python 3.5 and earlier did not maintain insertion order for dicts * Restrict core logic to use 'dim' as a dict--it will be converted into a dict on entry if it is a str or a sequence of str * Don't cast dim values (coords) as a list since IndexVariable/Variable will internally convert it into a numpy.ndarray. So just use IndexVariable((k,), v) * TypeErrors should be raised for invalid input types, rather than ValueErrors. * Force 'dim' to be OrderedDict for python 3.5

…etchm/xarray into feature/expand-dims-broadcast

shoyer · 2019-03-26T02:42:11Z

Thanks @pletchm!

pletchm force-pushed the feature/expand-dims-broadcast branch from bc7db5a to 64eba1d Compare February 9, 2019 08:15

fujiisoup reviewed Feb 10, 2019

View reviewed changes

shoyer reviewed Feb 11, 2019

View reviewed changes

pletchm force-pushed the feature/expand-dims-broadcast branch from f127d76 to e76c7d8 Compare February 14, 2019 05:56

shoyer reviewed Feb 27, 2019

View reviewed changes

xarray/core/dataarray.py Show resolved Hide resolved

jwenfai and others added 14 commits March 21, 2019 08:31

Add Dataset.drop_dims (pydata#2767)

ab268de

* ENH: Add Dataset.drop_dims() * Drops full dimensions and any corresponding variables in a Dataset * Fixes GH1949 * DOC: Add Dataset.drop_dims() documentation

Improve name concat (pydata#2792)

b393754

* Added tests of desired name inferring behaviour * Infers names * updated what's new

Don't use deprecated np.asscalar() (pydata#2800)

872b49c

It got deprecated in numpy 1.16 and throws a ton of warnings due to that. All the function does is returning .item() anyway, which is why it got deprecated.

Add support for cftime.datetime coordinates with coarsen (pydata#2778)

849eb18

Drop failing tests writing multi-dimensional arrays as attributes (py…

526a395

…data#2810) These aren't valid for netCDF files. Fixes GH2803

Push back finalizing deprecations for 0.12 (pydata#2809)

9f00c6f

0.12 will already have a big change in dropping Python 2.7 support. I'd rather wait a bit longer to finalize these deprecations to minimize the impact on users.

Release 0.12.0

ccb198f

Add whats-new for 0.12.1

6ec9910

Rework whats-new for 0.12

4ce03c2

DOC: Update donation links

97fdb83

DOC: remove outdated warning (pydata#2818)

a74ecd6

pletchm force-pushed the feature/expand-dims-broadcast branch from 4d4f403 to 21fa6e0 Compare March 21, 2019 16:21

Merge branch 'master' into feature/expand-dims-broadcast

78608d4

shoyer approved these changes Mar 25, 2019

View reviewed changes

xarray/core/dataset.py Outdated Show resolved Hide resolved

Merge branch 'feature/expand-dims-broadcast' of https://github.com/pl…

1ffc64e

…etchm/xarray into feature/expand-dims-broadcast

shoyer reviewed Mar 26, 2019

View reviewed changes

doc/whats-new.rst Outdated Show resolved Hide resolved

pletchm added 2 commits March 25, 2019 17:57

Merge branch 'feature/expand-dims-broadcast' of https://github.com/pl…

cac434e

…etchm/xarray into feature/expand-dims-broadcast

shoyer merged commit 16a2c03 into pydata:master Mar 26, 2019

davehemming mentioned this pull request Apr 13, 2019

expand_dims() modifies numpy.ndarray.flags to write only, upon manually reverting this flag back, attempting to set a single inner value using .loc will instead set all of the inner array values #2891

Closed

TomNicholas mentioned this pull request Feb 19, 2020

xarray.DataArray.expand_dims() can only expand dimension for a point coordinate #2710

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 #2757

Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 #2757

Uh oh!

pletchm commented Feb 8, 2019

Uh oh!

pep8speaks commented Feb 8, 2019 •

edited

Loading

Uh oh!

fujiisoup left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pletchm commented Feb 26, 2019

Uh oh!

Uh oh!

pletchm commented Mar 25, 2019

Uh oh!

shoyer left a comment

Uh oh!

Uh oh!

Uh oh!

shoyer commented Mar 26, 2019

Uh oh!

Uh oh!

Uh oh!

Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 #2757

Allow expand_dims() method to support inserting/broadcasting dimensions with size>1 #2757

Uh oh!

Conversation

pletchm commented Feb 8, 2019

Changes:

Included:

What's new:

Uh oh!

pep8speaks commented Feb 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2019-03-26 01:00:29 UTC

Uh oh!

fujiisoup left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pletchm commented Feb 26, 2019

Uh oh!

Uh oh!

pletchm commented Mar 25, 2019

Uh oh!

shoyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shoyer commented Mar 26, 2019

Uh oh!

Uh oh!

pep8speaks commented Feb 8, 2019 •

edited

Loading