Refactor shape masking #6129

hsteptoe · 2024-08-22T14:35:28Z

🚀 Pull Request

Description

Fixes #6126 by exposing the shape geometry to the user. Adds to docs and docstrings to make it clearer how to use this.

Consult Iris pull request check list

Add any of the below labels to trigger actions on this PR:

benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts

for more information, see https://pre-commit.ci

pp-mo · 2024-08-28T09:16:41Z

From @SciTools/peloton : discussed what we are seeing here and in #6126.
We think this is looking useful, but the two of you seem to have it covered @hsteptoe @acchamber ?
Please shout if you need help.

hsteptoe · 2024-08-28T09:55:53Z

Discussion with @acchamber notes:

Don't move imports out of functions as this creates circular import errors
Check target_crs assignment for occasion when cube doesn't have a CRS
Re-write conversion of 0-360 -> -180-180 coords in case of degree based CRSs

hsteptoe · 2024-11-22T11:31:36Z

What are @SciTools/peloton (@pp-mo) thoughts about adding rasterio as a Iris dependency?

This would facilitate better handling of shapes to rasters (which is essentially the problem we're solving) with the added bonus that we could also mask to other shape types, like lines, to extract a trajectory from a Cube, for example.

acchamber · 2024-11-29T14:06:06Z

I've had a few other discussions with other users now where they can't use this function with OSGB shapefiles which this PR would fix. Where are we on being able to include it in the next release?

@hsteptoe @pp-mo

hsteptoe · 2024-12-02T09:56:12Z

I've had a few other discussions with other users now where they can't use this function with OSGB shapefiles which this PR would fix. Where are we on being able to include it in the next release?

@hsteptoe @pp-mo

I was planning on going to the AVD surgery this week to discuss some changes and/or make the case for rasterio

trexfeathers · 2024-12-02T15:28:12Z

Thanks for your patience everyone. It's sometimes a struggle to balance everything and we just haven't had an opportunity to discuss this. Coming to the Surgery (UK Met Office) is an ideal next step.

stephenworsley · 2024-12-04T12:57:19Z

@pp-mo, @trexfeathers and @stephenworsley have discussed this at the Surgery and agree in principle that we could consider adding rasterio as an optional dependency.

for more information, see https://pre-commit.ci

trexfeathers · 2025-02-05T10:42:50Z

From @SciTools/peloton: would @hsteptoe and @acchamber appreciate any more input from core devs at this point?

hsteptoe · 2025-02-05T12:07:07Z

From @SciTools/peloton: would @hsteptoe and @acchamber appreciate any more input from core devs at this point?

I just need to find more time, which is in short supply as we get near the end of FY...

for more information, see https://pre-commit.ci

hsteptoe · 2025-02-14T16:38:23Z

This is now ready for some initial (alpha?) testing 🎉. @acchamber are you available to try out the new features? (Updates to docs etc. will follow if this is successful) Tests are incomplete are should not be review yet.

This is a fairly substantial re-write. New/improved features include:

Automatic CRS transforms. Will handle situation where the cube and geometry have different CRSs by transforming the geometry.
Handles all shape types, including Points, Lines and MultiPolygons.

There are some breaking changes, principally moving away from having a minimum_weight keyword, replacing it with an all_touched keyword. The rational here is for consistency across shape types. Minimum weight doesn't make sense for Points and Lines. Instead, Line pixilation is based on Bresenham's line algorithm. Polygon pixilation is based on either: (a) pixel centres that intersect the polygon, or (b) all pixels touched by the polygon irrespective of the area of intersection.

My informal tests so far show working behaviour for iris test data from rotated_pole.nc, toa_brightness_stereographic.nc, E1_north_america.nc, air_temp.pp. Know limitation (not-working) for semi-structured model grids, such as orca2_votemper.nc. Some test shapefiles 👉test_shapefiles.zip

CLAassistant · 2025-10-06T07:55:06Z

All committers have signed the CLA.

hsteptoe · 2025-10-07T13:28:28Z

@stephenworsley @trexfeathers Can you advise on apparently nox related errors that I'm seeing in the current CI failures, giving an error message: CondaHTTPError: HTTP 401 CONNECTION FAILED for url...

And, do you know why my local pre-commit version of numpydoc-validation might pass, but the github action one fails?

trexfeathers · 2025-10-14T14:54:35Z

@stephenworsley @trexfeathers Can you advise on apparently nox related errors that I'm seeing in the current CI failures, giving an error message: CondaHTTPError: HTTP 401 CONNECTION FAILED for url...

And, do you know why my local pre-commit version of numpydoc-validation might pass, but the github action one fails?

@hsteptoe really sorry for missing this. There is a MO-specific step you will need to do for the lockfiles, will message you offline.

I'm less sure about pre-commit. There is some nuance about only testing files which have been edited; perhaps the merge from main means that your PR is changing more than your local branch was.

hsteptoe · 2025-10-15T13:09:40Z

@stephenworsley @trexfeathers Can you advise on apparently nox related errors that I'm seeing in the current CI failures, giving an error message: CondaHTTPError: HTTP 401 CONNECTION FAILED for url...
And, do you know why my local pre-commit version of numpydoc-validation might pass, but the github action one fails?

@hsteptoe really sorry for missing this. There is a MO-specific step you will need to do for the lockfiles, will message you offline.

I'm less sure about pre-commit. There is some nuance about only testing files which have been edited; perhaps the merge from main means that your PR is changing more than your local branch was.

pre-commit related errors may have been something wrong going on with my pre-commit cache. Did a pre-commit clean and re-ran and that seemed to resolve the inconsistency issues.

trexfeathers

Still left to review:

_shapefiles.py
tests
lock files

docs/src/whatsnew/latest.rst

trexfeathers · 2025-10-15T11:06:02Z

pyproject.toml

+[project.optional-dependencies]
+RASTERIO = ["rasterio"]
+


We didn't. However we are attempting to standardise across our repositories and we don't have an agreed way to handle optional dependencies, so please remove this entry.

docs/src/whatsnew/latest.rst

trexfeathers · 2025-10-15T11:22:29Z

lib/iris/util.py

+    cube: iris.cube.Cube,
+    shape: shapely.Geometry,
+    shape_crs: cartopy.crs | pyproj.CRS,
+    in_place: bool = False,
+    minimum_weight: float = 0.0,
+    **kwargs,


You've changed the order of parameters. Anyone calling this without using keywords e.g. mask_cube_from_shapefile(my_cube, my_shape, 0.5, True) will get a failure.

Reverted back to depreciated function call, but have had to add a few extra lines to pull in a shape_crs as per https://github.com/SciTools/iris/blob/fa6d61dd5f358c9e6b585a132cc213acebf95b70/lib/iris/_shapefiles.py#L142C18-L144C6

trexfeathers · 2025-10-15T11:30:45Z

lib/iris/util.py

+        A number between 0-1 describing what % of a cube cell area must the shape overlap to be masked.
+        Only applied to polygon shapes.  If the shape is a line or point then this is ignored.
+
+    Other Parameters


How did you decide which ones were kwargs and which ones were named parameters?

I decided that kwargs would be parameters that are specifically used by rasterio functions only, but I'm happy to expose them if that's clearer?

lib/iris/util.py

trexfeathers · 2025-10-15T13:11:21Z

lib/iris/util.py

+
+    Extract a rectangular region covering the UK from a stereographic projection cube:
+
+    >>> cube = iris.load_cube(iris.sample_data_path("toa_brightness_stereographic.nc"))


It would be great to print some one line Cube summaries to demonstrate the effect that masking is having.

Example:

Suggested change

>>> cube = iris.load_cube(iris.sample_data_path("toa_brightness_stereographic.nc"))

>>> cube = iris.load_cube(iris.sample_data_path("toa_brightness_stereographic.nc"))

>>> print(cube.summary(shorten=True))

toa_brightness_temperature / (K) (projection_y_coordinate: 160; projection_x_coordinate: 256)

lib/iris/util.py

trexfeathers

Still left to review:

tests
lock files
confirm stable behaviour given the heavy refactor

trexfeathers · 2025-10-15T13:40:12Z

lib/iris/_shapefiles.py

+# to "OFF" in the environment. Having PROJ_NETWORK = "ON"
+# can lead to some coordinate transformations resulting in Inf values.


What a pointless feature! I'd be interested to know why.

trexfeathers · 2025-10-15T13:41:42Z

lib/iris/_shapefiles.py

+if "iris.cube" in sys.modules:
+    import iris.cube
+if "iris.analysis.cartography" in sys.modules:
+    import iris.analysis.cartography


This is unprecedented. Something larger is wrong if these modules are not available.

I can't remember anymore why I added this (!) so it's now removed.

lib/iris/_shapefiles.py

trexfeathers · 2025-10-15T16:46:44Z

lib/iris/_shapefiles.py

+    w = len(x_coord.points)
+    h = len(y_coord.points)
+    # Mask by weight if minimum_weight > 0.0
+    if minimum_weight > 0:


We discussed preserving existing behaviour with minimum_weight - #6129 (comment) - to avoid breaking changes. I'm having a hard time confirming this:

The code for this behaviour has been heavily refactored

The existing tests have been rewritten and can no longer be run in their original form due to API changes

Can you help? Do you have anything 'ready-made' that I can use to confirm stable behaviour? Otherwise I'll write something myself. Thanks

The integration tests lib/iris/tests/integration/test_mask_cube_from_shape.py include tests from the previous iteration of this function (which was itself benchmarking against ASCEND). Specifically, the tests test_4d_global_proj_brazil, test_rotated_pole_proj_germany_weighted_area and test_transverse_mercator_proj_uk are testing against identical conditions. Would this be sufficient to confirm stable behaviour?

I have added warning text from _shapefiles into utils to emphasise behaviour change that may be seen as breaking behaviour (although this wasn't stable behaviour in the first place):

Because shape vectors are inherently Cartesian in nature, they contain no inherent understanding of the spherical geometry underpinning geographic coordinate systems. For this reason, **shapefiles or shape vectors that cross the antimeridian or poles are not supported by this function to avoid unexpected masking behaviour**.

lib/iris/_shapefiles.py

trexfeathers · 2025-10-15T17:11:35Z

lib/iris/_shapefiles.py

+        # Define raster transform based on cube
+        # This maps the geometry domain onto the cube domain
+        tr = _make_raster_cube_transform(cube)
+        # Generate mask from geometry
+        mask_template = rfeatures.geometry_mask(
+            geometries=shapely.get_parts(geometry),
+            out_shape=(h, w),
+            transform=tr,
+            all_touched=all_touched,
+        )


Would you be up for giving me a 5 minute lesson about this?

lib/iris/util.py

Co-authored-by: Martin Yeo <[email protected]>

…bugfix-shape-masking

hsteptoe and others added 3 commits August 22, 2024 15:05

Updates to allow varying shape crs

ebee977

Ruff fixes

e32f911

[pre-commit.ci] auto fixes from pre-commit.com hooks

76eae38

for more information, see https://pre-commit.ci

Merge branch 'SciTools:main' into bugfix-shape-masking

bcdca30

hsteptoe and others added 11 commits December 6, 2024 13:37

Merge branch 'SciTools:main' into bugfix-shape-masking

9dab7c6

Adding rasterio as optional dependency

ebaa1ee

Adding rasterio functionality

103e3d7

First working version with rasterio

3c65dab

Add further shape geometry checks

25de18d

Update shapefile tests

10e4413

Reorganise test fixtures

cd5cf99

Reorganise test fixtures

edc25cc

Fix fixture fetching

789c98d

Merge branch 'SciTools:main' into bugfix-shape-masking

27a9025

[pre-commit.ci] auto fixes from pre-commit.com hooks

dc090a5

for more information, see https://pre-commit.ci

hsteptoe and others added 4 commits February 7, 2025 08:26

Merge branch 'SciTools:main' into bugfix-shape-masking

5b1054b

Recording Affine transform trials

8948ce6

Working masking

a77c628

[pre-commit.ci] auto fixes from pre-commit.com hooks

1793e1f

for more information, see https://pre-commit.ci

bjlittle added this to the v3.13 milestone Feb 17, 2025

scitools-ci bot removed this from 🚴 Peloton Sep 24, 2025

scitools-ci bot added this to 🚴 Peloton Sep 25, 2025

hsteptoe added 6 commits October 7, 2025 09:01

Merge branch 'main' into bugfix-shape-masking

4bba208

Resolve lock files with added rasterio and affine

84fd9c2

Sync branches

bb8ba7f

Resolve lock files with added rasterio and affine

6f42579

Test consistency updates

d17a4d5

Merge branch 'main' into bugfix-shape-masking

c4a7b63

hsteptoe added 4 commits October 15, 2025 10:06

Update lock file urls

1536638

Merge branch 'SciTools:main' into bugfix-shape-masking

3d2d60b

Trying to fix numpy-doc error

3a10a0e

Update lockfiles

1b2b0b3

trexfeathers moved this to 👀 In Review in 🦌 Iris 3.14 Oct 15, 2025

trexfeathers changed the title ~~Bugfix shape masking~~ Refactor shape masking Oct 15, 2025

hsteptoe added 2 commits October 15, 2025 14:01

Merge branch 'main' into bugfix-shape-masking

8333991

Unit test tweaks

3ee3790

trexfeathers requested changes Oct 15, 2025

View reviewed changes

lib/iris/util.py Outdated Show resolved Hide resolved

lib/iris/util.py Outdated Show resolved Hide resolved

trexfeathers requested changes Oct 15, 2025

View reviewed changes

trexfeathers reviewed Oct 16, 2025

View reviewed changes

lib/iris/util.py Show resolved Hide resolved

hsteptoe and others added 6 commits October 17, 2025 09:15

Update docs/src/whatsnew/latest.rst

a0f7e16

Co-authored-by: Martin Yeo <[email protected]>

Update lib/iris/util.py

1250496

Co-authored-by: Martin Yeo <[email protected]>

Update lib/iris/util.py

1997c8d

Co-authored-by: Martin Yeo <[email protected]>

More lockfile updating

192c013

Merge branch 'bugfix-shape-masking' of github.com:hsteptoe/iris into …

fd7bcdd

…bugfix-shape-masking

First batch of code-review changes

df75630


		Extract a rectangular region covering the UK from a stereographic projection cube:

		>>> cube = iris.load_cube(iris.sample_data_path("toa_brightness_stereographic.nc"))

		# to "OFF" in the environment. Having PROJ_NETWORK = "ON"
		# can lead to some coordinate transformations resulting in Inf values.

		[project.optional-dependencies]
		RASTERIO = ["rasterio"]

Refactor shape masking #6129

Are you sure you want to change the base?

Refactor shape masking #6129

Uh oh!

Conversation

hsteptoe commented Aug 22, 2024

🚀 Pull Request

Description

Uh oh!

pp-mo commented Aug 28, 2024

Uh oh!

hsteptoe commented Aug 28, 2024

Uh oh!

hsteptoe commented Nov 22, 2024

Uh oh!

acchamber commented Nov 29, 2024

Uh oh!

hsteptoe commented Dec 2, 2024

Uh oh!

trexfeathers commented Dec 2, 2024

Uh oh!

stephenworsley commented Dec 4, 2024

Uh oh!

trexfeathers commented Feb 5, 2025

Uh oh!

hsteptoe commented Feb 5, 2025

Uh oh!

hsteptoe commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsteptoe commented Oct 7, 2025

Uh oh!

trexfeathers commented Oct 14, 2025

Uh oh!

hsteptoe commented Oct 15, 2025

Uh oh!

trexfeathers left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

trexfeathers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

hsteptoe commented Feb 14, 2025 •

edited

Loading

CLAassistant commented Oct 6, 2025 •

edited

Loading

trexfeathers left a comment •

edited

Loading