image, image/draw: add interfaces for using RGBA64 directly

[zephyrtronium]

I have an application that involves calculating image data at extremely high resolutions, sometimes on the scale of gigapixels, then resampling the results to smaller sizes. Using package x/image/draw to perform the resampling gives excellent results, but rather slowly. One significant reason for this is that every call to At allocates, because the result type is an interface. A quick benchmark shows that this is responsible for 80% of the allocations in my application.

Package x/image/draw already contains specialized fast paths for many of the raw image types in package image. However, because my images are calculated by histogramming sometimes trillions of iterations of a chaotic process, I cannot use those types to accumulate the image. I also cannot copy to such a type prior to rescaling, because the images I'm working with already occupy most of the memory I have available.

In order to resolve this, I propose to create a standardized interface in package image that allows image processing routines to define fast paths that can avoid unnecessary allocations. The interface would be as follows:

package image

// RGBA64Image is an image with pixels that can be transformed directly to
// color.RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit values to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

Additionally, I propose that a corresponding interface be added to package image/draw (with an alias in x/image/draw):

package draw

// RGBA64Image is an Image with a SetRGBA64 method to set the color of a
// single pixel. It is like Image, but using it may mitigate allocations from
// converting concrete color types to the color.Color interface.
type RGBA64Image interface {
	SetRGBA64(x, y int, c color.RGBA64)
	Image
}

I propose that the image types Alpha, Alpha16, CMYK, Gray, Gray16, NRGBA, NRGBA64, Paletted, and RGBA be extended to implement both of these interfaces (noting that RGBA64 already does), and that NYCbCrA, Uniform, and YCbCr be extended to implement the former.

With these interfaces, image processing algorithms in image/draw, x/image/draw, and elsewhere can implement generic fast paths, rather than only having fallbacks for image.Image that force at least an allocation per pixel. An experiment using a modified version of x/image/draw to implement such a fast path for Kernel.Scale shows that resampling an image from 1280x1280 to 128x128 goes from 3296775 to 32775 allocations per operation, a 100x improvement, with approximately a 3x speed improvement. In particular, the primary culprit method in my application disappears completely from the memory profile.

---

I originally proposed to add an interface with RGBA64At to x/image/draw specifically, rather than to the standard library, because the interface allocations are most impactful when processing extremely large images, and resampling seems like the most likely operation for them. However, I've since talked with others who have had serious performance issues using image/draw for color transformations on many images, and @nigeltao suggested in a comment that these interfaces be added to the standard library.

[ianlancetaylor]

CC @nigeltao

[nigeltao]

Note that the RGBAAt(x, y int) (r, g, b, a uint32) proposal conflicts with the existing func (p *RGBA) RGBAAt(x, y int) color.RGBA in the standard library's package image.

Perhaps add some interfaces to the standard library (and make its concrete image types implement them):

---

package image

type ImageV2 interface {
    Image
    // AtDotRGBA(x, y) is equivalent to At(x, y).RGBA() but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    AtDotRGBA(x, y int) (r, g, b, a uint32)
}

// or perhaps

type ImageV2 interface {
    Image
    // RGBA64At(x, y) is equivalent to At(x, y).RGBA(), after packing RGBA's
    // four uint32 return values into one color.RGBA64, but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    RGBA64At(x, y int) color.RGBA64
}

---

package draw

type ImageV2 interface {
    Image
    // SetRGBA64(x, y, c) is equivalent to Set(x, y, c) but it can be more
    // efficient, as it does not allocate an intermediate color.Color.
    SetRGBA64(x, y int, c color.RGBA64)
}

---

golang.org/x/image/draw obviously gets:

type ImageV2 = draw.ImageV2

Yeah, the names are horrible and inconsistent, but we're constrained by Go 1 backwards compatibility...

[nigeltao]

CC @dmitshur

[zephyrtronium]

@nigeltao
I had forgotten about image.RGBA.RGBAAt. Considering this, the RGBA64At(x, y int) color.RGBA64 interface does seem more appropriate.

Reading through it again, package image also has somewhat of a convention that would suggest a better name. The concrete image types are RGBA, Gray16, &c. whereas the interface types are Image and PalettedImage. So, it would be natural to add RGBA64Image.

Unless there are objections, tomorrow I will retitle and change this proposal to adding

// RGBA64Image is an image with pixels that can be transformed directly to
// RGBA64.
type RGBA64Image interface {
	// RGBA64At returns the RGBA64 color of the pixel at (x, y). It is
	// equivalent to calling At(x, y).RGBA() and converting the resulting
	// 32-bit channels to color.RGBA64, but it may avoid allocations from
	// converting concrete color types to the color.Color interface type.
	RGBA64At(x, y int) color.RGBA64
	Image
}

to package image, updating the concrete image types to implement it, adding a corresponding RGBA64Image interface to image/draw and x/image/draw, updating the concrete image types in package image to implement these interfaces, and implementing fast paths in image/draw and x/image/draw using these types.

[nigeltao]

Sounds good to me.

With the new fast paths, I'd only bother with:

• "image.RGBA64Image on draw.RGBA64Image"

and not e.g.

• "image.Image on draw.RGBA64Image" or
• "image.RGBA64Image on draw.Image"

[naisuuuu]

I was about to submit a separate proposal to add additional fast paths to x/image/draw for more resampling destination types. This proposal would solve that and many more similar problems in a much more elegant way. Really hoping to see it come to fruition!

[nigeltao]

@ianlancetaylor as above (March 12), this proposal (affecting the std image package) Sounds Good To Me.

What's the formal process from here? Do I upgrade this issue from Proposals/Incoming to Proposals/Active?

[gopherbot]

Change https://golang.org/cl/311129 mentions this issue: image: add RGBA64Image interface

[ianlancetaylor]

@nigeltao No, that move will be done when the proposal review committee gets to this issue.

I'll add it to the list to consider in the next meeting.

[rsc]

Seems OK. Adding to minutes.