Enable images of arbitrary size

[antonfirsov]

Currently our images and internal buffers are limited to Int32.MaxValue size, ArgumentOutOfRange exception is thrown when a code attempts to allocate a larger buffer.

The most robust way to overcome this limitation would be enabling discontinuous buffers in our memory management, by adapting ReadOnlySequence<T> defining a writable memory sequence primitive. EDIT: ReadOnlySequence<T> is not the way to go, it's purpose is completely different.

This is a breaking change, so we really need to know how many users are interested in this use case, before commiting ourself. If you are one of them, please let us know about your use case in comments! (Platform, purpose of application etc.)

[JimBobSquarePants]

In the interim we could simply allocate the buffer?

if (bufferSizeInBytes >= int.MaxValue)
{
    return new BasicArrayBuffer<T>(new T[length]);
}

Personally though I'd like to go all in and use ReadOnlySequence<T>. We've had quite a few questions about large images.

[antonfirsov]

I also don't think it's worth to bother with non-scalable solutions. Workarounds like allocating new T[int.MaxValue] or new ReinterpretCastableLargeStruct[int.MaxValue] could easily lead to OOM.

[JimBobSquarePants]

I only meant to temporarily push our current upper limit to T[int.MaxValue-1] to give us some breathing room. I'd much rather go straight out with sequences though.

[rickbrew]

I'm not using Image# in Paint.NET, but I can provide some information on what I've seen and what I've thought about in this space.

I've been having a very slow, but also slowly increasing, drip of requests for correct operation when working with very large images. Usually in the 30K x 30K kind of range, not quite up to the current 64K x 64K maximum. Several things in PDN get in the way of this working, but the two big ones are: 1) MSFT's GDI+ codecs don't do it (e.g. loading/saving of PNG, JPEG, GIF), and 2) PDN's internal data structures for invalidation rects and rendering queues can't handle it (performance-wise). And of course 3) it requires a ton of memory (65535 x 65535 BGRA is 16GB per layer).

#1 is being solved in my upcoming update by migrating from GDI+ to WIC. Even though GDI+ is based on top of WIC since Win7, it does one big IWICBitmapSource::CopyPixels() call to the bitmap decoder instead of doing, e.g. 128 rows at a time which then works fine for max sized images, modulo a bug in the PNG codec I was tweeting about. (CopyPixels' cbBufferSize is a DWORD and thus limits CopyPixels to 4GB at a time ... woof.)

Some examples of what folks are actually wanting to use this for include: NVIDIA Ansel screenshots, where your video game has the ability to take super resolution screenshots. I have one I took in The Witcher 3 that has a size of 63360 x 35640 pixels.

Large image sizes are also sometimes used when working with maps and textures for indie games. Sometimes see weird image sizes when someone's trying to re-skin a plane in a flight simulator or something -- very wide or very tall images, for instance. Sometimes huge images are just a result of people not understanding that scanning documents at very high DPI results in images with a pixel size that the computer can't really handle. I've also seen people trying to work with hand-drawn maps for tabletop/board games or other kinds of D&D-ish games that get very large and PDN can't really handle them.

However, once you get up to this size of image you* really do need to switch to a tile-based storage and rendering system. If you allocate enormous images and do processing on them you end up with bad locality ((x,y) is very far away from (x,y+1) in memory, and probably in a different page), very high memory usage, slow rendering performance, and virtual address range claustrophobia (mostly on 32-bit). Even the OS seems to have bad performance when you ask VirtualAlloc for, say, 10GB. Having the ability to chop it up into (e.g.) 128x128 tiles means you can do sparse allocations, or have a tile describe itself as "I'm all blue 0xff0000ff" without having to store that repetitive data. This is very important for a layered image editor like PDN where you could have 10 layers but 9 of them are mostly empty. You can also allocate tiles on-demand instead of up-front and thus amortize the CPU cost of allocating.

The list goes on :) but the tl;dr is that when you get to this size of image you'll probably need to start thinking about rearchitecting your app. However, having support in the libraries and frameworks at least ensures things will work until you can spend the engineering resources to make it work fast. (another stop gap is buying a dual Xeon workstation or something)

* me

[rickbrew]

I also don't think it's worth to bother with non-scalable solutions. Workarounds like allocating new T[int.MaxValue] or new ReinterpretCastableLargeStruct[int.MaxValue] could easily lead to OOM.

When working with buffers of this size the app is going to be having to deal with OOMs anyway :) Even on 64-bit.

[antonfirsov]

@rickbrew wow, really useful and interesting insights, thanks for sharing!

Tiling is very complicated to manage for us, because we usually process images row-by-row. What I want to do is to store strips of the whole image in a sequence of contiguous buffer segments. This would be a manageable change architecture-wise, because if we ask for a pixel row (Span<TPixel>) at a y position, it doesn't really matter which buffer segment (strip) does the row span originate from from the POV of the processing code. If you have tiles, rows are also "broken".

This might be not the most optimal solution from locality point of view, but I guess it should be a solution that is good enough for us. An RGBA image strip of 65535 x 128 size is still "only" 32 MB in memory.

My hope is that if there is enough memory on the target machine, the runtime is smart enough allocate those 32 MB buffers wisely, avoiding OOM-s, but well ... this needs some research 😅. If the buffers are backed by Memory Mapped Files, I think we should be able to avoid OOM-s on most platforms.

PS:
Thanks for developing PDN, incredible software in it's category!

[antonfirsov]

@JimBobSquarePants

I only meant to temporarily push our current upper limit to T[int.MaxValue-1] to give us some breathing room.

The problem is that even a single new byte[int.MaxValue-1] allocation could lead to OOM, if you are unlucky. I hope that after #888 the breathing room is already there.

[JimBobSquarePants]

@antonfirsov How much effort do you think this requires? I'd like to set a reasonable milestone.

[antonfirsov]

For me it would take one well-rested & focused day for the experiments + design + POC. The rest is refactor only (could be done in a few evenings maybe). I wouldn't bother with it before 1.0.