Parametric Color Types

[SimonDanisch]

Hi,
is there any reason, not to have the color spaces parametric?

That's the major reason holding me back to use colors for OpenGL.
I guess it can get a little complicated with the ranges, if you introduce uint8 for the color spaces.
But I mainly need Float32 colors, because OpenGL has very limited to non-existent support for Float64 values.
So I would propose to use at least T<:FloatingPoint instead of Float64.

I guess I could just define convert(Array{Float32,1}, Color), but is that really the way to go?
Advantage: the user doesn't have to postfix every number with f0.
Disadvantage: large arrays, for example 3D arrays with color values, would be quite slow to convert...Which is definitely bad, if it can be avoided that easily.
Side note:
OpenGL is totally fine with any Array{Immutable, 1/2/3} for a texture or buffer, so an array with Float32 colors could get uploaded directly to VRAM)

[glennsweeney]

I'm not sure what the original rationale for this decision was. Perhaps @StefanKarpinski or @dcjones would be the best to provide insight here?

[dcjones]

We probably should switch to parametric types. Being able to use color types in OpenGL without conversion would indeed be nice. Honestly, I think original rationale was just that this was the first Julia code I wrote and I didn't entirely understand parametric types.

[glennsweeney]

To be fair, this is the first (and only) Julia code that I worked on, and I still don't entirely understand parametric types.

[timholy]

I almost have this done. Here's an important implementation decision: I'd like to implement support for RGB, RGBA, and ARGB with integers as well as FloatingPoint. Sounds easy, right? Just RGB{T} where T can be any number.

The problem comes in with the following: white is defined as RGB(1.0, 1.0, 1.0) and by RGB(0xff, 0xff, 0xff). RGB(1,1,1) would almost be black.

So, does there need to be a separate RGBI type that uses integer scaling? (e.g., white is RGBI{T}(typemax(T), typemax(T), typemax(T))`)

[SimonDanisch]

Yes, that's why I thought, you might just want to have floating point types in the beginning.
But this can be easily solved over the parameter, right?

whitepoint{T<:FloatingPoint}(::RGB{T}) = RGB{T}(one(T))
whitepoint{T<:Integer}(::RGB{T}) = RGB{T}(typemax(T))

That's exactly what parametric types and multiple dispatch should be used for, right?

[SimonDanisch]

I actually was just thinking about fixed size arrays and colors.
I'm highly interested in doing math with colors, and overhead less transformations between representations (vec3->rgb and the like).
The case, that you have some specialized immutable/type representing some special form of a vector seems to very common. Its basically just the process of naming the dimensions of the vector.
So, why not offer the fixed size arrays in the form of tools, that let you turn any immutable/type into a fixed size array, with all mathematical operations already defined, and mechanisms to convert between the different lengths and representations.
So you have array[index] as the default, which is used by all the abstract mathematical operations, and via the DataType you have your specialized access methods like array.r, array.oilpressure, array.altitude, and what not.
This is especially helpful for machine learning algorithms and visualizing data , as you can build really nice pipelines.
You can have some exotic vector type, which gets its own treatment in terms of preparing the visualization, you can use all the math operators available for that and you can upload the data seamlessly to VRAM in the end. And by just transforming it to another DataType (without touching the memory) you could trigger the visualization to show some different aspect of the data. (For example normal image with color values <-> color values interpreted as 3D space vectors, showing some info about color clusters)

[timholy]

@SimonDanisch (post 1): I initially thought that dispatch could be used too, and indeed my current implementation takes that route. But the example I gave shows the danger of this approach. I haven't seen a whitepoint function defined anywhere (in any package), but SomeColorValue(1,1,1) appears in multiple places. My concern is that relying on dispatch risks being an abuse of the type system.

As an example, sqrt(-1+0im) is a perfectly good use of the type system: you're saying, I'm expecting a complex-valued return value, so don't complain if I pass you a negative number. That's because -1+0im means exactly the same thing, mathematically, as -1, you're just representing it using a different type. In contrast, SomeColorValue(1,1,1) has a completely different meaning from SomeColorValue(1.0,1.0,1.0)---one means almost-black and the other means white. I worry that relying on types to convey such a subtle distinction will be bug-prone.

Here's a related example: suppose one performs an image transformation, like a rotation. In general, you'll need to do pixel interpolation. Using Julia's rules for arithmetic, the linear interpolation at a location halfway between two pixels with values RGB(0xd3, 0xa5, 0x79) and RGB(0xcd, 0x98, 0x84) is RGB(208.0, 158.5, 126.5). Because you've had to move to a floating-point type to represent the answer, suddenly you've also completely changed the meaning of this pixel value (it's super-saturated white rather than an orangeish-brownish color).

These are reasons why I suspect the scaling/interpretation of the meaning has to be separated from the raw data type used to represent the value.

[timholy]

Actually, it occurs to me that fundamentally this is an attribute that should be attached to the number. So perhaps the best approach would be to define

immutable RangedNumber{T,Max} <: Number
    x::T
end

So a value recorded by a 12-bit camera could be represented as x = RangedNumber{Uint16,int(0x0fff)}(0x0a01).

Not exactly sure how arithmetic on such types would work out, but it's interesting to consider.

[StefanKarpinski]

The range thing is similar to what I was going to suggest, but turned a bit sideways. A way to look at it is that when you use Uint8 like this, you're not really using it the normal way – i.e. to represent the integer values 0 through 255. Rather, you're using it as a kind of fixed point type that can represent the values 0/255, 1/255, ..., 255/255. Perhaps it would be better to always have the values be between 0 and 1 but define different standard ways of representing those values. Thus, instead of the type parameter being Uint8, it would be an 8-bit fixed-point float.

[SimonDanisch]

But that can't be a real solution, as it will break compatibility with a lot of other packages (OpenCV, OpenGL and the like).
For the interpolation problem, I would say you converted an Int color incorrectly to a Float color and that's why you end up with strange results.
Also, changing the type for filtering/interpolation always to Float64 seems uncommon to me.
Mostly, you don't want to change the type of your image and if you actually want to change the type of your image, you want control over it.
How I'm used to do these things is, that either I don't care, that the transformation is irreversible, or I transform it to floating point and also convert it to the range of 0-1, before applying transformations.
I don't see a reason, why we should do things differently than OpenCV and Matlab (It has been a while since I used those two, but I think they never just started converting your image to different types)

[SimonDanisch]

I think I just fully grasped why you would like to have RGBI or RangedNumber and don't feel comfortable with my solution.
You actually do want to have something like RGBI{FLoat64}, correct?
But I still stay with my opinion. From what I have perceived so far is, that the convention Float -> 0-1 and
Intger -> typemin -> typemax is pretty widespread and well accepted.
So if you end up with 125.0 its like you end up with UTF-8 code points in ASCII... You did some wrong conversions along the way.

[timholy]

@StefanKarpinski, that's a really interesting suggestion---thanks. I'm going to have to spend some time thinking about it, but indeed this might be the magic answer.

@SimonDanisch, correct. In scientific imaging, it's meaningful to say that the image intensity is 743: assuming you know the gain of your camera, this implies a certain number of photons that you collected, which in turn implies a certain signal-to-noise ratio, which in turn influences your confidence in your results. Saying you have to convert to a 0-1 scale to do arithmetic on images emphasizes "display" above "data", and I'm uncomfortable with that. Careful bookkeeping elsewhere can solve this problem, though.

[SimonDanisch]

Okay, so what you're saying is, you don't want to loose your scale, but need floatingpoints for arithmetic.
That would explain my different emphasize, as I'm surely more display driven.
But what you're talking about, are intensity values and not RGB values, right?
Do you actually plan to put intensity values into RGB values, without any transformation?

As for intensity values, it makes sense to introduce a new type like Intensity{T <: Real}, as they follow completely different semantics.
And normally, if you put intensity values into RGB values, you do that with a transition function, while loosing your "data semantics" in order to display them.

[timholy]

I agree that people who want to work with RGBs are unlikely to be the same people who need photons. Indeed I'm planning an SIUnits-like module for converting between Counts and Photons. Perhaps with enough number types, we can get everything we want!

I think I'm just going to start writing a FixedPointNumbers package and see how it works out.

[timholy]

Looks like Jeff already has a somewhat related package: https://github.com/JeffBezanson/FixedPoint.jl.

[SimonDanisch]

Ah you found it:) i just started searching for it.
Just out of curiousity, would you actually use RGB(Photon, Photon, Photon)? So you're actually collecting photons of different wavelenghts?
Obviously, I don't have any experience with the kind of work you're doing. Closest was LIDAR imaging I suppose.
And using SIUnits for disambiguating this seems like a great idea :) lets see, if we can please both sides. I just want to make sure, that if you're handling "normal" rgb uint8/float32 color values, that you don't have to create some exotic type, with three type parameters ;)

[timholy]

I'm not immediately interested in RGB(Photon, Photon, Photon). For me the key point is that I don't want to be forced to rescale my image just to do arithmetic on it.

And yes, it's a good idea to avoid exotic types that users have to create. I'm currently thinking RGB{Ufixed8} will be supported, but RGB{Uint8} will not. That way 1 always means "saturated" as far as RGB is concerned. imread("myimage.png") will, when appropriate, return an Image{RGB{Ufixed8}}.

[SimonDanisch]

Wait... With that you would achieve exactly the opposite, from what I need for interoperability.
RGB{Uint8} with 0->255 is one of the most common image formats after all. Or do Uint8 and Ufixed8 encode the same values on a bit level? (Even than, it would be confusing for people coming from other languages)
If not, I couldn't use imread without transforming the output.
I think it's rather crucial to support Integer color formats, with typemax(T) as white... Just not supporting this and always going with a 0-1 range sounds to me like a great way to pick fights with a lot of other libraries.
Wouldn't an Intensity type and/or SIUnits solve your use cases?

[timholy]

The idea is Ufixed8(0xff) == 1.0 evaluates to true, but the underlying bit-level representation is 0xff. You can pack them into arrays and OpenGL will think they are an array of Uint8s. Does that solve your concerns?

Example start of an implementation:

immutable Ufixed8 <: AbstractUFixedPoint
    value::Uint8

    Ufixed8(v::Uint8) = new(v)
end
Ufixed8(x::Real) = Ufixed8(iround(Uint8, 255*x))

asint(x::AbstractFixedPoint) = x.value

convert{T<:FloatingPoint}(::Type{T}, x::Ufixed8) = asint(x)/convert(T, 0xff)

[SimonDanisch]

I guess fixedpoint types are represented as the underlying integer in
memory... Well, i think i need to let this option sink in a little. With
proper documentation and memory compability, it might be an improvement
compared to the current mess in other libraries.
Am 25.07.2014 16:21 schrieb "Tim Holy" [email protected]:

I'm not immediately interested in RGB(Photon, Photon, Photon). For me the
key point is that I don't want to be forced to rescale my image just to do
arithmetic on it.

And yes, it's a good idea to avoid exotic types that users have to create.
I'm currently thinking RGB{Ufixed8} will be supported, but RGB{Uint8}
will not. That way 1 always means "saturated" as far as RGB is concerned.
imread("myimage.png") will, when appropriate, return an
Image{RGB{Ufixed8}}.

—
Reply to this email directly or view it on GitHub
#42 (comment).

[StefanKarpinski]

The core tension is that the two "sides" have quite different expectations from color values:

• colorimetry code wants color components to have the semantics of reals between 0 and 1;
• hardware and libraries like OpenGL want colors to be represented using unsigned 8-bit values.

Since these new types have the semantics of real values between 0 and 1 and an unsigned 8-bit representation, they give both sides what they want. It's a little early to say anything conclusive, but I kind of suspect that this will end up having a lot less dissonance at either end. You would only need new types if you have a new way of representing colors, which is not especially common.

[SimonDanisch]

I agree, it seems to be a brilliant solution!
After handling color values with the range of 0-255 for quite some time (even in photoshop they do this), I completely forgot, how this representation was born out of memory greed and no better alternative.
This is a big "I agree" from my side :)

@timholy
Just out of curiosity, will you start handling your case with RGB{Photons}(34,394,392)?
For a second, I thought it would be cool to represent any color like this.
You would end up with Vector3{ Red, Green, Blue}(0.1,0.2,0.3).
It would be quite nice as it goes in the direction of giving names to the vector dimensions and you don't have to create a new type for RGBA, ARGB, RG, HSV, etc...
Also, Single channel images wouldn't loose their semantics.
But first of all, colors aren't really SIUnits, and it would be a little cumbersome to define the number type on top of this.
We could define our own color semantic package, like Red{Float32}, or Intensity{Int}, which would work nicely with OpenGL, as it's quite common to have different bit sizes for different color channels.
Ah well, just another silly idea, with too many odd cases I guess...

[timholy]

It's not crazy, but I was going to wait to tackle such things when a serious need arose.

[timholy]

Close by bb94a80