-
Notifications
You must be signed in to change notification settings - Fork 33
Rounding errors in rem
of Normed
#150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Perhaps the following is more accurate than the current implementation, and less accurate than PR #131. function Base.rem(x::Float32, ::Type{T}) where {f, T <: Normed{UInt32,f}}
f <= 24 && return reinterpret(T, unsafe_trunc(UInt32, round(rawone(T)*x)))
r = unsafe_trunc(UInt32, round(Int32, x * Float32(0x1p24)))
reinterpret(T, r << UInt8(f - 24) - r >> 0x18)
end |
BTW, I reported this issue because I noticed that the documents of Colors.jl include the example with I think it would be easier to fix the bug than add a note to the document. |
Definitely worth fixing. Though I can't imagine why the docs use |
minor adjustment: function rem(x::Float32, ::Type{T}) where {f, T <: Normed{UInt32,f}}
f <= 24 && return reinterpret(T, _unsafe_trunc(UInt32, round(rawone(T) * x)))
r = _unsafe_trunc(UInt32, round(x * @f32(0x1p24)))
reinterpret(T, r << UInt8(f - 24) - unsigned(signed(r) >> 0x18))
end
function rem(x::Float64, ::Type{T}) where {f, T <: Normed{UInt64,f}}
f <= 53 && return reinterpret(T, _unsafe_trunc(UInt64, round(rawone(T) * x)))
r = _unsafe_trunc(UInt64, round(x * 0x1p53))
reinterpret(T, r << UInt8(f - 53) - unsigned(signed(r) >> 0x35))
end However, similar problem remains due to the overflow instead of rounding error. julia> 64.0f0 % N0f32 # OK
0.9999999853N0f32
julia> 127.99999f0 % N0f32 # OK
0.9999923413N0f32
julia> 128.0f0 % N0f32 # internal overflow
2.98e-8N0f32
julia> 128.1f0 % N0f32 # (somewhat) low accuracy
0.1000061333N0f32 Since julia> @btime x .% N8f24 setup=(x=rand(Float32,64,64));
3.962 μs (1 allocation: 16.13 KiB)
julia> @btime x .% N0f32 setup=(x=rand(Float32,64,64));
4.050 μs (1 allocation: 16.13 KiB)
julia> @btime x .% N11f53 setup=(x=rand(Float64,64,64));
6.080 μs (2 allocations: 32.08 KiB)
julia> @btime x .% N0f64 setup=(x=rand(Float64,64,64));
6.860 μs (2 allocations: 32.08 KiB) |
As I mentioned in #102, PR #131 did not change the
rem
, so this is a reminder.Originally posted by @kimikage in #131 (comment)
The text was updated successfully, but these errors were encountered: