Floating point semantics: unstable semantics inside/outside of a loop and among architectures

[RemiMattheyDoret]

What version of Go are you using (go version)?

$ go version go1.20.5 darwin/arm64

Does this issue reproduce with the latest release?

Yes (I have not tried the 1.21-rc though)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="arm64"

GOBIN=""

GOCACHE="/Users/remi/Library/Caches/go-build"

GOENV="/Users/remi/Library/Application Support/go/env"

GOEXE=""

GOEXPERIMENT=""

GOFLAGS=""

GOHOSTARCH="arm64"

GOHOSTOS="darwin"

GOINSECURE=""

GOMODCACHE="/Users/remi/go/pkg/mod"

GONOPROXY="github.com/edgelaboratories"

GONOSUMDB="github.com/edgelaboratories"

GOOS="darwin"

GOPATH="/Users/remi/go"

GOPRIVATE="github.com/edgelaboratories"

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/opt/homebrew/Cellar/go/1.20.5/libexec"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64"

GOVCS=""

GOVERSION="go1.20.5"

GCCGO="gccgo"

AR="ar"

CC="cc"

CXX="c++"

CGO_ENABLED="1"

GOMOD="/Users/remi/test/go.mod"

GOWORK=""

CGO_CFLAGS="-O2 -g"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-O2 -g"

CGO_FFLAGS="-O2 -g"

CGO_LDFLAGS="-O2 -g"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/75/728c40g56lzgyzhddw9scyp80000gn/T/go-build3544878519=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Consider the following code

func main() {
	var (
		negativeTwoThirds = -2.0 / 3.0
		negativeSix       = -6.0
		negativeThree     = -3.0
	)

	for _, v := range []float64{negativeTwoThirds} {
		four := negativeSix * v // 4 = (-6.0) * (-2/3)

		print(negativeThree + four)
		printAddition(negativeThree, four)
	}

	four := negativeSix * negativeTwoThirds // 4 = (-6.0) * (-2/3)
	print(negativeThree + four)
}

func print(c float64) {
	fmt.Printf("%f (%b)\n", c, math.Float64bits(c))
}

func printAddition(a, b float64) {
	c := a + b
	fmt.Printf("%f (%b)\n", c, math.Float64bits(c))
}

demo

What did you expect to see?

I expected

1.000000 (11111111110000000000000000000000000000000000000000000000000000)
1.000000 (11111111110000000000000000000000000000000000000000000000000000)
1.000000 (11111111110000000000000000000000000000000000000000000000000000)

This is indeed what I observe on the demo as well as on a linux (with 11th Gen Intel® Core™ i7-1165G7; x86) I tried this code on.

What did you see instead?

On two different macbook (with Apple M1; ARM64), I however observe the following output

1.000000 (11111111101111111111111111111111111111111111111111111111111110)  // print inside the loop
1.000000 (11111111110000000000000000000000000000000000000000000000000000)  // printAddition inside the loop
1.000000 (11111111110000000000000000000000000000000000000000000000000000)  // print outside of the loop

Problem

We observe a rounding error but not on all machines and only in specific circumstances (seemingly, only when the addition is directly performed within a for loop). I see two discrepancies in floating point semantics

• inside vs outside of the loop
• between different machine architecture

[rokkerruslan]

There is difference in results in different assembler instruction of float-point arithmetic. Simple analyse of assembler output of your code say that used different instructions.

One is FMUL and FADD, second is FMADD. Simplified example:

package main

import (
	"math"
)

func main() {
	println(math.Float64bits(Func1()))
	println(math.Float64bits(Func2()))
}

func Func1() float64
func Func2() float64

#include "textflag.h"

TEXT main·Func1(SB), NOSPLIT|NOFRAME, $0
	FMOVD	$(-0.6666666666666666), F0
	FMOVD	$(-6.0), F1
	FMULD	F1, F0, F0  // MULTIPLY

	FMOVD	$(-3.0), F1
	FADDD	F0, F1, F0 // NEXT ADD 

	FMOVD F0, out+0(FP)
	RET	(R30)

TEXT main·Func2(SB), NOSPLIT|NOFRAME, $0
	FMOVD	$(-0.6666666666666666), F0
	FMOVD	$(-3.0), F1
	FMOVD	$(-6.0), F2

	FMADDD  F0, F1, F2, F3 // MUL AND ADD

	FMOVD F3, out+0(FP)
	RET	(R30)

Output:

$ go env GOOS GOARCH
darwin
arm64
$ go run .          
4607182418800017408
4607182418800017406

---

As a result, it can be said that A+B*C not equals TMP := B*C; TMP + A. And:

inside vs outside of the loop

No its not depends of loop, if you will print result outside of printAddition all will be works fine :)

between different machine architecture

Yeap. To understand whether this is normal or not, you need to read https://en.wikipedia.org/wiki/IEEE_754 and analyze what requirements it puts on implementations (impl in ARM proc, not in Go).

[kostix]

@gavv suggested it's basically the same issue as, say dotnet/runtime#64591: FMADD is a so-called "fused" instruction which allows to use higher precision and/or not perform certain rounding after each constituent operation–hence the difference in behavor compared to combining individual instructions.

[bcmills]

This seems to be as documented in https://go.dev/ref/spec#Floating_point_operators:

An implementation may combine multiple floating-point operations into a single fused operation, possibly across statements, and produce a result that differs from the value obtained by executing and rounding the instructions individually. An explicit floating-point type conversion rounds to the precision of the target type, preventing fusion that would discard that rounding.

@RemiMattheyDoret, have you tried adding explicit conversions as suggested in that paragraph?

[jbardin]

@bcmills on darwin_arm64 with explicit conversions of the above, the compiler appears to treat the order of operations differently within the loop

negativeTwoThirds := float64(-2.0) / float64(3.0)
negativeSix := float64(-6.0)
negativeThree := float64(-3.0)

for _, negativeTwoThirds := range []float64{negativeTwoThirds} {
	c := negativeThree + (negativeSix * negativeTwoThirds)
	fmt.Printf("%0.20f\n", c)
}

c := negativeThree + (negativeSix * negativeTwoThirds)
fmt.Printf("%0.20f\n", c)

The individual expressions generating c in both cases are identical, but the output is

0.99999999999999977796
1.00000000000000000000

I agree it appears to be within spec, it's just the inconsistency based on context which could be surprising.

[RemiMattheyDoret]

@RemiMattheyDoret, have you tried adding explicit conversions as suggested in that paragraph?

Thank you @bcmills!
I confirm that replacing four := negativeSix * v by four := float64(negativeSix * v) resolved the problem. I do not fully understand whether that is a desired feature of a programming language (and whether similar complications exist for, say, C++) but at least it makes a bit more sense to me now.

The existence of a discrepancy among machines with different instruction sets (assuming the instruction set is what matters here) is still surprising to me and feels like a bug.

[bcmills]

The existence of a discrepancy among machines with different instruction sets (assuming the instruction set is what matters here) is still surprising to me and feels like a bug.

The instruction set determines which of the “fused operations” are available for the compiler to use.

[bcmills]

I agree it appears to be within spec, it's just the inconsistency based on context which could be surprising.

That is, unfortunately, the nature of compiler optimizations.

[ianlancetaylor]

This is an area where different languages make different optimization choices. Go has chosen to generate faster code on average. The results are not bit-for-bit identical on different processors, but as pointed out above Go has a documented language feature that you can use to get bit-for-bit identical results. This is based on the expectation that more people care about getting fast results on their processor than care about getting bit-for-bit identical results on multiple processors.

Java, for example, has chosen differently, and tries to get bit-for-bit identical results on all processors. This has led to papers like How Java’s Floating-Point Hurts Everyone Everywhere. Not that I want to fully endorse everything that that paper says, but it suggests that there is no ideal choice here.

I'm going to close this issue because I don't think there is anything to do here. Please comment if you disagree.