cmd/link: binary on darwin takes up more space on disk

[kokes]

What version of Go are you using (go version)?

$ go version
go version devel +8ab37b1baf Mon Apr 20 18:32:58 2020 +0000 darwin/amd64

Does this issue reproduce with the latest release?

No (as in this is a regression in the current master, the latest stable is fine)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.2_1/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build271357823=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I noticed my binary went from 8MB (1.14.2) to 13MB (master), but only when running du -sh or inspecting the binary in Finder (under "how much it takes on disk").

When stating both binaries, they are very similar in size, so this issue only revolves around disk usage, not size.

I replicated the issue by creating a hello world app

package main
import "fmt"
func main() {

fmt.Println("ahoy")

}

And then I bisected it, starting at 1811533 (good) and ending at cb11c98 (bad).

package main
import (

"log"

"os/exec"

)
func main() {

build := exec.Command("/Users/okokes/git/go/src/make.bash")

build.Dir = "/Users/okokes/git/go/src"

err := build.Run()

if err != nil {

log.Fatal("toolchain build failed", err)

}
cmd := exec.Command("/Users/okokes/git/go/bin/go", "build", "src/hello.go")
err = cmd.Run()
if err != nil {
	log.Fatal("program build failed", err)
}

sz := exec.Command("du", "-sh", "hello")
out, err := sz.Output()
if err != nil {
	log.Fatal("du failed", err)
}
num := out[0]
if num != '2' {
	log.Fatal(string(out))
}

}

Bisect identified 8ab37b1 as the first offending commit. I verified it manually - the commit before that leads to a 2.1MB binary on disk, this commit leads to 4.1MB on disk.

I could not replicate this on a Ubuntu 18.04 box, so I presume it's a Darwin thing.

What did you expect to see?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
2.1M	hello_8ab37b1
2.1M	hello_go1.14

What did you see instead?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
4.1M	hello_8ab37b1     <-- note the binary size here
2.1M	hello_go1.14

[josharian]

cc @jeremyfaller

[cagedmantis]

/cc @cherrymui @rsc @mdempsky @ianlancetaylor

[randall77]

Note that copying the result to a new file fixes the problem.

[cherrymui]

Hmmm. https://tip.golang.org/src/cmd/link/internal/ld/outbuf_darwin.go

func (out *OutBuf) fallocate(size uint64) error {
	store := &syscall.Fstore_t{
		Flags:   syscall.F_ALLOCATEALL,
		Posmode: syscall.F_PEOFPOSMODE,
		Offset:  0,
		Length:  int64(size),
	}

	_, _, err := syscall.Syscall(syscall.SYS_FCNTL, uintptr(out.f.Fd()), syscall.F_PREALLOCATE, uintptr(unsafe.Pointer(store)))
	if err != 0 {
		return err
	}

	return nil
}

I wonder if F_PEOFPOSMODE is right here. It seems

F_PEOFPOSMODE Allocate from the physical end of file.

So if we preallocate more than once, the second and later calls will allocate the whole size on top of the existing size...

[cherrymui]

Or we should subtract the existing size.

[gopherbot]

Change https://golang.org/cl/234481 mentions this issue: cmd/link: fix size calculation for file space preallocation on darwin

[networkimprov]

Shouldn't this be milestoned for 1.15?