runtime: SetFinalizer causes undesirable memory overhead

[zhangqiang-01]

What version of Go are you using (go version)?

$ go version
go version go1.15.1 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/gosrc/pkg/mod"
GONOPROXY="code.byted.org"
GONOSUMDB="code.byted.org"
GOOS="linux"
GOPATH="/root/gosrc/"
GOPRIVATE="code.byted.org"
GOPROXY="https://go-mod-proxy.byted.org,https://goproxy.cn,direct"
GOROOT="/root/go"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/root/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build392013595=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
	"runtime"
	"sync"
	"time"
)

type A struct {
	buf []byte
}

func finalize(a interface{}) {
}

func test(wg *sync.WaitGroup) {
	for i := 0; i < 100000; i++ {
		for j := 0; j < 100000; j++ {

			a := &A{
				buf: make([]byte, 1024*1024),
			}
			runtime.SetFinalizer(a, finalize)

		}
		time.Sleep(time.Microsecond)
	}

	wg.Done()
}

func main() {
	n := 100
	wg := &sync.WaitGroup{}
	wg.Add(n)
	
	for i := 0; i < n; i++ {
		go test(wg)
	}

	wg.Wait()
}

What did you expect to see?

I want the process to consume less memory.

What did you see instead?

Use SetFinalizer:
Actual memory usage 101GB
VmRSS: 106328688 kB

Unuse SetFinalizer:
Actual memory usage 4.5GB
VmRSS: 5481500 kB

[randall77]

I'm presuming that this is because the finalizer thread (which is serial) can't keep up with your testing goroutines (which run in parallel). From the SetFinalizer docs:

A single goroutine runs all finalizers for a program, sequentially.

Does this happen for smaller n?

[ianlancetaylor]

In any case runtime.SetFinalizer is not free. It does take memory to record the fact that a finalizer exists.

[zhangqiang-01]

It can only be this way， runtime.SetFinalizer will bring a relatively high budget.

I set n = 10.
Use SetFinalizer:
actual memory usage 9GB。
Unuse SetFinalizer:
actual memory usage 530MB。

I set n = 1
Use SetFinalizer:
actual memory usage 610MB。
Unuse SetFinalizer:
actual memory usage 147MB。