Skip to content

Conversation

jwebb
Copy link

@jwebb jwebb commented Nov 14, 2022

This is I believe a more correct version of #31 which was attempted and reverted back in 2018. The problem currently is that use of unsafe.Pointer leads to cgo defensively inserting calls to this monster: https://github.com/golang/go/blob/master/src/runtime/cgocall.go#L393

Aside from the cost of running that code, it also triggers allocations in order to wrap the slice headers into the any arguments. This is also the cause of the panics worked around previously in 0ead11a

By passing char* rather than void* we allow cgo to understand that there cannot possibly be any pointers hiding in our data buffers, and skip the checks entirely. In principle this could be applied to every use of unsafe.Pointer, but the three updated here seem the most likely cases to be used with small inputs, where the performance effect is noticeable.

Before:

$ PAYLOAD=LICENSE go test -bench='BenchmarkCompression|BenchmarkDecompression' -benchmem -run='^$' .
goos: linux
goarch: amd64
pkg: github.com/DataDog/zstd
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkCompression-12      	   75825	     15826 ns/op	  98.51 MB/s	      48 B/op	       2 allocs/op
BenchmarkDecompression-12    	  295032	      3954 ns/op	 394.28 MB/s	      72 B/op	       3 allocs/op
--- BENCH: BenchmarkDecompression-12
    zstd_test.go:339: Reduced from 1559 to 855
    zstd_test.go:339: Reduced from 1559 to 855
    zstd_test.go:339: Reduced from 1559 to 855
    zstd_test.go:339: Reduced from 1559 to 855
PASS
ok  	github.com/DataDog/zstd	6.293s

After:

$ PAYLOAD=LICENSE go test -bench='BenchmarkCompression|BenchmarkDecompression' -benchmem .
goos: linux
goarch: amd64
pkg: github.com/DataDog/zstd
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkCompression-12      	   75852	     15731 ns/op	  99.11 MB/s	       0 B/op	       0 allocs/op
BenchmarkDecompression-12    	  296256	      3645 ns/op	 427.73 MB/s	       0 B/op	       0 allocs/op
--- BENCH: BenchmarkDecompression-12
    zstd_test.go:340: Reduced from 1559 to 855
PASS
ok  	github.com/DataDog/zstd	7.998s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant