cmd/go: directories named `vendor` are ignored in `//go:embed`

[jakelucas]

What version of Go are you using (go version)?

$ go version
go version go1.16.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=D:\projects\go\bin
set GOCACHE=D:\devenv\go\1.16.3\..\cache
set GOENV=C:\Users\\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\projects\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\projects\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\devenv\go\1.16.3
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\devenv\go\1.16.3\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.3
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\~1\AppData\Local\Temp\go-build3923578036=/tmp/go-build -gno-record-gcc-switches

What did you do?

Used //go:embed <dir> to recursively embed static website assets into an embed.FS variable, and then ran go install <repo> to install the binary into gobin.

What did you expect to see?

All assets embedded into the embed.FS variable in the final binary in gobin.

What did you see instead?

Only some assets were embedded in the embed.FS variable in the final binary in gobin.

The problem comes from having folders named vendor within the directory that gets embedded.

If you create a project that embeds a directory, for example //go:embed static, which includes some sub-directory named vendor, for example static/vendor/style.css, then the embed behaviour is different depending on how you install the project into gobin.

If you have the entire project source locally and you run go install within the project then the vendor directory is correctly included and embedded into the final binary.

If you push the project to somewhere like GitHub and then try to install via go install <repo> or go get <repo>, then all files are correctly embedded according to the documentation except for the entire vendor folder, which is ignored. You can work around this by setting GO111MODULE=auto. If this is set then the vendor directory is no longer ignored and is correctly embedded in the final binary.

I believe this is just an unintended side-effect of go get/go install and how they deal with vendored dependencies. If it is actually intended then it at the very least needs to be documented and made consistent across the different methods of installation.

[cherrymui]

cc @bcmills @rsc

[rsc]

We can only embed files that are saved in the packaging of the module, and directories named vendor are excluded from the module. Therefore they need to be excluded from the embed as well.

[patrickbsf]

Could you just make sure it's at least mentioned in the go:embed documentation at least? This isn't obvious.

[bcmills]

This is currently mentioned in the embed package does, although it could stand to be clearer for vendor in particular.

Today (https://pkg.go.dev/embed@go1.17.2) it says:

Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links. Matches for empty directories are ignored. After that, each pattern in a //go:embed line must match at least one file or non-empty directory.

Perhaps we are also missing a check to cause the build to error out if the pattern does match files in the vendor directory? That should be an explicit error — it shouldn't implicitly drop files that match, although things get a bit tricky when we consider glob patterns (which should just stop at module boundaries without erroring out).

[gopherbot]

Change https://go.dev/cl/633815 mentions this issue: embed: document exclusions more explicitly