x/tools/cmd/godoc: ignored "too many open files" errors can cause incomplete /pkg/ subdirectories listings

[dmitshur]

What version of Go are you using (go version)?

go version go1.6rc2 darwin/amd64

What operating system and processor architecture are you using?

OS X 10.11.3 (64-bit).

What did you do?

1. Have a GOPATH workspace with many, many packages.
2. Run godoc -http=localhost:6060.
3. Visit http://localhost:6060/pkg/crypto/ (it's used as an example, it affects all packages equally).

What did you expect to see?

A complete list of subdirectories, with their package synopses.

What did you see instead?

Randomly incomplete listing of subdirectories, and some missing package synopses.

[dmitshur]

I've looked into why it happens. At first I thought it might be some race condition, but I've learned it's not.

Instead, the problem is here:

https://github.com/golang/tools/blob/fe74a4186116b8d7dd38a723993e0d84f8834b34/godoc/dirtrees.go#L107-L127

Because all GOROOT and GOPATH workspaces are walked in parallel (likely with some concurrency limits), it may run into "too many files open" limits and erroring out. The linked code segment ignores all situations where parsing the .go files results in errors (there's no if err != nil { ... } section).

I've added a simple else if err != nil { log.Println(err) } section there and I saw many "too many files open" errors occur both on .go files in GOROOT as well as in GOPATH during my test.

[bradfitz]

@adg improved this in https://go-review.googlesource.com/24060

I'll send a bit more.

[gopherbot]

CL https://golang.org/cl/47651 mentions this issue.

[kevinburke]

I think this has been addressed. At the very least, I have a pretty large source tree (every Github checkout I do in any language is in GOPATH) and I've never seen this issue, nor a godoc server with randomly missing files.

[kevinburke]

Closing, feel free to reopen if this is still a problem, or there's a way to reproduce on tip.