proposal: cmd/go: support local experiments with interdependent modules; then retire GOPATH

[ohir]

Go "tinker mode" proposal.

Last updated: 2021/04/11

Rationale:

@cosban wrote: I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications. (@cosban)

---

@rsc wrote: GOPATH is holding back the ecosystem and the toolchain. It's time to retire it. [...] we're happy to listen. But GOPATH needs to go. (@rsc)

---

@bcmills wrote: Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules (@bcmills)

The main workaround at the moment is to add replace directives among the modules to be edited, but maintaining those directives is tedious and error-prone. @rogpeppe's gohack tool automates away some of the tedium, but doesn't seem to remove the risk of accidentally checking in a go.mod with what were intended to be local, temporary replacements.

---

Related: #26640, #37755, #26377, #25053.

If an environment variable named GOTINKER is defined and set to an absolute path to the existing location on the local filesystem, and the import path of an include can be found under $GOTINKER/ directory, then build commands treat the $GOTINKER/src/import/path as a final authoritative source of the import; foregoing both vendor/ and any go.mod directive perpeting to this import/path.

Ie. build commands like 'go build' and 'go test' will compile modules present in the $GOTINKER directory instead of accessing the network, local module cache, or vendor directory.

Under tinker mode GOBIN, GOCACHE, GOMODCACHE, and GOENV are bound to locations relative to the GOTINKER: $GOTINKER/bin, $GOTINKER/cache, $GOTINKER/pkg/mod, and $GOTINKER/goenv - respectively.

$GOTINKER tree should be populated by the user. For yet some time to come the last version of Go to support GOPATH can be used to ease this task, ie. GOROOT=/where/go1.16 GO111MODULE=off GOPATH="$GOTINKER" go1.16 get

GOTINKER path last element may start with an underscore character so experiments can be kept inside any project tree.

Security considerations

Both object code and the executable built under the tinker mode should not accidentally leak to the production environment. Ie. while objects are built and cached under $GOTINKER, then built executable MUST be amended (by the compiler) to refuse to run in an environment where GOTINKER is not set, or it is set but does not match the last element of the GOTINKER path that was compiled in.

Ie. "tinkered with" executable preserves the last part of the GOTINKER path then matches it to the last part of GOTINKER string where it runs. If these do not match, exectutable exits immediately with "Experimental but GOTINKER is not set or did not match" error message.

---

How GOTINKER workflow is different to GOPATH''s one?

GOPATH is all-or-nothing regarding versions. Ie. under GOPATH tools operate on code as-is. If we are about debugging or changing interdependent code, it is up to us to do proper checkouts of everything our — possibly big — app or service uses.

In proposed GOTINKER mode tools operate on modules as usual, so anything else but code we pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for us, nothing will drift apart or leak accidentally.

---

edits:

• 2021/02/17 added "match compiled in GOTINKER with run one" in Security
• 2021/02/18 be explicit about possibility to keep GOTINKER tree inside a module tree, eg. in a _tinker/ subdirectory.
• 2021/04/11 compare GOPATH and GOTINKER workflows.

[ianlancetaylor]

CC @bcmills @jayconrod

[jayconrod]

cc @matloob @stamblerre @ianthehat

[complyue]

I like this proposal more than other alternatives to the problem, hope it get more attention.

I used to have GOPATH=/globally-cached/go-deps:/personal-tinkering/go-devs:/team-tinkering/go-pkgs, works perfectly well with organization-wide shared filesystems.

And I hesitate to pickup further development of my rusted Go projects at this time, just because go.mod is the right way to go, but it still lacks sufficient support rival to my historical workflows with GOPATH like that.

[Merovius]

I don't understand the significant difference between this proposal and just continuing to use GOPATH (and, I guess, giving it priority over modules). That is, GOTINKER seems to be almost identical in its interpretation to GOPATH. The most significant deviation seems to be the "refuse to run if GOTINKER is not set" part, correct? I don't see the benefit of that - to me, if the binary is able to run with some environment variable is set, even if it doesn't look at that, this seems to serve as conclusive proof that setting this environment variable doesn't provide any benefit.

[ohir]

That is, GOTINKER seems to be almost identical in its interpretation to GOPATH.

Yes it is. Yes — almost: the main and important difference is that it operates in module "units".

There are modules under the GOTINKER/src, not packages.
I can do cd‍ _tinker/src/import/path, clone/pull the source, mod-tidy it, then make changes at will. Repeat for other dependent modules pulled to _tinker/. When bug is fixed i can tag and push all the fixes to the modules repo simultanously.

Same for bootstraping and early experimenting with code structure.

When I am done with experiments, simple rm -rf _tinker && unset GOTINKER cleans up my workspace.

The most significant deviation seems to be the "refuse to run if GOTINKER is not set" part, correct?

The most significant deviation is described above. "Production fuse" is the second, along with builddirs moved inside GOTINKER. Both prevent experimental code leaks to the CI/CD pipelines. It shouldn't happen but it happens — and when it does, it hurts a lot.

if the binary is able to run with some environment variable is set, even if it doesn't look at that

It looks at, and if it sees wrong environment it gently refuses to do any work except saying why it won't run "here".

setting this environment variable doesn't provide any benefit.

To the developer, probably not much - she already has it set. But deployment people will be less nervous when asked to fire up a field test.

and just continuing to use GOPATH

GOPATH destiny, according to previous discussions, is to retire. GOTINKER needs the least code I could think of to get GOPATH benefits back. It boils down to:

• check GOTINKER first; if set do:
• in-memory: set GOENV (BIN CACHE)
• scan the tree under GOTINKER
• for every import/path found there and seen in go.mod too, recursive:
  • in-memory replace go.mod sourced entry relating to import/path with

  require import/path v0.0.0-unpublished
  replace import/path v0.0.0-unpublished => {{tinkerpath}}/src/import/path

...(skip code in vendor)...

[Merovius]

There are modules under the GOTINKER/src, not packages.
I can do cd‍ _tinker/src/import/path, clone/pull the source, mod-tidy it, then make changes at will. Repeat for other dependent modules pulled to _tinker/. When bug is fixed i can tag and push all the fixes to the modules repo simultanously.

Apparently I'm still not getting it. This doesn't sound very different to me. For example, I currently have GOPATH=$HOME and under ~/src/github.com/Merovius/nbd I have a checkout of the module module github.com/Merovius/nbd - i.e. the module is in the import path, relative to GOPATH. If I set GO111MODULE=off, I can cd ~/src/github.com/Merovius/nbd, work in GOPATH mode (and thus use the rest of my ~/src, where the rest of the modules I work with is also checked out under the correcti mport paths), commit and push. All of that seems exactly the layout of what you describe when you say GO111MODULE=off go get … can be used for the setup of GOTINKER.

So, I really don't see the difference, TBQH. To me, that seems to be exactly the setup and workflow you are describing.

GOPATH destiny, according to previous discussions, is to retire.

Yes. That's why I'm confused. To me, what you are describing is almost exactly the (for now) still existing GOPATH based workflow - except for a renamed environment variable. If we think we need that workflow, it seems more straight-forward not to retire GOPATH . And if we do want to retire GOPATH, surely we wouldn't just re-introduce it under a different name.

[ohir]

If I set GO111MODULE=off, I can [...]

Minus that with 1.17 the GO111MODULE will always be on.

what you are describing is almost exactly the (for now) still existing GOPATH based workflow

It is supossed to be almost identical workflow - except for all tools being now in modules mode, so I can work only with, say, three repositories out of thirty or eighty. Modules that I do not plan to tinker with will come from cache (copied or hardlinked to the $GOTINKER/mod/cache). Also all GOPATH related code, esp. VCS one, may retire.

GOTINKER code will need do a little bit of magic to internal representation of go.mods read — then anything else works in module terms as if you'd edited-in all those require/replace directives into proper places by hand.

If we think we need that workflow

Yes "we" think we need. See Rationale at top.

[Merovius]

If we think we need that workflow

Yes "we" think we need. See Rationale at top.

FTR, by "we" I mean "the Go project", not a specific subset of people. It certainly includes you and me and many others, many of whom likely agree with you and many of whom don't. It's possible for "the Go project" to come to a conclusion, even though some or many of its members disagree with it. The proposal process is how these conclusions are reached and are thus what decides what the "we" I was referring to "thinks".

So, let me rephrase my questions. If the outcome of this discussion is, that the workflow presented here is important enough to implement and given its extreme similarity (by design) to the existing GOPATH based workflow, wouldn't it be preferable to simply not retire GOPATH? And if, on the other hand, there are good, convincing reasons to retire GOPATH, why would those reasons not apply to this proposal? Surely, whether the environment variable is spelled GOPATH or GOTINKER does not affect those reasons?

Alternatively, this proposal is not as similar to GOPATH as it seems to me, which means I misunderstood something. That would certainly make these questions obsolete and provide justification for this proposal. Which is why I tried to understand these differences. So, is it actually the case that the workflow you describe is currently (we are not talking about go 1.17+) captured by setting GO111MODULE=off? If not, what are the differences? Your last answer "GOTINKER operates on modules, not packages" is confusing to me, as GOPATH seemingly contains exactly the same directory layout, containing exactly the same repositories with exactly the same files. But maybe I misunderstood something?

[complyue]

@Merovius As I understand it, one very important difference is that while GOPATH is an exclusive choice against go.mod, GOTINKER can work together with go.mod.

I for myself especially want the benefits of GOPROXY with go.mod on (to workaround GFW as one reason), I can't get those by choosing GOPATH, while GOTINKER is hopeful to provide equally ideal workflows like with GOPATH, and still have all modern & good things from go.mod.

[Merovius]

@complyue

As I understand it, one very important difference is that while GOPATH is an exclusive choice against go.mod, GOTINKER can work together with go.mod.

What does that mean? Concretely? Like, presumably the intent is to actually use local modifications, ignoring versions specified in go.mod. Doesn't that just mean "ignore go.mod"? If there are stanzas of go.mod we don't want to ignore, couldn't we just teach the go tool to use them in "GOPATH-mode"?

FWIW, as far as I can tell so far, the only reason to call it GOTINKER is to avoid association with GOPATH. There is no (significant) technical difference, but rather "GOPATH is going away, but maybe, if we call it GOTINKER, it can stay". That's an uncharitable interpretation, though, which is why I'm trying to coax out the actual differences that are proposed here.

I for myself especially want the benefits of GOPROXY with go.mod on (to workaround GFW as one reason), I can't get those by choosing GOPATH

I don't understand why this would require a new environment variable. To me, this seems to simply mean downloading the zip and unpacking it in the right directory (same as gohack already does). The go tool, with GO111MODULE=off" does not care whether the code in $GOPATHis from a git repository, or a zip-file - AFAIK it even completely ignores the VCS aftergo get`.

[ohir]

FTR, by "we" I mean "the Go project", not a specific subset of people.

I meant specific subset, hence quotation marks.

So, let me rephrase my questions. If [...] the workflow presented here is important enough to implement [...] given its extreme similarity to the existing GOPATH based workflow, wouldn't it be preferable to simply not retire GOPATH?

Probably keeping GOPATH mode would be a least friction solution to the interdependent edits problem in the short term. But, after a period of being in denial, I now second Rob Pike's "GOPATH must go" statement — because the technical debt from keeping GOPATH workflow intact will IMO hurt the ecosystem in the long run. Modules with their clearly, automatically accounted and double-checked versioning mechanics are superior to GOPATH for any software that evolves fast, for long, or both.

That said, modules workflow has a blindspot where interdependent edits of two or more modules come with a huge footgun loaded: you need to be very careful editing your replace directives by hand, in many places, then you need to be very careful again when you need to clean your laboratory table after. Citing @bcmills again:

Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules.

(Just now /04.2021/ I see three discussions about "how to do it with modules" active on the golang-nuts list. Such questions emerge there almost every month since go1.11)

And if, on the other hand, there are good, convincing reasons to retire GOPATH.

GOPATH is all-or-nothing. When it is on, all used modules repos must go there, and proper (re being investigated code) versions must be checked out by hand. Then vendoring status of each must be consulted too. GOPATH mode is currently, ie. in modules filled world, good only for bootstraping new code, it is unusable for the bughunt purposes.

why would those reasons not apply to this proposal? [...]
Alternatively, this proposal is not as similar to GOPATH as it seems to me

This proposal keeps only good things from GOPATH mode — all sandboxed and all implemented by already existing mechanics of the modules.

as GOPATH seemingly contains exactly the same directory layout, containing exactly the same repositories with exactly the same files.

No, not the same files. And almost certainly not the same as being investigated production code uses.

But maybe I misunderstood something?
[...] what are the differences?

In GOPATH mode tools operate on repo's HEAD. It is up to you to do proper checkouts of everything your, possibly big, app or service uses.

In proposed GOTINKER mode tools operate on modules as usual, so anything else but repos you pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for you, nothing will drift apart or leak accidentally.