proposal: x/mod: new repo for module mechanics

[rsc]

For the initial development of module support, we've kept everything in internal directories to make it easier to make changes as our understanding of what the pieces should look like becomes clearer. But now I think the basic pieces have become pretty clear, and we should think about exporting some packages to help other people who want to write tools working directly with module mechanics.

Note that I said "module mechanics". This is about supporting direct manipulation of modules themselves. It is not about supporting general development tools that need to do things like load packages in module mode. That use case, where modules are incidental rather than the focus, should remain in x/tools, specifically x/tools/go/packages. And the specific case of loading packages should still be done by invoking the go command, which will remain the single point of truth for those algorithms.

I propose to create a new repo golang.org/x/mod that will initially contain:

• golang.org/x/mod/module
  • defines type module.Version (struct {Path, Version string}) and !-encoding of paths and versions (from cmd/go/internal/module)
• golang.org/x/mod/modfile
  • go.mod file reader, editor (from cmd/go/internal/modfile)
• golang.org/x/mod/semver
  • semantic version parser (from cmd/go/internal/semver)
• golang.org/x/mod/sumdb
  • go checksum database client and server protocol (from golang.org/x/exp/sumdb/internal/sumweb)
• golang.org/x/mod/sumdb/storage
  • go checksum database storage layer (from golang.org/x/exp/sumdb/internal/tkv, database)
• golang.org/x/mod/sumdb/tlog
  • transparent log (from golang.org/x/exp/sumdb/internal/tlog)
• golang.org/x/mod/sumdb/note
  • signed note formats (from [golang.org/x/exp/sumdb/internal/note] (https://godoc.org/golang.org/x/exp/sumdb/internal/note))
• golang.org/x/mod/sumdb/dirhash
  • definition of h1: directory tree hash used in go.sum file (from cmd/go/internal/dirhash)
• golang.org/x/mod/sumdb/googlecloud...
  • sub-module with basic google cloud server bindings (from golang.org/x/exp/sumdb/internal/googlecloud...)
• golang.org/x/mod/proxy
  • go module proxy client and server protocol (like x/mod/sumdb; not written yet)
• golang.org/x/mod/index
  • go module index client and server protocol (like x/mod/sumdb; not written yet)
• golang.org/x/mod/gosumcheck
  • command: go.sum checker (from golang.org/x/exp/sumdb/gosumcheck)
• golang.org/x/mod/gomodproxy
  • command: trivial go module proxy for reference and trivial uses (not published yet; maybe also “go mod proxy”)

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

/cc @bcmills @jayconrod @ianthehat @myitcv @rogpeppe

[bcmills]

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

That's #28101.

[dmitshur]

I propose to create a new repo golang.org/x/mod

Does this proposal include a plan for how the new module, modfile, semver, etc., packages will be kept in sync with the ones in cmd/go/internal/...?

[bcmills]

I would assume that they'll be vendored in using go mod vendor.

That does imply that we'll have to have a clean (and perhaps smaller) API boundary between golang.org/x/mod and cmd/go/internal, but that's arguably a good thing anyway.

[rsc]

@dmitshur, yes they would be vendored into cmd/vendor using "go mod vendor".
All the ones I've chosen should already have clean API boundaries, and if not, we'd work on that.

[ianthehat]

I need quite a few of these in order to support mod files in editors. I have an experiment I have been working on that has them copied to an internal directory of x/tools, but it would be much nicer if I could go straight to using this repository instead.

[hyangah]

Consider pseudo-version related logic as well
e.g. https://go.googlesource.com/go/+/refs/heads/master/src/cmd/go/internal/modfetch/pseudo.go

[jayconrod]

The list of packages above looks good. I expect this will be useful for developing gorelease and a lot of other tools.

I notice that none of internal/modload, internal/modfetch, internal/modfetch/coderepo, internal/mvs are in the list. I think that's the right decision for now -- these are all pretty tightly coupled with cmd/go. It might be worth moving these to x/mod after some refactoring though. I think the proxy and discovery projects have run into use cases where query and download functionality would be useful, and the Go command doesn't do quite what they need.

[rsc]

Sentiment here seems overwhelmingly in favor. Accepting proposal.