proposal: cmd/go: extend syntax go.mod to allow overriding fetch protocol

[narqo]

Update

The original proposal was build on the idea that we could extend replace directive (see the original text below).

After several rounds of discussions here in the comments, the current semantics of replace isn't about fetching the dependency but about local substitution of the source path and version (see this comment from @bcmills).

I have re-title the proposal and suggested the new syntax to solve the described problem.

Original Proposal

I propose to extend the syntax of go.mod's replace directive to support overwriting the protocol, which Go will use to fetch the module. That is

replace example.com/foo/bar => git+ssh://private.example.com/foo/bar

Background

As it’s for Go 1.14, the syntax of replace directive in go.mod allows overwriting module’s import path by defining an alternative version, alternative import path, or a path on the local FS (refer to [1]).

For non-FS-based replaces, Go fetches the replaced modules from modules proxy server as it does for other dependencies.

If a project uses private dependencies or private forks to replace a dependency, managing the modules become an extra burden for the developers, who work on the project. Even though the dependency is an attribute of the project itself, it’s the responsibility of every developer of the project to define the correct values for GOPRIVATE, GONOPROXY, etc, which alters their "normal" development workflow.

With private dependencies explicitly listed as the ones, that can't be fetched from any modules-proxy server, the project's workflow wouldn't be any different from a project, that only uses public modules (providing that a developer has the required accesses).

Specifying the protocol explicitly would also solve the issue where “go get” chooses to fetch a private dependency via HTTPS, which usually forces developers to manipulate their host’s .gitconfig, adding custom insteadOf rules to force go-get to use SSH when downloading modules from GitHub repositories.

Prior Experience

Dependency-management systems in other programming languages have similar mechanisms. For example:

• Node.js + NPM
  package.json "dependency" syntax accepts URLs, Git URLs and some other [2].

• Rust + Cargo
  Cargo.toml allows overwriting the source of the dependent crate, passing project’s Git URL [3].

• Go + dep
  Gopkg.toml allows specifying dependencies' source as an alternate location, passing the URL [4].

[mvdan]

I'm a bit confused by the proposal - go.mod only deals with modules and their paths and versions. How go get then fetches those modules is an entirely separate concern.

If we start allowing full URLs in replace directives, should we also allow them in other places that usually get import paths, such as go get commands?

What should happen if I tell go.mod to use git+ssh://, but I have also set up GOPROXY=https://proxy.my.corp? What if go.mod uses https://?

Have you thought about modules where the import path is not a git server? For example, https://mvdan.cc/sh works, but git+ssh://mvdan.cc/sh will not. Conversely, git+ssh://github.com/mvdan/sh will work, since that's the repository URL. But of course, the two URLs are not the same, and go.mod is meant to contain module paths, not source code repository URLs.

[narqo]

Have you thought about modules where the import path is not a git server? For example, https://mvdan.cc/sh works, but git+ssh://mvdan.cc/sh will not. Conversely, git+ssh://github.com/mvdan/sh will work, since that's the repository URL. But of course, the two URLs are not the same, and go.mod is meant to contain module paths, not source code repository URLs.

I'm not sure I got this part right, please, correct me if not. The current restriction is that replace — if used with FS-based paths — must point to a location where go.mod is located (as per Modules wiki).
If you used git+ssh://github.com/mvdan/sh as the replacement for mvdan.cc/sh/v3 it would work because that replacement point to a target with go.mod file, which defines the proper module.

---

I didn't explicitly (or implicitly) proposed to support the whole URL. More to that, I think w/o a formal definition of what is the difference between "package", "module" and URL, the discussion might become very confusing.

We can consider adding the protocol to the existing syntax of go get, when in the module's mode. I.e. go get proto://foo/bar@version.

Regarding the conflicting cases, we should come up with the priorities, yes. I expect that a package, which is replaced with FS-based replace, isn't affected by GOPROXY. I would expect the similar behaviour for packages, whose fetch-protocol is explicitly defined.

[mvdan]

So, in a way, these URL-based replaces would be like local directory replaces - just pointing at a remote location/machine. That makes more sense to me.

We can consider adding the protocol to the existing syntax of go get, when in the module's mode. I.e. go get proto://foo/bar@version.

I would certainly expect go get to not accept URLs, because it similarly does not accept paths to modules on disk like replace directives do.

Similarly, I think supporting proto://foo/bar@version is a mistake. We don't support replace example.com/foo/bar => ../bar@v2.0.0, precisely because there is no protocol at play that would let us figure out exactly what v2.0.0 is. We just support using whatever is found at ../bar, as-is.

You can't assume that the other end will be a git repository or clone, either. This applies for local directory paths, and by extension, your URLs too I would say.

I expect that a package, which is replaced with FS-based replace, isn't affected by GOPROXY. I would expect the similar behaviour for packages, whose fetch-protocol is explicitly defined.

Assuming we piggyback off of local directory replace directives, I agree that these rules become clearer.

[narqo]

So, in a way, these URL-based replaces would be like local directory replaces - just pointing at a remote location/machine.

Exactly. I don't know yet the deeps of the current implementation. But when I wrote the proposal, this was exactly how though about it 👍

[narqo]

@mvdan sorry for the ping but, based on the discussion above, do you have any suggestions to how to move this forward?
I know @rsc and the team look through the "Proposal" issues weekly, although I expect the overall number of those issues is too big to observe.

[ianlancetaylor]

CC @bcmills @jayconrod

[bcmills]

The nice property of filesystem-local replace directives is that the files are already in a location where they are accessible to cmd/compile. In contrast, cmd/compile does not know how to read a git+ssh:// URL, so the go command would have to clone it somewhere — and where would it put that? (It presumably can't go in the module cache, because git+ssh:// is not a well-formed module path component and is not immutable in the same way as a versioned module anyway.)

[narqo]

@bcmills I feel there is a confusion or maybe I don't follow this part:

so the go command would have to clone it somewhere — and where would it put that? (It presumably can't go in the module cache, because git+ssh:// is not a well-formed module path component and is not immutable in the same way as a versioned module anyway

It could be, that the description didn't show the idea clearly — I will appreciate suggestions to improve the wording — but I expected the same behaviour for GOPRIVATE=foo/bar go build and a replace foo/bar => git+ssh://foo/bar. In the replace case though go command would bail out if foo/bar couldn't be fetched using the specified protocol.

[bcmills]

@narqo, the replace directive performs a local substitution of the source code for the given module path (and possibly version), so it changes the meaning of a path@version pair (and may change the content to have a different checksum). In contrast, GOPRIVATE does not change the meaning of a specific version — it merely instructs the go command not to fetch that version directly from the origin, but expects the content to have the same (consistent, immutable) checksum.

If what you have in mind is analogous to the behavior of GOPRIVATE, then the replace keyword is not a good semantic fit.

[narqo]

If what you have in mind is analogous to the behaviour of GOPRIVATE, then the replace keyword is not a good semantic fit.

Good point. My initial hopes were to piggyback the existing replace to get the behaviour of GOPRIVATE, but allowing to explicitly require this behaviour in a project's go.mod. Providing that, this goal makes sense, maybe a new explicit source directive would solve this better:

// go.mod

require foo/bar 1.0.0

source foo/bar => git+ssh://git.corp.com/user/repo

That is, source directive instructs go, that module foo/bar must be fetched from the origin git.corp.com/user/repo using git over SSH protocol.

What do you think?