proposal: cmd/go: support selecting a VCS transport

[draftcode]

Problem

Git supports multiple transports. The built-in transports are git://, ssh://, https://, but it can support a custom transport, such as persistent-https://. Git repository hosting services typically support multiple transports.

With #26232, it will become possible to support private Git repositories with go-get. A server that supports ?go-get=1 URLs needs to return one <meta name="go-import"> tag. However, the server cannot tell which Git transport URL it should return.

For example, https://git.mycompany.com/private-repo is a Git repository that can be accessed with git clone https://... and git clone ssh://.... Both require an authentication. A user may choose to use https:// or ssh:// for git clone. When https://git.mycompany.com/private-repo?go-get=1 returns a tag <meta name="go-import" content="git.mycompany.com/private-repo git https://git.mycompany.com/private-repo">, users who set up credentials only for SSH cannot run go get because it internally tries to run git clone https://git.mycompany.com/private-repo and it fails because of the lack of credentials. The opposite is also true. If the server returns a tag with ssh://..., users who use https://... cannot run go get.

Multiple workarounds exist for this problem.

• Use Git's url.insteadOf
• Use a different go-import URL for a different transport (like golang.org for go.googlesource.com. Though, probably with cmd/go: define HTTP authentication extension mechanism #26232, we can support go-get=1 directly on googlesource.com.)
• Encourage users to set up credentials for the transport that go-get=1 URL returns
• Abuse GOAUTH so that it sends an indicator for the transport preference, assuming that the user can change the server side

Because of these workarounds, this problem is not critical. But it's nice if it's addressed.

Goal

Let users choose a VCS transport for go get git.mycompany.com/private-repo.

Idea 1 (add a query param for selecting a transport)

In addition to go-get=1, go get will add another query parameter so that the server can decide which go-import URL it returns. A user specifies GOGET_VCS_TRANSPORT=git-ssh in the environment variables, then go get will fetch "https://git.mycompany.com/private-repo?go-get=1&vcs-transport=git-ssh". The server returns a different URL based on the added query param.

A nice part of this option is that this is compatible with old/new client/server. A new server that supports this new query param can be compatible with the old go toolchains. When the new go toolchains that support GOGET_VCS_TRANSPORT fetch a URL with a new query param, as long as the server ignore unknown params, it's compatible with old servers.

We can abuse GOAUTH mechanism for this purpose. GOAUTH is meant for auth, but users can add any HTTP header that is not related to auth. A user can write a GOAUTH program that adds a special HTTP header that specifies their VCS transport preference and a supported server can return a different go-import URL based on that.

Idea 2 (let the server return multiple go-import URLs)

Let servers return multiple go-import tags with a different transport. The go toolchains need to choose which one it uses.

This might lose the backward compatibility, so I'm not sure if this is a good option.

[liujianping]

Idea3(local go get config for multiple VCS paths)

If there's a go get config ~/.go/get or current/mod/.go/get, user can configure different VCS paths. Configures like the following:

[my.private.repo.prefix]
vcs = git
root=ssh://git@git.my.private.repo:2201

[some.blocked.domain.prefix]
vcs = mod
root = https://goproxy.io

# others will go get through standard flow

Because, env var $GoProxy will turn off the Go Get Import Meta method， for the private repo, use the ssh+git will be more secure

[bcmills]

This is closely related to #26134.

[bcmills]

Let servers return multiple go-import tags with a different transport. […] This might lose the backward compatibility

I think it's possible to allow the server to return multiple go-import URLs in a backward-compatible way.

cmd/go/internal/get.matchGoImport currently issues an error if multiple go-import tags match, but...
cmd/go/internal/get.parseGoImports currently ignores go-import tags that don't provide exactly three values.

So, if I understand correctly, we can add go-import fallback tags without breaking existing go binaries as long as we use additional (redundant) tags with four or more fields in the content attribute.

[bcmills]

The tricky part about multiple paths, I think, is attempting them in the right order. We can (and currently do) use GIT_TERMINAL_PROMPT to suppress the password prompt, so the question is, is there an order that will always succeed (without trapping the user in a password prompt) if appropriate credentials are present?

Consider:

• If the server is configured to allow password authentication on the SSH endpoint but the user has HTTPS credentials, then if we try SSH first with terminal prompts enabled, the user will receive a spurious SSH login prompt.
• If the user has configured git credential for HTTPS authentication, but has SSH credentials for the host in question, and we try HTTPS first with terminal prompts enabled, then the user will receive a spurious HTTPS login prompt.

Also note that we have existing logic to probe the scheme of an arbitrary VCS server — we just don't use that for go-import tags today.

[bcmills]

Per https://tip.golang.org/cmd/go/#hdr-Remote_import_paths, the go-import response today is defined as:

<meta name="go-import" content="import-prefix vcs repo-root">

where

The repo-root is the root of the version control system containing a scheme and not containing a .vcs qualifier.

So perhaps the scheme-probing version could be something like:
<meta name="go-import" content="import-prefix vcs repo-root...">
where repo-root... is a list of complete URL paths (each containing a scheme) to try in order.

For example, the tags returned for github.com/golang/protobuf/proto might be something like:

<!--for new clients-->
<meta name="go-import" content="github.com/golang/protobuf git https://github.com/golang/protobuf ssh://git@github.com/golang/protobuf">
<!--for older clients-->
<meta name="go-import" content="github.com/golang/protobuf git https://github.com/golang/protobuf">

That would combine with GOAUTH in the following way: if the user provides credentials via GOAUTH, then the server can map the provided HTTPS username to an ssh username as appropriate.

[bcmills]

CC @jayconrod

[rsc]

I think we should leave this alone. The fact that Git has lots of complexity does not require us to import that complexity into the go command.

If we are talking about servers with <meta> tags, presumably the server operator has some sense of the preferred transport. They can use that scheme in the URL in the <meta> tag. Users who for whatever reason disagree with that transport can use insteadOf.

[cstockton]

I've gone through as many of the linked issues as I can find and want to make sure my understanding is correct:

For a Go import to work with the Go tooling there must be an available https endpoint available at the canonical URL of the repository

It appears that this "proxy" declares such a minimal amount of data:
<meta name="go-import" content="example.org git https://code.org/r/p/exproj">

Was there already a discussion on just specifying this inside a config file or env vars like I do for gitconfig? Many of the repos I work on regularly are only available via ssh and have no public https endpoint. As the toolchain has got more complex and I'm trying to adapt to modules I don't have an alternative anymore for writing Go. It seems all Go programmers are expected to eventually run a local proxy server ... just to map a hostname to a git url (as done by my .gitconfig). I feel like this use case is common and handled perfectly in .gitconfig, I've managed to expand the simple rewriting for every configuration I've come across. It's small, concise and easy to maintain.

I understand that the proxy ship has probably sailed and will run it if I must, but I can't seem to find the code or how to run and configure it? Maybe there is another way for my simple use case of using my preferred VCS settings for obtaining code remotely?

[bcmills]

@cstockton

Was there already a discussion on just specifying this inside a config file or env vars like I do for gitconfig?
Many of the repos I work on regularly are only available via ssh and have no public https endpoint.

Search for “.vcs suffix” in https://tip.golang.org/cmd/go/#hdr-Remote_import_paths. If we know that the module path is a VCS path, then we do try the VCS directly.

[cstockton]

@bcmills Thank you for the suggestion, I noticed the link was for tip but I gave it a whirl anyways and was pleased to see the execve calls and it immediately downloaded the module. It does fail a check against the downloaded modules name as seen below:

go: _internal.tld_/pkg/acme.git@v0.0.0-20190314120038-dededededed: parsing go.mod: unexpected module path "_internal.tld_/pkg/acme"
go: error loading module requirements

Does the module need a git suffix as well, i.e. module _internal.tld_/pkg/acme.git or is this something that should work come 1.13 as is?