cmd/go: get doesn't work with repositories from git.apache.org

[kostya-sh]

Apache maintains read-only Git mirrors at http://git.apache.org/ (See http://www.apache.org/dev/git.html for more details). Only HTTP and Git protocols are supported (i.e. no HTTPS at the moment)

go get fails to clone such repositories. E.g.:

> go get git.apache.org/thrift.git/lib/go/thrift
fatal: repository 'http://git.apache.org/thrift/' not found

This happens because go get extracts incorrect remote repository URL from import path.

E.g. for git.apache.org/thrift.git/lib/go/thrift import path the remote URL used by go is http://git.apache.org/thrift/ while the correct remote URL should be http://git.apache.org/thrift.git (compare git ls-remote http://git.apache.org/thrift.git vs git ls-remote http://git.apache.org/thrift/)

To fix this a special case needs to be added for git.apache.org in cmd/go/vcs.go similar to (github, launchpad, etc).

I am happy to contribute and test a patch if adding support for another special Git repo to go get seems to be reasonable to Go Team.

[bradfitz]

Another special case is fine. Send it in. See https://golang.org/doc/contribute.html if you haven't already.

[kostya-sh]

https://go-review.googlesource.com/#/c/10092/

[gopherbot]

CL https://golang.org/cl/10092 mentions this issue.

[nightlyone]

Hmm go get git.apache.org/thrift.git/lib/go/thrift works right now without any changes, because go get prefers the schema git:// over everything else. Is your goal to prefer the http schema instead?

[bradfitz]

In Go 1.5, neither of those will work without --insecure.

[kostya-sh]

Sorry for the confusion.

But when I tested this a few days ago git:// scheme was not available (I was getting connection refused errors. And no, I am not behind firewall ;). In fact other users reported similar problems 1 year ago: https://issues.apache.org/jira/browse/THRIFT-2537. So I assumed that apache only provides http:// scheme.

Still I believe to make git.apache.org repos work with Go 1.5 without the need to specify --insecure there are two things required:

• git.apache.org provides HTTPS
• my patch applied

If/when apache adds HTTPS support I will test my change request again. But if there will be no plans to add HTTPS support, then I guess this ticket can be closed. Lets wait some time and see what happens.

[nightlyone]

Ok , I think I found the culprit: All git urls passed to git are actually intended to end in .git. For the git:// schema, git seems to be clever and adds it, when it is missing, so it works by accident.

The fix might be as simple as

diff --git a/src/cmd/go/vcs.go b/src/cmd/go/vcs.go
index 2179000..876fd36 100644
--- a/src/cmd/go/vcs.go
+++ b/src/cmd/go/vcs.go
@@ -104,7 +104,7 @@ var vcsGit = &vcsCmd{
        name: "Git",
        cmd:  "git",

-       createCmd:   "clone {repo} {dir}",
+       createCmd:   "clone {repo}.git {dir}",
        downloadCmd: "pull --ff-only",

        tagCmd: []tagCmd{
@@ -123,7 +123,7 @@ var vcsGit = &vcsCmd{
        tagSyncDefault: "",

        scheme:     []string{"git", "https", "http", "git+ssh"},
-       pingCmd:    "ls-remote {scheme}://{repo}",
+       pingCmd:    "ls-remote {scheme}://{repo}.git",
        remoteRepo: gitRemoteRepo,
 }

But I'm adding tests for repoRootForImportPathStatic now, to avoid regressions. I will add also maximum coverage to it, to avoid further breakage there.

Stay tuned...

[nightlyone]

Ok, to actually test this, I would need to write some mocking around the execs and fake their results.
The returned repoRoot structure in the current test setup stays the same :-/
So the current tests work the same before and afterwards.

Actually the difference between the createCmd and pingCmd is pretty obvious and lead to same or better results.

The amount of code added would be a few lines and will check command construction (unit testing), not command execution (integration testing).
It might look like this: https://github.com/Jimdo/periodicnoise/blob/master/cmd/pn/monitor.go#L102-L146

But I am not really comfortable adding so much (test) code that late in the release cycle and I am also unsure with what repositories the generic repository stuff handling has been tested. It would be the most affected.

Added https://golang.org/cl/10171 for it.

@bradfitz Please advise how to move forward here.

[gopherbot]

CL https://golang.org/cl/10171 mentions this issue.

[kostya-sh]

@nightlyone

As far as I am aware ".git" suffix is just a convention and not a requirement for Git URLs.

When using git:// scheme in case if git://repo exists (and git://repo.git doesn't):

• Git won't attempt to read from git://repo.git
• Git will fail to read from git://repo.git URL.

So basically if someone sets up a repository without ".git" suffix "go get" with the proposed change won't work.