cmd/go: go get doesn't work with private gitbucket repo

[tundratim]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8

What operating system and processor architecture are you using (go env)?

x64 RHEL7

What did you do?

Attempt to do a deep import (go get) on private gitbucket repo

What did you expect to see?

Correct import

What did you see instead?

Fails because clone URL is not getting correctly parsed from gitbucket

I am doing devops support for a client doing a bunch of new go stuff. They are using an internal instance of gitbucket for their corporate code. I am not a go expert by any means, but as I understand it ....

• Ordinary "go get" works fine, but
• A "deep" get - somewhere down the repo tree fails because the resultant URL go tries to use is wrong

One of the devs showed me the list of rexeges baked into go for the various vcs products out there, but gitbucket seems not to be on there as best as I could tell.

So ... I guess the ask is either:

• Fix go to properly interact with gitbucket
• Provide some clean way for devops folk to add new vcs regex definitions without having to recompile the language.

Any thoughts here would be welcome.

[bradfitz]

See go help importpath (or https://golang.org/cmd/go/#hdr-Remote_import_paths). While there are some specially-recognized cases, any URL works fine if it has the right <meta ...> tags.

gitbucket is not a hosted service, right? So we couldn't regexp it anyway, since users can run it on any hostname?

So I think the best option is for you to modify gitbucket to add the right <meta> tags when cmd/go makes a request to it. The cmd/go command always sends a request with ?go-get=1, which gitbucket can check for and go into a special code path.

Let me know if I misunderstand, but I think there's nothing for us to do here so I'm going to close this. Holler if I should reopen.

[tundratim]

I think the essence of the problem occurs when we use a git repo not in your list of public servers, we default to the regex found at line 902 in:

https://golang.org/src/cmd/go/vcs.go

This regex creates the following problem as I understand it: For any git repository, if you go get domain.of.repo/repo/name.git, go tries to clone domain.of.repo/repo/name instead of name.git.

IOW, the belief by our devs is that this is incorrect behavior for the default case.

As an aside - I see more and more customers spinning up their own repos on their own hosts wherein this default behavior might come back to bite them.

P.S. We also believe there is an issue with GitBucket itself here as well:

If you try to import domain.of.repo/repo/name/deep/import, go tries to http get domain.of.repo/repo/name/deep/import, which 404s, because gitbucket serves that path at domain.of.repo/repo/name/tree/master/deep/import.

Thanks for the quick turnaround here!

[bradfitz]

What is your concrete problem?

And you didn't reply to my suggestion of using the Go <meta> tags and adding support for that to gitbucket.

[bradfitz]

(And by concrete problem, I mean to give me an actual command go get command that fails.)

[tundratim]

I have contacted the devs involved and asked for more specific details.

[bentucker]

Hey Brad,

Here's the issue that I'm seeing:

$  go get -v git.example.local/repo/repo-name.git
# cd .; git ls-remote https://git.example.local/repo/repo-name
fatal: repository 'https://git.example.local/repo/repo-name/' not found
# cd .; git ls-remote git+ssh://git.example.local/repo/repo-name
Unknown command: git-upload-pack '/repo/repo-name'
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://git.example.local/repo/repo-name
Unknown command: git-upload-pack '/repo/repo-name'
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Please note that git ls-remote ssh://git.example.local/repo/repo-name.git works as expected.

From what I can tell, the default vcsPath is being matched, and the regexp is stripping the '.git' from the vcs repo. (At line 902 in vcs.go)

[bradfitz]

What does:

git ls-remote https://git.example.local/repo/repo-name.git

... say?

[bentucker]

That returns the same references that are returned by git ls-remote ssh://...

[rsc]

If your Git server is not one of the recognized ones (say, Github) and not pointed at by a custom domain using tags, then the final fallback is that we'll recognize x.git/y as meaning checkout x with git and then look at y inside the repo. Note that some Git servers (like Github) make the .git optional so that x and x.git mean the same thing, which can be a source of confusion here, but the .git in this syntax is so go get knows to use git at all, not sent to the server. Otherwise it would not be possible to use a repo that didn't end in .git.

If on your server x needs to end in .git, you need to say it twice, once for your server and once for go get:

go get -v git.example.local/repo/repo-name.git.git

Of course that looks awful but it's implied by the use of this fallback. In general it's far preferable to set up an HTTPS redirector to let you separate the import paths from the exact location of the git servers (for example curl https://rsc.io/pdf). I'd suggest doing that if .git.git is deemed too ugly.

The fallback here predates the custom URLs. If I had it to do again I'd probably just remove the fallback entirely and require HTTPS tags.

[hex108]

Hi @bradfitz @rsc , could we add a configure to add our own git servers like the public serves in code(https://golang.org/src/cmd/go/vcs.go)? Then go get could skip sending a request with ?go-get=1. It is very useful for those private repos that does not response ?go-get=1 request correctly.

[bcmills]

#27344 appears to be a duplicate of this issue. The steps to reproduce the error there are easier for me to follow, so I'm going to consolidate the issue there.