cmd/go: go get fails with private bitbucket repositories

[savaki]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOOS="linux"
GOARCH="amd64"

What did you do?

I attempted to go get from a private bitbucket repository.

1. add the following to ~/.gitconfig

[url "[email protected]:"]
	insteadOf = https://bitbucket.org/

2. add the following to ~/.ssh/config

Host bitbucket.org
  HostName bitbucket.org
  User git
  IdentityFile ~/.ssh/id_bitbucket
  StrictHostKeyChecking no

3. ensure that a valid ssh key exists for bitbucket, ~/.ssh/id_bitbucket

4. use go get on a private bitbucket repo with modules enabled

What did you expect to see?

I expected the private repository to be pulled to the appropriate ${GOPATH}/pkg/mod
directory.

What did you see instead?

go: bitbucket.org/myprivaterepo/[email protected]: https://api.bitbucket.org/2.0/repositories/myprivaterepo/projectname?fields=scm: 403 Forbidden

In the golang-nuts group, Russ had previous posted that this was an issue and could be resolved by hand. With the advent of modules, manually checking out the correct version and putting it into the correct location becomes way more problematic.

[guybrand]

I had Similar issues in more than one laptop with Ubunto, MacOs and windows, and instead of blind troubleshooting (or reinstall git that sometimes helped mainly on windows), I resolved/worked around with these (temp/troubleshooting) changes/steps :

1. edit /usr/local/go/src/cmd/go/internal/get/vcs.go, I added (in bold)
   ...
   func bitbucketVCS(...
   for _, vcs := range []string{"git", "hg"} {
   fmt.Println("root:",root,"vcs",vcs)
2. still editing /usr/local/go/src/cmd/go/internal/get/vcs.go
   func (v *vcsCmd) runVerboseOnly(dir string, cmd string, keyval ...string) error {
   _, err := v.run1(dir, cmd, keyval, true) //was previously false
   save
3. terminal -> cd /usr/local/go/src/cmd/go
   sudo go build
4. (you may want to backup your previous bin/go...)
   sudo mv ./go /usr/local/go/bin/go
5. cd /go-projects-dir/:
   env GIT_TERMINAL_PROMPT=1 go get

that would:

1. should you the resolution for each repo (just a helper, in case you added "url" to git config etc) per vcs
2. would turn the silent bash script that pings bitbucket to interactive mode .
   Once you do that, you can see the actual issue which may vary from wrong ssh key, credential cash missing or invalid etc, and even allow you to interact with the command (e.g. if its a missing PW, you can type the password in)

In other words - this will not SOLVE your problem but has a good success rate of showing where the issue is.

tips:
If you are not actively working using mutiple bitbucket accounts make sure you have config --global git name, email, and url (not as you stated):
git config --global user.name "myBKuser"
git config --global user.email "[email protected]"
git config --global url."https://[email protected]/".insteadOf "https://bitbucket.org/"
in one case - the latter resolved the issue for me, adding a credential.helper or alternatively ssh key

and last ... return your original bin/go :)

[guybrand]

Reason and suggested resolution:
The above behavior is a result of missing global git config/credentials, and going through the code - there can be many other similar issues - unless ALL parts are in place.

When go get command tries to pull a bitbucket repo, it first tries to get the repo details as a public repo, resolving in the "403 Forbidden" reply.

Then it tries to check if a private repo is git or hg (as per issue #5375) , first using git, that will also resolve in 403 - if global git config does not include user.name , user.email.
Yet, this would not be enough without cached credentials on cred manager OR ssh key configured for your bitbucket user.
Last, to make sure the git command goes to the right private repo , git global config should either identify this private repo group/initiator whenever you ping bitbucket, or at least when you ping the specific subdomain, for this you need to either
git config --global url."https://[email protected]/".insteadOf "https://bitbucket.org/"
or
git config --global url."https://[email protected]/".insteadOf "https://bitbucket.org/MyCompany"

vcs.go main problem is not reporting right statuses that will aid the user to resolve the configuration by himself.
A fix would need to:

1. Replace the "403 Forbidden" reply that looks like an error (but is actually just a ping for a public repo) in : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo' - trying to connect to a private repo
2. check if git global config parameters includes user.name , user.email, if not, check hg connection, if that does not work - echo the user with a friendly error that says : "cant access public repo 'https://api.bitbucket.org/2.0/repositories/my-repo', if this is a private repo, please run:

git config --global user.name "your-bitbucket-user"
git config --global user.email "your-bitbucket-email"
git config --global url."https://[email protected]/".insteadOf "https://bitbucket.org/domain"

3. if no ssh key is defined/not defined well, the git command will try to prompt for a git password (go get will not support that, but rather through an error), when this fails, vcs.go is currently discarding the error and treats this as a "general error with pinging the repo", in this case, user error should display a friendly error "in order to allow access to a private repo either your git password should be cached in a credential manager, or an ssh key must be supplied" (perhaps with extra details on how to do that on according to the running OS) and the error from the request to bitbucket displayed alongside the friendly error, for cases like the case above (@savaki says he has set up an ssh key, probably something wrong there, and the ping to bitbucket probably returned some error there, my guess is the last git config I suggested using domain@ would have solved it, displaying a friendly error can save the time for guessing) .

Comments:
all the above "warnings/errors" should be stored during the process, and only display if hg fails as well.
github repo's may have similar issues, but lesser, since no need to determine if the repo is git or hg.

If no comments, I can probably set some of the above logic coded in place (vcs.go).

[bcmills]

A fix would need to: […]

For (1), we shouldn't print anything that looks like an error until we're sure it's actually an error. If we get all the way through the chain of attempts and it's still failing, then we should display a helpful message.

For (2), that seems too specific (to git and to BitBucket). Perhaps a URL (such as https://golang.org/doc/faq#git_https or a Wiki page) would be more helpful? That could collect configuration steps for various providers and tools without baking it into the go tool itself: that way, if we discover a better way to configure private repos we can publish it without updating the tool.

For (3), we should probably pass through the explicit error from the tool. (That's #25982.)

[guybrand]

1. Agreed then, I have a good concept to allow that.
2. Good idea, I would need access to edit the faq in order to do that, I was given go/wiki edit permission, but I dont see an option to edit the faq, any idea who's authorizing that ?
3. The git error in this case is quite generic (403 Forbidden) adding some local checks (e.g git.global exists) can give a better indication, however we can concatenate the errors to cover both the tool error and local so the user reading https://golang.org/doc/faq#git_https would go to the right section (e.g. "please read the git.global section on https://golang.org/doc/faq#git_https) - I do agree its specific (for git, not for bitbucket), but its not that we have too many vcs's, and git IS the most common.

[dsymonds]

This looks to be a duplicate of #16315.

[030]

https://stackoverflow.com/a/53518444/2777965

[mrjrieke]

Is it possible to have an option to use GOPATH by default and for anything it doesn't find, then vgo the rest? That way, private repo's and others like this one could be set up either manually prior to an automated like build.

[ernsheong]

Anyone with a workaround that actually works? (sorry for empty comment)

This has been rather painful and I think my next stop is to go back to GOPATH and restore vendoring

[guybrand]

Jonathan, we are constantly working with private bitbucket repos and modules, and the workarounds I described above works for multiple developer's on various os. If you are experiencing a similar, yet not exact issue, please use the "what I did/expect/actually got" Template to describe the issue, including what os are you using. I will try to visit our developer onboarding documents, to see if the description above is still accurate, and if it's not, post here.

…

On Wed, 25 Sep 2019, 6:26 Jonathan ES Lin, ***@***.***> wrote: Anyone with a workaround that *actually works*? (sorry for empty comment) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27344?email_source=notifications&email_token=ABNEY4VNPGCBJHYTK3GZKXTQLLK6XA5CNFSM4FSHNFYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QOYTA#issuecomment-534834252>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNEY4TYRWURSROX4EW3BJTQLLK6XANCNFSM4FSHNFYA> .

[ernsheong]

I downgraded to 1.12.9 and was able to do a go get on a private Bitbucket repo

Something is not working right with regard to this in 1.13

[guybrand]

In Order to be able to help you (and decide if its the same issue as 27344 or need a new related issue), can you complete in the format of this issue details
(i.e. What version of Go are you using (go version)? ... What operating system and processor architecture are you using (go env)? etc)

[langohr]

I have a similar issue (I think it is the same)
I have reported it here
#34528 (solved)

[harrisa1]

I downgraded to 1.12.9 and was able to do a go get on a private Bitbucket repo

Something is not working right with regard to this in 1.13

I can confirm that this issue happened for me too, I recently updated to 1.13.5 and it wasn't working, downgraded to 1.12.14 and it worked first try.

[bcmills]

@harrisa1, @ernsheong: see https://golang.org/wiki/NoPlusOne.

Since the original post here was reported against Go 1.11, a failure that occurs with 1.13.5 but not 1.12.14 is unlikely to be the same root cause. Please open a new issue with steps to reproduce the failure.

[bcmills]

Duplicate of #25982

[arulrajnet]

The answer for whoever coming here from google search.

To setup private bitbucket cloud

git config --global url."[email protected]:YOUR_COMPANY".insteadOf "https://bitbucket.org/YOUR_COMPANY"

Make sure you have setup SSH key access in bitbucket https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html . Your private key id_rsa file should be in ${HOME}/.ssh/ folder.

[ivansaid]

The answer for whoever coming here for google search.

To setup private bitbucket cloud

git config --global url."[email protected]:YOUR_COMPANY".insteadOf "https://bitbucket.org/YOUR_COMPANY"

Make sure you have setup SSH key access in bitbucket https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html . Your private key id_rsa file should be in ${HOME}/.ssh/ folder.

I did that same thing after that did this export GOPRIVATE="bitbucket.org/YOUR_COMPANY" closed the console window, restarted my IDE and it worked, i had tried everything already but i worked for me.

[n-insaidoo]

What also works is configuring the ~/.netrc file as such:

machine github.com login $GitHubUserName password $GitHubAccessToken
machine bitbucket.org login $BitBucketUserName password $BitBucketAppPassword

(In this case I also specify credentials to be use for GitHub private repos)

And populating the environment variable GOPRIVATE as appropriate.

[bansalvarun]

Expose the variable as given below may solve the given problem.
export GOPRIVATE="bitbucket.org/YOUR_COMPANY"

References:
https://community.atlassian.com/t5/Bitbucket-questions/How-do-I-run-go-get-on-a-private-repo/qaq-p/1385040
https://stackoverflow.com/questions/38668444/go-get-private-repo-from-bitbucket