Closed
Description
Please answer these questions before submitting your issue. Thanks!
What did you do?
$ cat main.go
package main
import (
"crypto/x509"
"fmt"
"log"
)
func main() {
certs, err := x509.SystemCertPool()
if err != nil {
log.Fatal(err)
}
fmt.Printf("Num System Certs: %d\n", len(certs.Subjects()))
}
$ CGO_ENABLED=0 go run main.go
Num System Certs: 188
$ CGO_ENABLED=1 go run main.go
Num System Certs: 168
What did you expect to see?
I expected to see the same number of certificates regardless of whether I used cgo.
What did you see instead?
The implementation using CGO resulted in fewer system certificates, which causes problems for our tooling that relies on one of those missing certificates to be in the SystemCertPool
.
System details
go version go1.10.1 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jhenke/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jhenke"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_b/gz_w_nfj0_33f5y3s_0pg8xs080pym/T/go-build925272903=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.10.1 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10.1
uname -v: Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G1114
lldb --version: lldb-900.0.64
Swift-4.0
Metadata
Metadata
Assignees
Type
Projects
Relationships
Development
No branches or pull requests
Activity
adamdecaf commentedon Apr 3, 2018
Could you tell us more about the certificates not found in the cgo path? Are they set with specific trust policies? Could you paste a certificate that isn't found, but should be?
nmiyake commentedon Apr 3, 2018
Do you know of a way to print/examine the trust policies of a certificate? After some experimentation, we have found that the certificates that aren't showing up seem to say "This certificate has custom trust settings" in the Keychain UI. However, expanding the "Trust" section doesn't reveal any specifics:
I'm not sure how the certificate was added/got to this state. It seems that if we manually update the state to "Always Trust" in the Keychain UI, then the certificate is returned. However, we'd like to understand this further, since most apps seem to trust the certificate even with these "Custom" trust settings but Go with CGo does not, which is causing issues for us.
*To clarify, I suspect that the issue has to do with the trust settings marked for the certificate rather than with the certificate itself
bcmills commentedon Apr 3, 2018
(CC: @FiloSottile)
adamdecaf commentedon Apr 3, 2018
@jdhenke @nmiyake I've had to add the certificates into either the login keychain or
/Library/Keychains/System.keychain
. Doing this with thesecurity
tool looks like:How are you adding the certs?
The Keychain UI is pretty much the easiest way to view trust, but it's not always clear. I've been working on better ways to parse the
security
tool output.You can inspect the certs with the tool by running something like, which is what non-cgo Go does:
I've talked about the inverse of this problem (explicit distrust of certificates) at #24084, which has the same confusion/problem.
nmiyake commentedon Apr 3, 2018
Unfortunately, I'm not sure about the provenance of how the cert was added. However, I do suspect that how it was added/how it was upgraded from a previous add is at the root of the issue here.
I have a certificate in my System store called "MSCA-ROOT-01-CA". I'm not sure how it was added, but the screenshot earlier in the issue shows that it displays as "This certificate has custom trust settings", although the UI shows "Always Trust" for all the parts.
I added some debugging code to
root_cgo_darwin.go
(edits at the end of this post), and the resulting output for this cert is:Based my reading of the code, this certificate is in the "Admin" domain and has a non-NULL but empty trust setting. Because the trust setting is empty and it isn't a system cert, it decides not to trust it.
This logically makes sense to me, but I guess the resulting behavior isn't consistent with other applications (other applications seem willing to use this certificate for verification).
Modified
FetchPEMRoots
incrypto/x509/root_cgo_darwin.go
:adamdecaf commentedon Apr 3, 2018
@nmiyake Could you rebase that change off the latest commit on master? There's another change which mixes up the diff a bit.
I'm not sure if trusting a certificate without any policies from the user/admin domain would cause problems. If an attacker is modifying your trust policies they can already install a root CA.
There is a
kSecTrustSettingsResultDeny
policy I'd expect to be set (to explicitly distrust), but I don't know how widespread that's used. Keychain does use that.nmiyake commentedon Apr 3, 2018
Sure. Here's the modified code on master:
Output was the same:
From what I can tell, I have 3 certificates in my keychain that fit this criteria.
Adding this logic fixes the specific issue that we're seeing:
(if this were to be done, I would presume it should probably be done for the case where
trustSettings == NULL
as well for consistency)Interestingly, this doesn't fully resolve the diff for the number of certificates between CGO_ENABLED=0 and 1. My breakdown is:
master
, CGO_ENABLED=1: 172 certificatestrustSettings ==0
, CGO_ENABLED=1: 175 certificatesmaster
, CGO_ENABLED=0: 196 certificatesadamdecaf commentedon Apr 4, 2018
@nmiyake Cool. If you want to submit that
if (CFArrayGetCount(trustSettings) == 0) {
change we can get it reviewed.There's probably something up with the trust policies on those remaining certificates. Can you run the following?
This dumps plist (xml) files of your certificate trust. The best way to find a specific cert is by the sha1 hash. It's the
<key>...</key>
in the following snippet.Can you find a certificate that's added into a keychain, but isn't showing up in Go? I'm curious what
<key>trustSettings</key>
is saying.The values there are mapped to
SecTrustSettingsResult
.I'm building a quick tool to help debug these files. You can run it over the exported plist files and get something that's a bit easier to parse. https://github.com/adamdecaf/plist-parser
nmiyake commentedon Apr 4, 2018
OK, performed more digging and diagnostics.
Here's the overview of my state with the current Go:
After running with my modification proposed above, I get:
I ran your parsing tool on my output, and indeed for some reason there are 3 certificates that explicitly have an empty trust settings set (only showing one here):
I'm not sure how this entry was created for me, but that's clearly the issue. Two of these certificates are valid and are added, and thus increment the "common" count by 2. One of these certificates is expired. This one appears only in the CGo code, which accounts for the "1 CGo only".
Here are some of the certificates that show up only for non-CGo (out of the 13):
2 of these have an entry in my
user-trust.plist
withtrustSettings=map[string]string{"kSecTrustSettingsResult":"-2147409654"}
:None of the other entries show up in my plist. In the keychain, these show up as "no value specified" for trust:
The one curious thing is the extra certificate that shows up for CGo only after the local modification. That certificate is an old CA certificate I have that is expired:
It's showing up because before the modification it was in my
admin-trust.plist
withmap[string]string{}
as the trustSettings, so the change now includes it. I'm guessing the non-CGo code automatically prunes it based on expiration.I don't think this should be an issue since even if the cert is added as a root cert, any code that does validation should properly check the expiration status.
gopherbot commentedon Apr 4, 2018
Change https://golang.org/cl/104735 mentions this issue:
crypto/x509: add certs with empty trust settings for cgo_darwin
adamdecaf commentedon Apr 4, 2018
@nmiyake Cool on that CL. You'll probably want to assign @FiloSottile as a reviewer.
As far as the certificates with
"kSecTrustSettingsResult":"-2147409654"
. I think this is why they're not marked as trusted.-2147409654
is not a valid value. The relevant Go code only checks a couple of the values (seeSecTrustSettingsResult
).Are your trust settings managed by an enterprise or tool by chance? It looks like some tool generated partially invalid policy settings.
I've had to work around this and seen it before. Here's an example plist I've seen in the wild. (Note: it's the same
-2147409654
value)On those 13-2 certificates I wonder again if the plist/trust policies were generated properly. Just search for
DigiCert SHA2 High Assurance Server CA
gives a different cert fingerprint than what your policy has. See: https://crt.sh/?id=2900424 (SHA1 prefixA031C467
) https://crt.sh/?id=2900424.Can you find any of those certificates on crt.sh? https://crt.sh/?a=1
nmiyake commentedon Apr 4, 2018
Yes, this behavior is on a machine that's managed by a company and uses tools to do so -- something along the way there writing an invalid policy entry is definitely a possibility.
I guess the difference in observed behavior is that most macOS applications (or Apple's cert API itself?) are more lenient on their verification here? Even though strict validation may technically be correct, if it results in an observable difference in behavior between native macOS apps and Go apps with CGo enabled that stills seems like it could be an issue (and I don't really have any good way of knowing how common or uncommon this scenario may be more broadly).
adamdecaf commentedon Apr 4, 2018
I don't think we've determined that quite yet. Are any of those 13-2 certs not CA's?
I'm reading through the code paths and noticing only the cgo path checks the certificate is a Root CA (by checking Issuer == Subject).
Edit: Yep. I verified a non-ca certificate would show up in a non-cgo call to
x509.SystemCertPool()
.85 remaining items
diligiant commentedon Jan 9, 2019
@FiloSottile failed
dmitshur commentedon Jan 9, 2019
Latest tip (commit 99ea99e) passed on my personal Mac with macOS Mojave 10.14.2 (18C54).
zdjones commentedon Jan 20, 2019
Failed on macOS High Sierra 10.13.6 (17G4015)
vdemario commentedon Jan 21, 2019
Tests are passing for me now! Yay 🎉
Sorry for the delay.
We're skipping TLS verification! Bug in Mac OS for Golang, golang/go#…
gopherbot commentedon Feb 15, 2019
Change https://golang.org/cl/162860 mentions this issue:
[release-branch.go1.11] crypto/x509: fix root CA extraction on macOS (cgo path)
gopherbot commentedon Feb 15, 2019
Change https://golang.org/cl/162861 mentions this issue:
[release-branch.go1.11] crypto/x509: fix root CA extraction on macOS (no-cgo path)
Lax77 commentedon Feb 15, 2019
I am running go 1.11.5 version on Mac 10.13.6 version, I keep getting the
x509: certificate signed by unknown authority
when I try to get dependencies. eg: k8s.io/api. My gotip test result is a pass for me though. All I was doing is trying to fetch dependencies with operator-sdk. Any suggestions on how I can go about resolving it ?$go get k8s.io/api
$dep ensure
FiloSottile commentedon Feb 18, 2019
@Lax77 Please run the test from #24652 (comment). If it passes, just wait for 1.11.6 or 1.12. If not, please open a new issue with the output and tag me.
(Locking this issue because we shipped a fix, it's getting hard to follow and we shouldn't keep pinging everyone. If you have a similar issue or if you run the tests and they fail, please open a new issue referencing this one and tagging me.)
gopherbot commentedon Apr 14, 2020
Change https://golang.org/cl/227037 mentions this issue:
crypto/x509: use Security.framework without cgo for roots on macOS
agar.io