x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095

joedian · 2023-06-30T19:55:38Z

This issue tracks porting the freebsd-amd64-130 (FreeBSD 13.0 with GOARCH=amd64) GCE host image to LUCI.

dmitshur · 2024-09-06T23:37:40Z

Based on https://www.freebsd.org/releases/, FreeBSD 13.0 is EOL, and its most recent releases are 13.3, 14.0, and 14.1.

CC @mengzhuo Is it right that https://ci.chromium.org/ui/p/golang/builders/luci.golang.ci/gotip-freebsd-riscv64 is testing the freebsd/riscv64 port with FreeBSD 13.1?

CC @golang/freebsd Would it make sense to target one of those newer versions, instead of this FreeBSD 13.0 one? Also see discussion about FreeBSD versions in #54355.

mengzhuo · 2024-09-10T14:40:04Z

Yes, it's FreeBSD 13.1 but it can't upgrade to 13.3 by pkg since riscv64 is not Tier 1 for FreeBSD.

But I can do a full upgrade to 14.1 if you want.

cagedmantis · 2024-09-19T15:18:27Z

@mengzhuo I think an upgrade to 14.0 or 14.1 would be helpful.

mknyszek · 2024-09-30T19:04:55Z

I'll make an attempt at this this week.

gopherbot · 2024-09-30T19:08:41Z

Change https://go.dev/cl/616836 mentions this issue: env/freebsd-amd64: port to LUCI and upgrade to 14.1

Port to LUCI includes: - Install Python. - Install bootstrapswarm. - Use bootstrapswarm instead of buildlet. For golang/go#61095. Change-Id: Ie787f6d098186259398ad135b60758e1ce8b489c Reviewed-on: https://go-review.googlesource.com/c/build/+/616836 Auto-Submit: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

mknyszek · 2024-10-03T16:36:16Z

As an update here, I got the machines up and running and they pass most of the tests.

Unfortunately, we're seeing persistent failures of the form:

--- FAIL: TestLookupNoData (36.75s)
    lookup_test.go:1439: forced cgo resolver: IsNotFound is set to false
    lookup_test.go:1444: forced cgo resolver: error message is not equal to: no such host
    lookup_test.go:1455: forced cgo resolver: backoff 1s after failure lookup golang.rsc.io.: Address family for hostname not supported
    lookup_test.go:1439: forced cgo resolver: IsNotFound is set to false
    lookup_test.go:1444: forced cgo resolver: error message is not equal to: no such host
    lookup_test.go:1455: forced cgo resolver: backoff 5s after failure lookup golang.rsc.io.: Address family for hostname not supported
    lookup_test.go:1439: forced cgo resolver: IsNotFound is set to false
    lookup_test.go:1444: forced cgo resolver: error message is not equal to: no such host
    lookup_test.go:1455: forced cgo resolver: backoff 30s after failure lookup golang.rsc.io.: Address family for hostname not supported
    lookup_test.go:1439: forced cgo resolver: IsNotFound is set to false
    lookup_test.go:1444: forced cgo resolver: error message is not equal to: no such host
    lookup_test.go:1461: forced cgo resolver: unexpected error: lookup golang.rsc.io.: Address family for hostname not supported
--- FAIL: TestLookupNoSuchHost (36.48s)
--- FAIL: TestLookupNoSuchHost/LookupHost_NODATA (36.15s)
--- FAIL: TestLookupNoSuchHost/LookupHost_NODATA/forced_cgo_resolver (36.11s)
    lookup_test.go:1610: IsNotFound is set to false
    lookup_test.go:1614: error message is not equal to: no such host
    lookup_test.go:1623: backoff 1s after failure lookup _ldap._tcp.google.com.: Address family for hostname not supported
    lookup_test.go:1610: IsNotFound is set to false
    lookup_test.go:1614: error message is not equal to: no such host
    lookup_test.go:1623: backoff 5s after failure lookup _ldap._tcp.google.com.: Address family for hostname not supported
    lookup_test.go:1610: IsNotFound is set to false
    lookup_test.go:1614: error message is not equal to: no such host
    lookup_test.go:1623: backoff 30s after failure lookup _ldap._tcp.google.com.: Address family for hostname not supported
    lookup_test.go:1610: IsNotFound is set to false
    lookup_test.go:1614: error message is not equal to: no such host
    lookup_test.go:1628: unexpected error: lookup _ldap._tcp.google.com.: Address family for hostname not supported

I'm not really sure where to start with this. @neild maybe you have some ideas?

CC @golang/freebsd

mknyszek · 2024-10-03T17:48:52Z

Also, freebsd/amd64 gomotes should now be available, where this is reproducible.

mknyszek · 2024-10-04T15:17:15Z

Clearing my assignment as this is my last day on the rotation this week.

AFAICT, the only steps left are:

Fix the test failure in x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095 (comment).
Fix the build failure in x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095 (comment).
Remove the "known issue" flag on these builders.

mknyszek · 2024-10-04T15:30:55Z

Okay actually, I think this error is related to the image upgrade, not LUCI.

I found this bug, which is tagged as related to FreeBSD 13.2. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273912 It appears that this is indeed new behavior and it's intentional? It looks like at least the first test failure is indeed trying to look up a host without an A record (https://cs.opensource.google/go/go/+/master:src/net/lookup_test.go;l=1426?q=TestLookupNoData&ss=go%2Fgo).

I think this actually fully explains the issue, but I have no idea whether the fix is to go into the net package(s) and detect this new returned error, or to just skip the test altogether.

mknyszek · 2024-10-04T15:35:12Z

I think I actually see the fix; I think we need to look for EAI_ADDRFAMILY here: https://cs.opensource.google/go/go/+/master:src/net/cgo_unix.go;l=196;drc=71d84ee7b4e42ea9f35f409f169bebd44a360331.

gopherbot · 2024-10-04T15:51:07Z

Change https://go.dev/cl/618015 mentions this issue: net: detect EAI_ADDRFAMILY for cgo host lookup on FreeBSD

mknyszek · 2024-10-04T16:05:31Z

Wow! That actually worked. Cool, all we need then is to fix cmd/cgo/internal/test:internal:

cannot parse row "# cmd/cgo/internal/test.test", invalid character '#' looking for beginning of value
cannot parse row "cmd/cgo/internal/test(.text): relocation target __mulsc3 not defined", invalid character 'c' looking for beginning of value
cannot parse row "cmd/cgo/internal/test(.text): relocation target __muldc3 not defined", invalid character 'c' looking for beginning of value
cannot parse row "cmd/cgo/internal/test(.text): relocation target __absvsi2 not defined", invalid character 'c' looking for beginning of value
cannot parse row "cmd/cgo/internal/test/issue9510a(.text): relocation target __muldc3 not defined", invalid character 'c' looking for beginning of value
cannot parse row "cmd/cgo/internal/test/issue9510b(.text): relocation target __muldc3 not defined", invalid character 'c' looking for beginning of value
FAIL	cmd/cgo/internal/test [build failed]

cherrymui · 2024-10-07T14:53:59Z

Does FreeBSD use clang by default? I think __muldc3 etc. should be provided by compiler-rt. Is compiler-rt not installed, or not in the search path of the C linker?

prattmic · 2024-10-21T20:05:59Z

In addition to this cgo issue, I also see occasional infra failures due to bots dying. e.g., https://ci.chromium.org/ui/p/golang/builders/ci/gotip-freebsd-amd64/b8733453678465886913/infra. It's not clear to me why the bot is dying.

prattmic · 2024-10-21T20:53:48Z

The infra failures seem to be due to the LUCI infra deleting the instance while it is running tests. It should not be doing that.

mknyszek · 2024-12-02T18:28:32Z

Welp, I'm back on rotation again. We think we know why the bots are being deleted, so I'll update the image with a fix for that. I'll see if maybe something else needs updating in the image generation script for clang. Thanks for the deep dive @prattmic!

mknyszek · 2024-12-02T19:02:20Z

So, AFAICT, there's nothing special with clang in this FreeBSD image. Clang is installed by default, so there's no extra installation step, and there are no packages like clang-dev (or something) that I can find. I suspect we need a workaround instead.

gopherbot · 2024-12-02T19:17:34Z

Change https://go.dev/cl/632975 mentions this issue: cmd/link/internal/ld: work around FreeBSD 14.1 cmd/cgo test link failure

mknyszek · 2024-12-02T19:32:39Z

Unfortunately, naively applying OpenBSD's workaround is not enough.

gopherbot · 2024-12-02T20:22:02Z

Change https://go.dev/cl/622040 mentions this issue: env/freebsd-amd64: exit daemon when bootstrapswarm has a zero exit code

Part of the contract of running swarming_bot is that we need to keep restarting it if it has a non-zero exit code. But if it has a zero exit code, we must NOT restart it. Currently the daemon just always keeps restarting it. Do this by writing the pid of the daemon supervisor and using that to kill the supervisor when we get a zero exit code from bootstrapswarm. bootstrapswarm will exit with a zero exit code only if swarming_bot does. For golang/go#61095. Change-Id: I92bdea3fa35fed0883cd9e996aa231e26cdb7155 Reviewed-on: https://go-review.googlesource.com/c/build/+/622040 Auto-Submit: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

mknyszek · 2024-12-03T17:57:01Z

OK so I found a fix, but I'm not certain if it's the right one.

clang --print-libgcc-file-name prints /usr/lib/clang/18/lib/freebsd/libclang_rt.builtins-x86_64.a on FreeBSD 14.1. (As you can see, the clang version is 18.) This is missing symbols. We can ask for compiler_rt.a explicitly, and this fixes the problem. I am not sure if this is a robust fix for older versions of FreeBSD, but it does fix the broken test.

mknyszek · 2024-12-03T17:59:29Z

OK it appears that libclang_rt.builtins-x86_64.a is an explicit subset of libcompiler_rt.a, if I'm understanding correctly. It sounds like we should be looking for libcompiler_rt.a explicitly? I can try the SlowBots to just make sure this doesn't break old FreeBSD.

mknyszek · 2024-12-03T18:14:45Z

Yay, the CL above now works, and doesn't break on old versions of FreeBSD either. If we land that, I think we're done here. We can probably also enable FreeBSD in presubmit, the builder is pretty fast.

On FreeBSD 14.1 we fail to link against C code with internal linking. The symptom is apparently undefined symbols, but explicitly pointing the linker at compiler-rt for -libgcc fixes the issue. This looks a lot like the workaround on OpenBSD, but the symptom is different. --print-libgcc-file-name produces libclang_rt.builtins-x86_64.a which appears to be an insufficient subset of libcompiler_rt.a. For #61095. Change-Id: Iff5affbc923d69c89d671a69d8f4ecaadac42177 Cq-Include-Trybots: luci.golang.try:gotip-freebsd-amd64 Reviewed-on: https://go-review.googlesource.com/c/go/+/632975 Auto-Submit: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Run-TryBot: Michael Knyszek <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Cherry Mui <[email protected]>

gopherbot · 2024-12-03T20:08:03Z

Change https://go.dev/cl/633198 mentions this issue: main.star: remove known issue from freebsd-amd64

gopherbot · 2024-12-05T21:59:38Z

Change https://go.dev/cl/634042 mentions this issue: [release-branch.go1.23] net: detect EAI_ADDRFAMILY for cgo host lookup on FreeBSD

gopherbot · 2024-12-05T22:04:46Z

Change https://go.dev/cl/634043 mentions this issue: [release-branch.go1.22] net: detect EAI_ADDRFAMILY for cgo host lookup on FreeBSD

gopherbot · 2024-12-06T15:15:45Z

Change https://go.dev/cl/634079 mentions this issue: main.star: disable FreeBSD 14.1 on old release branches

We're not planning to backport the fixes needed to make it work. For golang/go#61095. Change-Id: I6dfb039d0bec335e26332aef7ace17691609011d Reviewed-on: https://go-review.googlesource.com/c/build/+/634079 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>

All remaining issues appear to be fixed. Fixes golang/go#61095. Change-Id: I3c2a6446cd5308df4d5d257bb7d0a3de5d0fb1ca Reviewed-on: https://go-review.googlesource.com/c/build/+/633198 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Auto-Submit: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>

gopherbot · 2024-12-06T15:26:35Z

Closed by merging CL 633198 (commit golang/build@40adee2) to luci-config.

gopherbot · 2025-02-04T15:15:06Z

Change https://go.dev/cl/646515 mentions this issue: internal/migration: stop running freebsd-amd64-{12_3,13_0,race} builders

gopherbot · 2025-02-04T15:15:50Z

Change https://go.dev/cl/646136 mentions this issue: main.star: add freebsd-amd64-race

These builders have been migrated to LUCI. For golang/go#61095. For golang/go#67761. Change-Id: I677b1a4fabe300c048c38f34f3712c118922eb86 Reviewed-on: https://go-review.googlesource.com/c/build/+/646515 Reviewed-by: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]>

Move the remaining freebsd-amd64 builder configuration from old infra, and then stop it there (CL 646515). For golang/go#61095. For golang/go#71556. Change-Id: I5b12e139458f7ee9ff97061d23d27fe47ee9937e Reviewed-on: https://go-review.googlesource.com/c/build/+/646136 Reviewed-by: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]>

joedian mentioned this issue Jun 30, 2023

x/build: create LUCI GCE host images #60468

Open

35 tasks

cagedmantis added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder labels Jun 30, 2023

cagedmantis added this to the Backlog milestone Jun 30, 2023

joedian changed the title ~~host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230)~~ x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) Jul 5, 2023

bcmills added the OS-FreeBSD label Jul 24, 2023

cherrymui self-assigned this Dec 14, 2023

dmitshur modified the milestones: Backlog, Unreleased Feb 13, 2024

cherrymui removed their assignment Apr 4, 2024

dmitshur mentioned this issue Jun 24, 2024

crypto/tls: frequent "connection reset by peer" failures in TestVerifyCertificates and other tests on freebsd/amd64 #68155

Closed

mknyszek self-assigned this Sep 30, 2024

dmitshur added this to Go Release Oct 2, 2024

dmitshur moved this to In Progress in Go Release Oct 2, 2024

mknyszek removed their assignment Oct 4, 2024

mknyszek self-assigned this Dec 2, 2024

dmitshur mentioned this issue Dec 6, 2024

x/build: add doas package on NetBSD systems #70708

Open

gopherbot closed this as completed Dec 6, 2024

github-project-automation bot moved this from Planned to Done in Go Release Dec 6, 2024

dmitshur mentioned this issue Jan 4, 2025

x/build: update OpenBSD builders to 7.6 #69823

Closed

seankhliao mentioned this issue Feb 11, 2025

proposal: embed: embed and expose also media/mime type #66121

Closed

dmitshur mentioned this issue Mar 19, 2025

x/build/env: host-freebsd-amd64-12_3 (VM Image: freebsd-amd64-123-stable-20211230) #61094

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095

x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095

joedian commented Jun 30, 2023 •

edited by dmitshur

Loading

dmitshur commented Sep 6, 2024 •

edited

Loading

mengzhuo commented Sep 10, 2024

cagedmantis commented Sep 19, 2024

mknyszek commented Sep 30, 2024

gopherbot commented Sep 30, 2024

mknyszek commented Oct 3, 2024

mknyszek commented Oct 3, 2024

mknyszek commented Oct 4, 2024 •

edited

Loading

mknyszek commented Oct 4, 2024

mknyszek commented Oct 4, 2024

gopherbot commented Oct 4, 2024

mknyszek commented Oct 4, 2024

cherrymui commented Oct 7, 2024

prattmic commented Oct 21, 2024

prattmic commented Oct 21, 2024

mknyszek commented Dec 2, 2024

mknyszek commented Dec 2, 2024

gopherbot commented Dec 2, 2024

mknyszek commented Dec 2, 2024

gopherbot commented Dec 2, 2024

mknyszek commented Dec 3, 2024

mknyszek commented Dec 3, 2024

mknyszek commented Dec 3, 2024

gopherbot commented Dec 3, 2024

gopherbot commented Dec 5, 2024

gopherbot commented Dec 5, 2024

gopherbot commented Dec 6, 2024

gopherbot commented Dec 6, 2024

gopherbot commented Feb 4, 2025

gopherbot commented Feb 4, 2025

x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095

x/build/env: host-freebsd-amd64-13_0 (VM Image: freebsd-amd64-130-stable-20211230) #61095

Comments

joedian commented Jun 30, 2023 • edited by dmitshur Loading

dmitshur commented Sep 6, 2024 • edited Loading

mengzhuo commented Sep 10, 2024

cagedmantis commented Sep 19, 2024

mknyszek commented Sep 30, 2024

gopherbot commented Sep 30, 2024

mknyszek commented Oct 3, 2024

mknyszek commented Oct 3, 2024

mknyszek commented Oct 4, 2024 • edited Loading

mknyszek commented Oct 4, 2024

mknyszek commented Oct 4, 2024

gopherbot commented Oct 4, 2024

mknyszek commented Oct 4, 2024

cherrymui commented Oct 7, 2024

prattmic commented Oct 21, 2024

prattmic commented Oct 21, 2024

mknyszek commented Dec 2, 2024

mknyszek commented Dec 2, 2024

gopherbot commented Dec 2, 2024

mknyszek commented Dec 2, 2024

gopherbot commented Dec 2, 2024

mknyszek commented Dec 3, 2024

mknyszek commented Dec 3, 2024

mknyszek commented Dec 3, 2024

gopherbot commented Dec 3, 2024

gopherbot commented Dec 5, 2024

gopherbot commented Dec 5, 2024

gopherbot commented Dec 6, 2024

gopherbot commented Dec 6, 2024

gopherbot commented Feb 4, 2025

gopherbot commented Feb 4, 2025

joedian commented Jun 30, 2023 •

edited by dmitshur

Loading

dmitshur commented Sep 6, 2024 •

edited

Loading

mknyszek commented Oct 4, 2024 •

edited

Loading