Skip to content

cmd/compile: sporadic memory corruption on 386 (32-bit) builders #37881

New issue
New issue
Closed
Closed
cmd/compile: sporadic memory corruption on 386 (32-bit) builders#37881
Labels
FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.release-blocker
Milestone
 
Go1.15
@bcmills

Description

@bcmills
bcmills
opened on Mar 16, 2020 · edited by bcmills

Various sweep increased allocation count errors have started cropping up in or near go/build invocations in various go commands.

2020-03-15T08:13:55-32dbccd/linux-386-clang
2020-03-14T07:03:15-d774d97/linux-386-sid
2020-03-16T20:59:27-ff1eb42/linux-386-387
2020-03-15T08:13:55-32dbccd/linux-386-clang
2020-03-14T07:03:15-d774d97/linux-386-sid
2020-03-13T20:43:12-e2a9ea0/openbsd-386-62

Since go/build is involved, this may be related to the go/types crash cluster (#37602, #37507, #37690).

CC @griesemer @matloob @aclements @danscales @mknyszek @cherrymui

Activity

bcmills
added
NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
on Mar 16, 2020
bcmills
added this to the Go1.15 milestone on Mar 16, 2020
bcmills

bcmills commented on Mar 17, 2020

@bcmills
bcmills
on Mar 17, 2020
ContributorAuthor

These found bad pointer in Go heap errors seem to fit the same root cause:

2020-03-16T22:31:39-2fbca94/linux-386-sid
2020-03-13T19:43:47-cbcb031/openbsd-386-62

bcmills
changed the title [-]runtime: "sweep increased allocation count" during cmd/go package loading on 32-bit builders[/-] [+]runtime: memory corruption in cmd/go on 32-bit builders[/+] on Mar 17, 2020
bcmills
added
release-blocker
on Mar 17, 2020
bcmills

bcmills commented on Mar 17, 2020

@bcmills
bcmills
on Mar 17, 2020
ContributorAuthor

This appears to be a dramatic uptick in memory corruption across the board on the 32-bit builders.

Given that it is only appearing on the 32-bit builders, and that we have not (to my knowledge) made any particularly racy or dangerous changes in cmd/go this cycle, marking as release-blocker for Go 1.15.

bcmills

bcmills commented on Mar 17, 2020

@bcmills
bcmills
on Mar 17, 2020
ContributorAuthor

I also can't rule out a compiler bug as the cause, given the number of changes to the rewrite rules so far. (CC @josharian @randall77)

bcmills

bcmills commented on Mar 17, 2020

@bcmills
bcmills
on Mar 17, 2020
ContributorAuthor

The first failure in this cluster was the same day that CL 222782 (#36468) was merged, so I suspect that may be related.

bcmills
changed the title [-]runtime: memory corruption in cmd/go on 32-bit builders[/-] [+]cmd/compile: sporadic memory corruption on 32-bit builders[/+] on Mar 17, 2020
bcmills

bcmills commented on Mar 17, 2020

@bcmills
bcmills
on Mar 17, 2020
ContributorAuthor

Note that some of the affected builders are also TryBots:
https://go-review.googlesource.com/c/go/+/223745/5#message-7c0b4f426d49ae03a1ec88007cf714c5134d4800

randall77
mentioned this on Mar 18, 2020
ianlancetaylor

ianlancetaylor commented on Mar 19, 2020

@ianlancetaylor
ianlancetaylor
on Mar 19, 2020
Contributor

https://storage.googleapis.com/go-build-log/a19069d2/linux-386_272d029f.log

jayconrod

jayconrod commented on Mar 19, 2020

@jayconrod
jayconrod
on Mar 19, 2020
Contributor

https://storage.googleapis.com/go-build-log/29e1c579/linux-386_8cf5204b.log

randall77

randall77 commented on Mar 20, 2020

@randall77
randall77
on Mar 20, 2020
Contributor

I'm kinda stumped on this one. I've tried to reproduce to no avail, and pouring over CL 222782 doesn't reveal anything that might cause intermittent issues like this.

On a possibly related note, I just mailed a change to add more addressing mode modifications for amd64. When that goes in, whether or not we start seeing this on amd64 will be illuminating.

bcmills
changed the title [-]cmd/compile: sporadic memory corruption on 32-bit builders[/-] [+]cmd/compile: sporadic memory corruption on 386 (32-bit) builders[/+] on Mar 20, 2020

14 remaining items

gopherbot
added 3 commits that reference this issue on Mar 24, 2020

Revert "cmd/compile: disable addressingmodes pass for 386"

f975485

Revert "cmd/compile: disable mem+op operations on 386"

e0deacd

Revert "cmd/compile: convert 386 port to use addressing modes pass"

c785633
gopherbot

gopherbot commented on Mar 27, 2020

@gopherbot
gopherbot
on Mar 27, 2020
Contributor

Change https://golang.org/cl/225798 mentions this issue: cmd/compile: convert 386 port to use addressing modes pass (take 2)

randall77

randall77 commented on Mar 27, 2020

@randall77
randall77
on Mar 27, 2020
Contributor

I might have figured out what is wrong. Read the description of the CL above for all the gory details.
I'm not 100% sure, but enough so to try and submit this again and see what happens (I still can't reproduce outside trybots).

gopherbot
added a commit that references this issue on Mar 27, 2020

cmd/compile: convert 386 port to use addressing modes pass (take 2)

af7eafd
bcmills

bcmills commented on Mar 27, 2020

@bcmills
bcmills
on Mar 27, 2020
ContributorAuthor

Unfortunately, https://build.golang.org/log/b634dbda2b877a9859e65cd5eba87577e9e33dee looks like it may be another instance of this failure after the most recent attempt.

freebsd-386-12_0 at 9ceb1e5f5caca5666f9db50864c45ca1f88da1df
…

##### ../misc/reboot
Building Go cmd/dist using /tmp/workdir/go. (devel 9ceb1e5f5caca5666f9db50864c45ca1f88da1df freebsd/386)
Building Go toolchain1 using /tmp/workdir/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for freebsd/386.
# crypto/tls
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x862b4c0]

goroutine 38 [running]:
cmd/compile/internal/gc.buildssa(0x0, 0x2, 0x0)
	/tmp/workdir/tmp/reboot-goroot082852438/src/cmd/compile/internal/gc/ssa.go:297 +0xb0
cmd/compile/internal/gc.compileSSA(0x0, 0x2)
	/tmp/workdir/tmp/reboot-goroot082852438/src/cmd/compile/internal/gc/pgen.go:296 +0x4c
cmd/compile/internal/gc.compileFunctions.func2(0x3a59ccc0, 0x39016c70, 0x2)
	/tmp/workdir/tmp/reboot-goroot082852438/src/cmd/compile/internal/gc/pgen.go:361 +0x35
created by cmd/compile/internal/gc.compileFunctions
	/tmp/workdir/tmp/reboot-goroot082852438/src/cmd/compile/internal/gc/pgen.go:359 +0xf7
go tool dist: FAILED: /tmp/workdir/tmp/reboot-goroot082852438/pkg/tool/freebsd_386/go_bootstrap install -gcflags=all= -ldflags=all= std cmd: exit status 2
--- FAIL: TestRepeatBootstrap (98.43s)
    reboot_test.go:50: exit status 2
cherrymui

cherrymui commented on Mar 27, 2020

@cherrymui
cherrymui
on Mar 27, 2020
Member

This looks somewhat different. The original failure looks like the GC found (temporary) bad pointers. The new one is a segfault.

gopherbot

gopherbot commented on Mar 30, 2020

@gopherbot
gopherbot
on Mar 30, 2020
Contributor

Change https://golang.org/cl/226437 mentions this issue: cmd/compile: fix ephemeral pointer problem on amd64

gopherbot
added a commit that references this issue on Mar 30, 2020

cmd/compile: fix ephemeral pointer problem on amd64

33b648c
randall77

randall77 commented on Apr 2, 2020

@randall77
randall77
on Apr 2, 2020
Contributor

I'm going to declare this fixed. The original CL with fixed rules is in, both for 386 and amd64.
I'm not seeing any similar failures on the dashboard since the fixes went in.

randall77
closed this as completedon Apr 2, 2020
mknyszek
mentioned this on May 26, 2020
golang
locked and limited conversation to collaborators on Apr 2, 2021
gopherbot
added
FrozenDueToAge
on Apr 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.release-blocker

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @jayconrod@rasky@mknyszek@ianlancetaylor@bcmills

        Issue actions
          cmd/compile: sporadic memory corruption on 386 (32-bit) builders · Issue #37881 · golang/go