Closed
Description
greplogs --dashboard -md -l -e FAIL\\s+heapsampling\\.go --since=2021-01-01
2021-11-12T18:57:22-b1b6d92/windows-386-2008
2021-11-12T16:58:34-95d0657/windows-386-2008
2021-11-05T20:59:43-dbd3cf8/windows-386-2008
2021-11-05T18:20:07-53bab19/windows-386-2008
2021-11-05T17:46:59-e48e4b4/windows-386-2008
2021-11-05T15:52:40-a0d661a/windows-386-2008
(CC @bufflig)
Metadata
Metadata
Assignees
Labels
Type
Projects
Relationships
Development
No branches or pull requests
Participants
Activity
aclements commentedon Dec 1, 2021
We haven't seen any more of these failures.
Our working hypothesis is that these were related to the new pacer. That landed 2021-11-04. In all of these failures, the heap is around 1.7 GB when it fails, which is way bigger than we'd expect from this test. Based on printing MemStats.Sys at the end of this test, the largest memory footprint I've seen on my linux/amd64 laptop is 120 MB.
mknyszek commentedon Dec 1, 2021
I want to amend @aclements' a little bit just to say that we've seen these large memory footprints (I think around 113 MiB?) with the old pacer too (
GOEXPERIMENT=nopacerredesign). It's still possible this 1.7 GiB uptick was somehow related to the new pacer, but it's less likely to be solely related to that, I think.What's strange is that the failures showed up twice, then for a period of about 8 days, then disappeared.
toothrot commentedon Dec 8, 2021
Checking on this as a release blocker. Is this still occurring @bufflig?
bcmills commentedon Dec 8, 2021
Still no more failures in the logs. Maybe this was a bad interaction with some other
runtimetest that has since been fixed?greplogs --dashboard -md -l -e FAIL\\s+heapsampling\\.go --since=2021-11-13cherrymui commentedon Dec 15, 2021
Seems still no new failure since 2021-11-12.
aclements commentedon Dec 15, 2021
It's unfortunate that we don't know what caused this, but at this point I think we can close this issue. We'll reopen it if the issue returns.
bcmills commentedon Feb 16, 2022
greplogs --dashboard -md -l -e FAIL\\s+heapsampling\\.go --since=2021-12-152022-02-15T21:18:59-08ed488/windows-386-2008
Given the timing (and the relative rarity since the last failure), retargeting to Go 1.19.
21 remaining items
mknyszek commentedon May 11, 2022
It's not so surprising to me. :) I encountered this failure on js/wasm when landing the SetMemoryLimit changes. Basically, js/wasm is very slow and single-threaded, so it can trigger the GC CPU limiter if it gets particularly unlucky, so it's especially sensitive to bugs in the GC CPU limiter, at least when a test allocates in a loop like this one.
I suspect https://go.dev/cl/404304 will help on that front, since there are still bugs. :)
[-]runtime: "out of memory" in heapsampling.go on windows-386 and js-wasm since 2021-11-05[/-][+]runtime: "out of memory" in heapsampling.go on 386 and wasm since 2021-11-05[/+]bcmills commentedon May 12, 2022
A different
386platform this time!greplogs -l -e FAIL\\s+heapsampling\\.go --since=2022-05-112022-05-11T23:39:27-ec5bdef/freebsd-386-12_3
mknyszek commentedon May 23, 2022
Looking back over this, I think the only thing to do here is to slow down the allocation rate of this test. If it remains an issue after that, we should consider just skipping the test on some platforms. I hope (🤞 ) that the js/wasm issues are resolved by the fixes to the GC CPU limiter.
gopherbot commentedon May 27, 2022
Change https://go.dev/cl/408825 mentions this issue:
test/heapsampling.go: slow down allocation rate and reduce iterationstest/heapsampling.go: slow down allocation rate and reduce iterations