CI: New Intermittent segfault

[Keno]

We appear to have a new intermittent segfault on CI. I regularly see ProcessExited(139) on macos builders, but it's possible that failures on various other systems are also related. See below for a table of recent macos logs that failed with this error. Looks like this might have started around Jul 4 or so.

Table of recent ProcessExited(139) failures

12 rows × 9 columns

	agent	commit	date	elapsed	key	log_url	state	uuid	web_url
	String	String	DateTime	Float64	String	String	String	UUID	String
1	default-macmini-x64-5.0	743578a	2022-07-22T05:11:28.008	4911.28	test_x86_64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/14170/jobs/01822451-f8cd-43a7-a4b4-83d37d4236e4/log.txt	failed	01822451-f8cd-43a7-a4b4-83d37d4236e4	https://buildkite.com/julialang/julia-master/builds/14170#01822451-f8cd-43a7-a4b4-83d37d4236e4
2	default-macmini-x64-5.0	028e9ff	2022-07-20T15:15:51.024	8267.88	test_x86_64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/14094/jobs/01821c2e-dd00-4443-89d2-a1f346cf8bda/log.txt	failed	01821c2e-dd00-4443-89d2-a1f346cf8bda	https://buildkite.com/julialang/julia-master/builds/14094#01821c2e-dd00-4443-89d2-a1f346cf8bda
3	default-macmini-aarch64-1.0	e96b19d	2022-07-14T18:22:37.245	4810.07	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13912/jobs/0181fdf3-8b9c-4341-b2a6-2301ec77ce8f/log.txt	failed	0181fdf3-8b9c-4341-b2a6-2301ec77ce8f	https://buildkite.com/julialang/julia-master/builds/13912#0181fdf3-8b9c-4341-b2a6-2301ec77ce8f
4	default-macmini-aarch64-1.0	9815a8e	2022-07-13T15:43:51.353	3703.01	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13836/jobs/0181f95e-1f7d-420f-b0c6-86a38d2872ad/log.txt	failed	0181f95e-1f7d-420f-b0c6-86a38d2872ad	https://buildkite.com/julialang/julia-master/builds/13836#0181f95e-1f7d-420f-b0c6-86a38d2872ad
5	default-macmini-aarch64-1.0	201d4f6	2022-07-13T09:02:25.148	3797.28	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13824/jobs/0181f6cc-1f9e-40df-98e4-9858799342a9/log.txt	failed	0181f6cc-1f9e-40df-98e4-9858799342a9	https://buildkite.com/julialang/julia-master/builds/13824#0181f6cc-1f9e-40df-98e4-9858799342a9
6	default-macmini-x64-2.0	201d4f6	2022-07-13T09:02:25.148	5804.43	test_x86_64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13824/jobs/0181f6cc-1dd9-4c19-a21b-8c1115c7933a/log.txt	failed	0181f6cc-1dd9-4c19-a21b-8c1115c7933a	https://buildkite.com/julialang/julia-master/builds/13824#0181f6cc-1dd9-4c19-a21b-8c1115c7933a
7	default-macmini-x64-2.0	c1d21e1	2022-07-13T07:32:31.174	5450.19	test_x86_64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13821/jobs/0181f679-cd57-41c5-b95e-853529344ad5/log.txt	failed	0181f679-cd57-41c5-b95e-853529344ad5	https://buildkite.com/julialang/julia-master/builds/13821#0181f679-cd57-41c5-b95e-853529344ad5
8	default-macmini-aarch64-2.0	87558f6	2022-07-08T19:40:26.457	4463.86	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13634/jobs/0181df54-7655-4fcd-97ec-5df1b9421462/log.txt	failed	0181df54-7655-4fcd-97ec-5df1b9421462	https://buildkite.com/julialang/julia-master/builds/13634#0181df54-7655-4fcd-97ec-5df1b9421462
9	default-macmini-aarch64-1.0	da13d78	2022-07-07T08:37:50.394	2840.45	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13599/jobs/0181d7cf-7da6-4f17-8de0-b0a49c6ff550/log.txt	failed	0181d7cf-7da6-4f17-8de0-b0a49c6ff550	https://buildkite.com/julialang/julia-master/builds/13599#0181d7cf-7da6-4f17-8de0-b0a49c6ff550
10	default-macmini-aarch64-3.0	4d50ff8	2022-07-06T12:35:07.440	6386.15	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13567/jobs/0181d385-141c-436f-bfb9-4eb9af26a3d4/log.txt	failed	0181d385-141c-436f-bfb9-4eb9af26a3d4	https://buildkite.com/julialang/julia-master/builds/13567#0181d385-141c-436f-bfb9-4eb9af26a3d4
11	default-macmini-aarch64-4.0	438bde4	2022-07-04T19:45:11.110	5068.01	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13491/jobs/0181cac1-34d3-4c25-9a6c-c7a7d39352fc/log.txt	failed	0181cac1-34d3-4c25-9a6c-c7a7d39352fc	https://buildkite.com/julialang/julia-master/builds/13491#0181cac1-34d3-4c25-9a6c-c7a7d39352fc
12	default-macmini-aarch64-5.0	fec6951	2022-06-30T17:21:33.759	4715.9	test_aarch64-apple-darwin	https://api.buildkite.com/v2/organizations/julialang/pipelines/julia-master/builds/13360/jobs/0181b5a2-791b-4ec9-a4fe-6c7aec87d849/log.txt	failed	0181b5a2-791b-4ec9-a4fe-6c7aec87d849	https://buildkite.com/julialang/julia-master/builds/13360#0181b5a2-791b-4ec9-a4fe-6c7aec87d849

[Keno]

The commit before that first failure was the merge of #45861. @vtjnash any chance that's related?

Actually I missed a previous failure on Jun 30.

[vtjnash]

Is that the atexit hook test failing? I think someone did change that recently (#45765). I think that test pattern might be still expected to fail sometimes if you are running if threads are forced on, so can we please stop doing that in CI?

[giordano]

I think someone did change that recently (#45765).

That'd match the the fact the first failing job was found on June 30th: the PR was merged the day before.

[Keno]

The test that's failing is not the one that was added in that PR though. It's usually either the sysimg-code-native=no test or one of the LibGit2 tests.

[vtjnash]

Both would be very likely to fail if the PR was incompletely fixing the issue, and if people keep configuring CI to run with threads (which is not supported yet)

[DilumAluthge]

and if people keep configuring CI to run with threads (which is not supported yet)

JuliaCI/julia-buildkite#185

[Keno]

Appears to have been resolved by turning off threading.