a1325b902d breaks build on OpenBSD

[jkeenan]

After a recent commit we have begun to experience build and test failures on OpenBSD.

See, for example, either of these smoke-test reports:

https://perl5.test-smoke.org/report/5001906 (@cguevara)
v5.33.6-42-g6dc0bc8817
openbsd 6.8 using clang version OpenBSD Clang 10.0.1

https://perl5.test-smoke.org/report/5001921 (@jkeenan)
v5.33.6-44-g874732f8cc
openbsd 6.6 using cc version 4.2.1 Compatible OpenBSD Clang 8.0.1

Some individual smoke-runs are getting through make successfully, but then encountering test failures. Some are not even making it all the way through make. Here are excerpts from the smokecurrent.log for 5001921:

[2021-02-07 00:31:30+0000] make ...
[2021-02-07 00:31:30+0000] [make ]
[2021-02-07 00:36:18+0000] Unable to make anything but miniperl in this configuration
[2021-02-07 00:36:18+0000] TSTENV = minitest
[2021-02-07 00:36:18+0000]
[2021-02-07 00:36:18+0000] You may see some irrelevant test failures if you have been unable
[2021-02-07 00:36:18+0000] You may see some irrelevant test failures if you have been unable
[2021-02-07 00:36:18+0000] to build lib/Config.pm, or the Unicode data files.
[2021-02-07 00:36:18+0000] to build lib/Config.pm, or the Unicode data files.

...
[2021-02-07 00:39:22+0000] Error while reading test-results: 1
[2021-02-07 00:39:22+0000]     You may see some irrelevant test failures if you have been unable
[2021-02-07 00:39:22+0000]     to build lib/Config.pm, or the Unicode data files.
[2021-02-07 00:39:22+0000]     t/re/pat ....................... FAILED--no leader found
[2021-02-07 00:39:22+0000]     t/re/regex_sets ................ FAILED--no leader found
[2021-02-07 00:39:22+0000]     t/re/script_run ................ FAILED--no leader found
[2021-02-07 00:39:22+0000]     t/op/split_unicode ............. FAILED--no leader found
[2021-02-07 00:39:22+0000]     t/op/tr ........................ FAILED--no leader found

I decided to explore the build failures first. On my OpenBSD-6.6 VM, I then built blead at HEAD:

# On OpenBSD-6.6
$ git describe
v5.33.6-44-g874732f8cc
$ sh ./Configure -des -Dusedevel Duseithreads -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing" \
  && make minitest_prep
...
# terminates with:
Compiling Perl properties
Creating Perl synonyms
Writing tables
*** Signal SIGSEGV in . (makefile:392 'uni.data')
*** Error 1 in target 'minitest_prep' (ignored)
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)

So the problems are showing up even before we attempt to test minitest. But proceeding:

$ make minitest
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)
cd t &&   ./perl -Ilib -I. TEST base/*.t comp/*.t cmd/*.t run/*.t io/*.t re/*.t opbasic/*.t op/*.t uni/*.t perf/*.t </dev/tty
t/base/cond .................... ok
...
t/re/overload .................. ok
t/re/pat ....................... Can't locate unicore/Name.pm in @INC (you may need to install the unicore::Name module) (@INC contains: ../lib . ../ext/re /home/jkeenan/gitwork/perl/cpan/AutoLoader/lib /home/jkeenan/gitwork/perl/dist/Carp/lib /home/jkeenan/gitwork/perl/dist/PathTools /home/jkeenan/gitwork/perl/dist/PathTools/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Install/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-MakeMaker/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Manifest/lib /home/jkeenan/gitwork/perl/cpan/File-Path/lib /home/jkeenan/gitwork/perl/ext/re /home/jkeenan/gitwork/perl/dist/Term-ReadLine/lib /home/jkeenan/gitwork/perl/dist/Exporter/lib /home/jkeenan/gitwork/perl/ext/File-Find/lib /home/jkeenan/gitwork/perl/cpan/Text-Tabs/lib /home/jkeenan/gitwork/perl/dist/constant/lib /home/jkeenan/gitwork/perl/cpan/version/lib /home/jkeenan/gitwork/perl/cpan/Getopt-Long/lib /home/jkeenan/gitwork/perl/cpan/Text-ParseWords/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-PL2Bat/lib /home/jkeenan/gitwork/perl/lib .. . .) at ../lib/_charnames.pm line 10.
BEGIN failed--compilation aborted at ../lib/_charnames.pm line 10.
Compilation failed in require at re/pat.t line 1364.
BEGIN failed--compilation aborted at re/pat.t line 1364.
FAILED--no leader found
t/re/pat_advanced .............. skipped
t/re/reg_posixcc ............... ok
t/re/regex_sets ................ Can't locate unicore/Name.pm in @INC (you may need to install the unicore::Name module) (@INC contains: ../lib . ../ext/re /home/jkeenan/gitwork/perl/cpan/AutoLoader/lib /home/jkeenan/gitwork/perl/dist/Carp/lib /home/jkeenan/gitwork/perl/dist/PathTools /home/jkeenan/gitwork/perl/dist/PathTools/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Install/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-MakeMaker/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Manifest/lib /home/jkeenan/gitwork/perl/cpan/File-Path/lib /home/jkeenan/gitwork/perl/ext/re /home/jkeenan/gitwork/perl/dist/Term-ReadLine/lib /home/jkeenan/gitwork/perl/dist/Exporter/lib /home/jkeenan/gitwork/perl/ext/File-Find/lib /home/jkeenan/gitwork/perl/cpan/Text-Tabs/lib /home/jkeenan/gitwork/perl/dist/constant/lib /home/jkeenan/gitwork/perl/cpan/version/lib /home/jkeenan/gitwork/perl/cpan/Getopt-Long/lib /home/jkeenan/gitwork/perl/cpan/Text-ParseWords/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-PL2Bat/lib /home/jkeenan/gitwork/perl/lib .. . .) at ../lib/_charnames.pm line 10.
BEGIN failed--compilation aborted at ../lib/_charnames.pm line 10.
Compilation failed in require at re/regex_sets.t line 58.
BEGIN failed--compilation aborted at re/regex_sets.t line 58.
FAILED--no leader found
t/re/regex_sets_compat ......... skipped
...
\t/re/rt122747 .................. skipped
t/re/rxcode .................... ok
t/re/script_run ................ Can't locate unicore/Name.pm in @INC (you may need to install the unicore::Name module) (@INC contains: ../lib /home/jkeenan/gitwork/perl/cpan/AutoLoader/lib /home/jkeenan/gitwork/perl/dist/Carp/lib /home/jkeenan/gitwork/perl/dist/PathTools /home/jkeenan/gitwork/perl/dist/PathTools/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Install/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-MakeMaker/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Manifest/lib /home/jkeenan/gitwork/perl/cpan/File-Path/lib /home/jkeenan/gitwork/perl/ext/re /home/jkeenan/gitwork/perl/dist/Term-ReadLine/lib /home/jkeenan/gitwork/perl/dist/Exporter/lib /home/jkeenan/gitwork/perl/ext/File-Find/lib /home/jkeenan/gitwork/perl/cpan/Text-Tabs/lib /home/jkeenan/gitwork/perl/dist/constant/lib /home/jkeenan/gitwork/perl/cpan/version/lib /home/jkeenan/gitwork/perl/cpan/Getopt-Long/lib /home/jkeenan/gitwork/perl/cpan/Text-ParseWords/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-PL2Bat/lib /home/jkeenan/gitwork/perl/lib .. . .) at ../lib/_charnames.pm line 10.
BEGIN failed--compilation aborted at ../lib/_charnames.pm line 10.
Compilation failed in require at re/script_run.t line 24.
BEGIN failed--compilation aborted at re/script_run.t line 24.
FAILED--no leader found
t/re/speed ..................... skipped
...
t/op/split ..................... ok
t/op/split_unicode ............. Can't locate unicore/Name.pm in @INC (you may need to install the unicore::Name module) (@INC contains: /home/jkeenan/gitwork/perl/cpan/AutoLoader/lib /home/jkeenan/gitwork/perl/dist/Carp/lib /home/jkeenan/gitwork/perl/dist/PathTools /home/jkeenan/gitwork/perl/dist/PathTools/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Install/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-MakeMaker/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Manifest/lib /home/jkeenan/gitwork/perl/cpan/File-Path/lib /home/jkeenan/gitwork/perl/ext/re /home/jkeenan/gitwork/perl/dist/Term-ReadLine/lib /home/jkeenan/gitwork/perl/dist/Exporter/lib /home/jkeenan/gitwork/perl/ext/File-Find/lib /home/jkeenan/gitwork/perl/cpan/Text-Tabs/lib /home/jkeenan/gitwork/perl/dist/constant/lib /home/jkeenan/gitwork/perl/cpan/version/lib /home/jkeenan/gitwork/perl/cpan/Getopt-Long/lib /home/jkeenan/gitwork/perl/cpan/Text-ParseWords/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-PL2Bat/lib /home/jkeenan/gitwork/perl/lib .. . .) at /home/jkeenan/gitwork/perl/lib/charnames.pm line 5.
BEGIN failed--compilation aborted at /home/jkeenan/gitwork/perl/lib/charnames.pm line 5.
Compilation failed in require at op/split_unicode.t line 13.
BEGIN failed--compilation aborted at op/split_unicode.t line 13.
FAILED--no leader found
t/op/sprintf ................... ok
...
t/op/time_loop ................. ok
t/op/tr ........................ Can't locate unicore/Name.pm in @INC (you may need to install the unicore::Name module) (@INC contains: ../lib /home/jkeenan/gitwork/perl/cpan/AutoLoader/lib /home/jkeenan/gitwork/perl/dist/Carp/lib /home/jkeenan/gitwork/perl/dist/PathTools /home/jkeenan/gitwork/perl/dist/PathTools/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Install/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-MakeMaker/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-Manifest/lib /home/jkeenan/gitwork/perl/cpan/File-Path/lib /home/jkeenan/gitwork/perl/ext/re /home/jkeenan/gitwork/perl/dist/Term-ReadLine/lib /home/jkeenan/gitwork/perl/dist/Exporter/lib /home/jkeenan/gitwork/perl/ext/File-Find/lib /home/jkeenan/gitwork/perl/cpan/Text-Tabs/lib /home/jkeenan/gitwork/perl/dist/constant/lib /home/jkeenan/gitwork/perl/cpan/version/lib /home/jkeenan/gitwork/perl/cpan/Getopt-Long/lib /home/jkeenan/gitwork/perl/cpan/Text-ParseWords/lib /home/jkeenan/gitwork/perl/cpan/ExtUtils-PL2Bat/lib /home/jkeenan/gitwork/perl/lib .. . .) at ../lib/_charnames.pm line 10.
BEGIN failed--compilation aborted at ../lib/_charnames.pm line 10.
Compilation failed in require at op/tr.t line 512.
BEGIN failed--compilation aborted at op/tr.t line 512.
FAILED--no leader found
t/op/tr_latin1 ................. ok
...
t/perf/taint ................... skipped
Failed 5 tests out of 323, 98.45% okay.
	op/split_unicode.t
	op/tr.t
	re/pat.t
	re/regex_sets.t
	re/script_run.t
...
Elapsed: 150 sec
u=0.37  s=1.09  cu=35.39  cs=34.59  scripts=323  tests=90848
*** Error 1 in /home/jkeenan/gitwork/perl (makefile:817 'minitest')

The test failures above are similar to what I got in the smoke-test runs.

I got the same failures in make minitest_prep on unthreaded builds.

I then manually bisected and attempted threaded builds.

$ git checkout 99dbf64538
...
$ git describe
v5.33.6-34-g99dbf64538

make minitest_prep completed without any evidence of shortcoming. make minitest PASSed.

Next:

$ git checkout a1325b902d57aa7a99bed3d2ec0fa5ce42836207
Previous HEAD position was 99dbf64538 Rename CxTRYBLOCK() to CxEVALBLOCK()
HEAD is now at a1325b902d Initial attempt at feature 'try'
$ git describe
v5.33.6-35-ga1325b902d

$ sh ./Configure -des -Dusedevel Duseithreads -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing" && make minitest_prep
...
Creating Perl synonyms
Writing tables
*** Signal SIGSEGV in . (makefile:392 'uni.data')
*** Error 1 in target 'minitest_prep' (ignored)
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)

make minitest then failed in the files reported above.

Calling in the bomb squad: @leonerd, @khwilliamson, @afresh1

Thank you very much.
Jim Keenan

[jkeenan]

Now, as if life weren't interesting enough ...

When I reconfigured and rebuilt miniperl at the problematic commit above a second time, in each of unthreaded and threaded builds, (a) I got no problematic output in make minitest_prep; (b) the tests in make minitest that failed first time through passed; but (c) I got a segfault in t/op/sort.t.

$ ./miniperl -Ilib t/op/sort.t
1..204
ok 1 - upper first 1
ok 2 - upper first 2
...
ok 38 - sortname local 1
ok 39 - sortname local 2
ok 40 - sortname local 3
ok 41 - sortname local 4
ok 42 - sortname local 5
ok 43 - sortname local 6
ok 44 - sortname local 7
ok 45 - sortname local 8
Segmentation fault (core dumped) 

[jkeenan]

Now, as if life weren't interesting enough ...

When I reconfigured and rebuilt miniperl at the problematic commit above a second time, in each of unthreaded and threaded builds, (a) I got no problematic output in make minitest_prep; (b) the tests in make minitest that failed first time through passed; but (c) I got a segfault in t/op/sort.t.

$ ./miniperl -Ilib t/op/sort.t
1..204
ok 1 - upper first 1
ok 2 - upper first 2
...
ok 38 - sortname local 1
ok 39 - sortname local 2
ok 40 - sortname local 3
ok 41 - sortname local 4
ok 42 - sortname local 5
ok 43 - sortname local 6
ok 44 - sortname local 7
ok 45 - sortname local 8
Segmentation fault (core dumped) 

This morning, I moved to a different git checkout directory on the same OpenBSD-6.6 VM. I did git clean -dfx, then configured for a threaded build:

sh ./Configure -des -Dusedevel Duseithreads -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"

I then called make minitest_prep. It terminated with this output:

Creating Perl synonyms
Writing tables
*** Signal SIGSEGV in . (makefile:392 'uni.data')
*** Error 1 in target 'minitest_prep' (ignored)
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)

I sensed that trouble lay ahead. I decided to run just one file.

$ ./miniperl -Ilib t/op/sort.t
...
ok 163 - overload string called twice
which_perl: cannot find ./miniperl from ./miniperl at ./test.pl line 864.
not ok 164 - RT \#72334
# Failed test 164 - RT \#72334 at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
not ok 165 - RT \#72334
# Failed test 165 - RT \#72334 at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; @_ = 0..2; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
ok 166 - None before we start
ok 167 - 2 here
ok 168
ok 169 - sorted!
ok 170 - still the same 2 here
ok 171 - all gone
not ok 172 - [perl \#77930] cx_stack reallocation during sort
# Failed test 172 - [perl \#77930] cx_stack reallocation during sort at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "ok"
# PROG: 
# 
#    $sub = sub {
#     local $count = $count+1;
#     ()->$sub if $count < 1000;
#     $a cmp $b
#    };
#    () = sort $sub qw<a b c d e f g>;
#    print "ok"
#  
# STATUS: 32512
ok 173 - Match vars do not leak from one plain sort sub to the next
ok 174 - Match vars do not leak from one $$ sort sub to the next
ok 175 - stubborn AUTOLOAD
ok 176 - AUTOLOAD without stub
ok 177 - AUTOLOAD with stubref
ok 178 - (sort) does not die
ok 179 - (sort) returns empty list
ok 180 - sort; does not die
ok 181 - sort; returns empty list
ok 182 - {sort} does not die
...
ok 203
ok 204 - GH \#18081

The program ran to completion but experienced test failures in each of the 3 unit tests which are run with fresh_perl_is().

I next ran make minitest. This time I got 6 test failures. t/op/sort.t concluded prematurely with:

t/op/sort ...................... FAILED--expected 204 tests, saw 45

The other 5 test failures were:

	op/split_unicode.t
	op/tr.t
	re/pat.t
	re/regex_sets.t
	re/script_run.t

... and displayed Can't locate unicore/Name.pm in @INC ... output as previously reported.

I then cleaned that directory, re-configured for the same threaded build, and again called make minitest_prep. It terminated cleanly.

$ make minitest_prep
...
Creating Perl synonyms
Writing tables
Making pod file
Making test script
Updating 'mktables.lst'
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)

Note that in my first run of the day, the last individual command within minitest_prep to complete successfully was:

Writing tables

But here 3 more commands completed:

Making pod file
Making test script
Updating 'mktables.lst'

So now I had expectations of better test results.

$ ./miniperl -Ilib t/op/sort.t                                             
1..204
ok 1 - upper first 1
ok 2 - upper first 2
...
ok 38 - sortname local 1
ok 39 - sortname local 2
ok 40 - sortname local 3
ok 41 - sortname local 4
ok 42 - sortname local 5
ok 43 - sortname local 6
ok 44 - sortname local 7
ok 45 - sortname local 8
Segmentation fault (core dumped) 

Grrrr! That early segfault again.

$ make minitest
...
t/op/sort ...................... FAILED--expected 204 tests, saw 45
...
Failed 1 test out of 353, 99.72% okay.
	op/sort.t

The 5 files that failed during my first run of make minitest now PASS, but t/op/sort.t still segfaults early in the game.

At this point I ran make test_harness to complete the build and run the entire test suite with ./perl. All tests PASSed -- including t/op/sort.t!

So why is it that t/op/sort.t runs to completion, albeit with 3 unit test failures, the first time it's tested with miniperl -- but then afterwards segfaults before any of those 3 failing unit tests are reached?

And why is it that make minitest_prep fails to execute 3 commands the first time through but completes all commands the second time?

And why is it that when configuring and building a second time within the same checkout, 5 test files now PASS?

[jkeenan]

See also discussion on list starting here.

[afresh1]

My sparc64 smoker seems to sometimes dislike sorting too, but a plain ./Configure -de -Dusedevel at 874732f told me "All tests successful." so not sure what the difference is yet, but I haven't read through the above linked thread yet.

[jkeenan]

Related: #18540

[dur-randir]

It fails in mktables at

foreach my $property ($perl,
                          sort { return -1 if defined $a->file;
                                 return 1 if defined $b->file;
                                 return $a->name cmp $b->name;
                                } grep { $_ != $perl } property_ref('*'))

which is exactly #18540. The random nature is due to how OpenBSD allocator works.

[jkeenan]

It fails in mktables at

foreach my $property ($perl,
                          sort { return -1 if defined $a->file;
                                 return 1 if defined $b->file;
                                 return $a->name cmp $b->name;
                                } grep { $_ != $perl } property_ref('*'))

which is exactly #18540. The random nature is due to how OpenBSD allocator works.

Thanks. Although I'm running these tests on OpenBSD, I don't know what the allocator is or how it works. Can you elaborate?

And can you suggest a remedy?

Thank you very much.
Jim Keenan