OpenBSD: 3 unit test failures in `./miniperl -Ilib t/op/sort.t`

[jkeenan]

This ticket is an attempt to more precisely specify a problem which we've been discussing in #18536, "a1325b902d breaks build on OpenBSD".

On OpenBSD, ./miniperl -Ilib t/op/sort.t generates 3 unit test failures which we do not see on other OSes and which do not fail when we run the file either via make minitest, ./perl -Ilib t/op/sort.t or cd t; ./perl harness op/sort.t; cd-.

It turns out that we can reproduce these problems in older perls. Here's output from perl-5.24.0.

$ git checkout v5.24.0
$ sh ./Configure -des -Dusedevel && make test_prep
$ ./miniperl -v | head -2 | tail -1
This is perl 5, version 24, subversion 0 (v5.24.0 (UNKNOWN-miniperl)) built for OpenBSD.amd64-openbsd

$ rm err; ./miniperl -Ilib t/op/sort.t 2>err; wc -l err
27

$ cat err
which_perl: cannot find ./miniperl from ./miniperl at ./test.pl line 842.
# Failed test 159 - RT \#72334 at ./test.pl line 1032
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
# Failed test 160 - RT \#72334 at ./test.pl line 1032
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; @_ = 0..2; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
# Failed test 167 - [perl \#77930] cx_stack reallocation during sort at ./test.pl line 1032
#      got "sh: ./miniperl: not found"
# expected "ok"
# PROG: 
# 
#    $sub = sub {
#     local $count = $count+1;
#     ()->$sub if $count < 1000;
#     $a cmp $b
#    };
#    () = sort $sub qw<a b c d e f g>;
#    print "ok"
#  
# STATUS: 32512

Each of the 3 unit test failures is associated with an invocation of fresh_perl_is() out of t/test.pl. These unit tests were first added in 2010 in the 5.13 development cycle. Not surprisingly, perl-5.14.0 is the first production release in which I've gotten these failures.

Note this error output (perl-5.24):

which_perl: cannot find ./miniperl from ./miniperl at ./test.pl line 842.
# Failed test 159 - RT \#72334 at ./test.pl line 1032
#      got "sh: ./miniperl: not found"

That suggests that the problem is some interaction between the shell, t/test.pl and the built miniperl. In the case of OpenBSD, man sh says:

     This version of sh is actually ksh in disguise.  As such, it also
     supports the features described in ksh(1).  This manual page describes
     only the parts relevant to a POSIX compliant sh.  If portability is a
     concern, use only those features described in this page.

Since this error output has helped confuse our analysis in other tickets, I would like to be able to fix the problem soon. Assistance from anyone familiar with any of OpenBSD, Korn shell, miniperl and t/test.pl will be appreciated.

For reference, here is the error output in blead (v5.33.6-44-g874732f8cc):

$ cat err
which_perl: cannot find ./miniperl from ./miniperl at ./test.pl line 864.
# Failed test 164 - RT \#72334 at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
# Failed test 165 - RT \#72334 at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "0 1 2 3"
# PROG: 
# sub w ($$) {my ($l, $r) = @_; my $v = \@_; undef @_; @_ = 0..2; $l <=> $r}; print join q{ }, sort w 3, 1, 2, 0
# STATUS: 32512
# Failed test 172 - [perl \#77930] cx_stack reallocation during sort at ./test.pl line 1077
#      got "sh: ./miniperl: not found"
# expected "ok"
# PROG: 
# 
#    $sub = sub {
#     local $count = $count+1;
#     ()->$sub if $count < 1000;
#     $a cmp $b
#    };
#    () = sort $sub qw<a b c d e f g>;
#    print "ok"
#  
# STATUS: 32512

Thank you very much.
Jim Keenan

[dur-randir]

Those 3 tests use fresh_perl_is(), and only them use it in this file. Actual pwd is t/, and there's no miniperl in it.

[jkeenan]

Those 3 tests use fresh_perl_is(), and only them use it in this file. Actual pwd is t/, and there's no miniperl in it.

We use fresh_perl_is() in 47 test files in t/op/, but we only get these failures in t/op/sort.t and (AFAICT) only on OpenBSD. Do you have any insight into why that is so?

Thank you very much.
Jim Keenan

[dur-randir]

but we only get these failures in t/op/sort.t

If I run ./miniperl -Ilib t/op/warn.t, I get the same style failures. But if I run any of those two with ./perl instead of ./miniperl, they pass. I don't know enough about the test framework to say should or should not it be possible to run them with miniperl.

[tonycoz]

Try like this:

make minitest_prep
cd t
./perl -I../lib op/sort.t

After make minitest_prep t/perl is actually miniperl.

This matches how the tests expect to be run.

As to why there's a difference between OpenBSD and Linux, on Linux we have a way to get a full path to the executable:

$ uname
Linux
$ ./miniperl -le 'print $^X'
/home/tony/dev/perl/git/perl4/miniperl

but on OpenBSD we don't have access to that:

$ uname
OpenBSD
$ ./miniperl -le 'print $^X'
./miniperl

There doesn't seem to be a mechanism to do this in OpenBSD, and the developers don't seem to be inclined to add one.

So this isn't a bug.

[jkeenan]

Try like this:

make minitest_prep
cd t
./perl -I../lib op/sort.t

After make minitest_prep t/perl is actually miniperl.

Aha! That was crucial data I was lacking!

Can we/should we document that somewhere?

This matches how the tests expect to be run.

As to why there's a difference between OpenBSD and Linux, on Linux we have a way to get a full path to the executable:

$ uname
Linux
$ ./miniperl -le 'print $^X'
/home/tony/dev/perl/git/perl4/miniperl

but on OpenBSD we don't have access to that:

$ uname
OpenBSD
$ ./miniperl -le 'print $^X'
./miniperl

Confirmed. Can we/should we document that somewhere?

There doesn't seem to be a mechanism to do this in OpenBSD, and the developers don't seem to be inclined to add one.

So this isn't a bug.

So it would seem. Thanks.

[tonycoz]

After make minitest_prep t/perl is actually miniperl.

Can we/should we document that somewhere?

There's not much documentation of the different build targets, just a bit in INSTALL and perlhack (mostly for testing that I see).

I do think the targets could be better documented, both as to what they do but also why.

This matches how the tests expect to be run.
As to why there's a difference between OpenBSD and Linux, on Linux we have a way to get a full path to the executable:

$ uname
Linux
$ ./miniperl -le 'print $^X'
/home/tony/dev/perl/git/perl4/miniperl

but on OpenBSD we don't have access to that:

$ uname
OpenBSD
$ ./miniperl -le 'print $^X'
./miniperl

Confirmed. Can we/should we document that somewhere?

From perlvar:

Depending on the host operating system, the value of C<$^X> may be
a relative or absolute pathname of the perl program file, or may
be the string used to invoke perl but not the pathname of the
perl program file.  Also, most operating systems permit invoking
programs that are not in the PATH environment variable, so there
is no guarantee that the value of C<$^X> is in PATH.  For VMS, the
value may or may not include a version number.

[jkeenan]

Between (a) the explanation provided by @tonycoz of how to run tests after make minitest_prep and a series of commits by @hvds and @leonerd, it appears that the problems reported in this ticket have been resolved.

I will self-assign the ticket, wait for some more smoke-test results to come in, then close the ticket.

Thank you very much.
Jim Keenan

[jkeenan]

Between (a) the explanation provided by @tonycoz of how to run tests after make minitest_prep and a series of commits by @hvds and @leonerd, it appears that the problems reported in this ticket have been resolved.

I will self-assign the ticket, wait for some more smoke-test results to come in, then close the ticket.

Smoke-test reports look good. Closing ticket.