IO-Socket-SSL fails tests on OpenBSD

[jkeenan]

Today, in the course of trying to install Task::CPAN::Reporter on OpenBSD-6.9 against Perl 5 blead, IO::Socket::SSL repeatedly failed its tests (using both cpan and cpanm as installers.)

$ ./bin/perl -v | head -2 | tail -1
This is perl 5, version 35, subversion 10 (v5.35.10 (v5.35.9-18-g714a0a851b)) built for OpenBSD.amd64-openbsd

Configuring S/SU/SULLR/IO-Socket-SSL-2.074.tar.gz with Makefile.PL
Should I do external tests?
These test will detect if there are network problems and fail soft,
so please disable them only if you definitely don't want to have any
network traffic to external sites.  [Y/n] n
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for IO::Socket::SSL
Writing MYMETA.yml and MYMETA.json
(/home/jkeenan/testing/blead/bin/perl Makefile.PL exited with 0)
CPAN::Reporter: Makefile.PL result is 'pass', No errors.
  SULLR/IO-Socket-SSL-2.074.tar.gz
  /home/jkeenan/testing/blead/bin/perl Makefile.PL -- OK
Running make for S/SU/SULLR/IO-Socket-SSL-2.074.tar.gz
cp lib/IO/Socket/SSL/PublicSuffix.pm blib/lib/IO/Socket/SSL/PublicSuffix.pm
cp lib/IO/Socket/SSL/Intercept.pm blib/lib/IO/Socket/SSL/Intercept.pm
cp lib/IO/Socket/SSL.pod blib/lib/IO/Socket/SSL.pod
cp lib/IO/Socket/SSL/Utils.pm blib/lib/IO/Socket/SSL/Utils.pm
cp lib/IO/Socket/SSL.pm blib/lib/IO/Socket/SSL.pm
(/usr/bin/make exited with 0)
CPAN::Reporter: make result is 'pass', No errors.
  SULLR/IO-Socket-SSL-2.074.tar.gz
  /usr/bin/make -- OK
Running make test for SULLR/IO-Socket-SSL-2.074.tar.gz
PERL_DL_NONLAZY=1 "/home/jkeenan/testing/blead/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
# openssl version compiled=0x20000000 linked=0x20000000 -- LibreSSL 3.3.2
# Net::SSLeay version=1.92
# parent IO::Socket::IP version=0.41
t/01loadmodule.t .................. ok
t/acceptSSL-timeout.t ............. ok
t/alpn.t .......................... ok

#   Failed test 'connection to server.local/ldap succeeded'
#   at t/auto_verify_hostname.t line 61.

#   Failed test 'connection to bla.server.local/www succeeded'
#   at t/auto_verify_hostname.t line 61.

#   Failed test 'connection to www7.other.local/www succeeded'
#   at t/auto_verify_hostname.t line 61.

#   Failed test 'connection to bla.server.local/ldap succeeded'
#   at t/auto_verify_hostname.t line 61.

#   Failed test 'ssl upgrade of connection to server.local/ldap succeeded'
#   at t/auto_verify_hostname.t line 79.

#   Failed test 'ssl upgrade of connection to bla.server.local/www succeeded'
#   at t/auto_verify_hostname.t line 79.

#   Failed test 'ssl upgrade of connection to www7.other.local/www succeeded'
#   at t/auto_verify_hostname.t line 79.

#   Failed test 'ssl upgrade of connection to bla.server.local/ldap succeeded'
#   at t/auto_verify_hostname.t line 79.
# Looks like you planned 30 tests but ran 22.
# Looks like you failed 8 tests of 22 run.
t/auto_verify_hostname.t .......... 
Dubious, test returned 8 (wstat 2048, 0x800)
Failed 16/30 subtests 
t/cert_formats.t .................. ok
t/cert_no_file.t .................. ok
t/compatibility.t ................. ok
t/connectSSL-timeout.t ............ ok
t/core.t .......................... ok
t/dhe.t ........................... ok
t/ecdhe.t ......................... ok
t/io-socket-inet6.t ............... skipped: no IO::Socket::INET6 available
t/io-socket-ip.t .................. ok
t/memleak_bad_handshake.t ......... ok
t/mitm.t .......................... ok
t/multiple-cert-rsa-ecc.t ......... 
Failed 5/12 subtests 
t/nonblock.t ...................... ok
t/npn.t ........................... skipped: NPN not available in Net::SSLeay
# -- test: newINET start_SSL stop_SSL start_SSL
# server accepted new client
# wait for initial data from client
# got 0x666f6f from client
# server: got plain data at start of connection
# server: TLS upgrade
# server: TLS downgrade
# server: TLS upgrade#2
# -- test: newSSL stop_SSL connect_SSL
# server accepted new client
# wait for initial data from client
# got 0x160301 from client
# server: TLS upgrade
# server: TLS downgrade
# server: TLS upgrade#2
# -- test: newSSL:0 connect_SSL stop_SSL connect_SSL
# server accepted new client
# wait for initial data from client
# got 0x666f6f from client
# server: got plain data at start of connection
# server: TLS upgrade
# server: TLS downgrade
# server: TLS upgrade#2
# -- test: newSSL:0 start_SSL stop_SSL connect_SSL
# server accepted new client
# wait for initial data from client
# got 0x666f6f from client
# server: got plain data at start of connection
# server: TLS upgrade
# server: TLS downgrade
# server: TLS upgrade#2
# server accepted new client
# wait for initial data from client
# got 0x656e64 from client
# client requested end of tests
t/plain_upgrade_downgrade.t ....... ok
# looks like OpenSSL was compiled without SSLv3 support
# looks like OpenSSL was compiled without TLSv1_3 support

#   Failed test 'accept SSLv23 with TLSv1_2'
#   at t/protocol_version.t line 135.
#          got: 'TLSv1_3'
#     expected: 'TLSv1_2'
# Looks like you failed 1 test of 10.
t/protocol_version.t .............. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests 
t/public_suffix_lib_encode_idn.t .. ok
t/public_suffix_lib_libidn.t ...... ok
t/public_suffix_lib_uri.t ......... ok
t/public_suffix_ssl.t ............. ok
t/readline.t ...................... ok
t/session_cache.t ................. ok
# listen at 127.0.0.1:32946
# listen at 127.0.0.1:21411
# connect to 0: success reuse=0 version=TLSv1_3
# connect to 0: success reuse=0 version=TLSv1_3

#   Failed test 'reuse with the next session and secret[0]'
#   at t/session_ticket.t line 79.
#          got: '0'
#     expected: '1'
# connect to 1: success reuse=0 version=TLSv1_3

#   Failed test 'reuse even though server changed, since they share ticket secret'
#   at t/session_ticket.t line 79.
#          got: '0'
#     expected: '1'
# connect to 1: success reuse=0 version=TLSv1_3
# connect to 0: success reuse=0 version=TLSv1_3
# connect to 0: success reuse=0 version=TLSv1_3

#   Failed test 'reuse again since got ticket with secret[0] in last step'
#   at t/session_ticket.t line 79.
#          got: '0'
#     expected: '1'
# Looks like you failed 3 tests of 6.
t/session_ticket.t ................ 
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/6 subtests 
Use of uninitialized value in string eq at t/sessions.t line 87.
t/sessions.t ...................... 
Failed 11/17 subtests 
t/set_curves.t .................... ok
t/signal-readline.t ............... ok
t/sni.t ........................... 
Failed 4/17 subtests 
	(less 4 skipped subtests: 9 okay)
t/sni_verify.t .................... 
Failed 4/17 subtests 
	(less 4 skipped subtests: 9 okay)
t/start-stopssl.t ................. ok
t/startssl-failed.t ............... ok
t/startssl.t ...................... ok
t/sysread_write.t ................. ok
t/verify_fingerprint.t ............ ok
t/verify_hostname.t ............... 
Failed 2/45 subtests 
t/verify_hostname_standalone.t .... ok
t/verify_partial_chain.t .......... ok

Test Summary Report
-------------------
t/auto_verify_hostname.t        (Wstat: 2048 Tests: 22 Failed: 8)
  Failed tests:  3, 5-6, 8, 12, 16, 18, 22
  Non-zero exit status: 8
  Parse errors: Bad plan.  You planned 30 tests but ran 22.
t/multiple-cert-rsa-ecc.t       (Wstat: 0 Tests: 12 Failed: 5)
  Failed tests:  3-4, 8, 10-11
t/protocol_version.t            (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  10
  Non-zero exit status: 1
t/session_ticket.t              (Wstat: 768 Tests: 6 Failed: 3)
  Failed tests:  2-3, 6
  Non-zero exit status: 3
t/sessions.t                    (Wstat: 0 Tests: 17 Failed: 11)
  Failed tests:  5, 7-16
t/sni.t                         (Wstat: 0 Tests: 17 Failed: 4)
  Failed tests:  6, 8, 14, 16
t/sni_verify.t                  (Wstat: 0 Tests: 17 Failed: 4)
  Failed tests:  6, 8, 14, 16
t/verify_hostname.t             (Wstat: 0 Tests: 45 Failed: 2)
  Failed tests:  44-45
Files=40, Tests=775, 58 wallclock secs ( 0.23 usr  0.23 sys +  9.25 cusr  3.63 csys = 13.34 CPU)
Result: FAIL
Failed 8/40 test programs. 38/775 subtests failed.
*** Error 255 in /home/jkeenan/.cpan/build/IO-Socket-SSL-2.074-1 (Makefile:868 'test_dynamic')
(/usr/bin/make test exited with 512)
CPAN::Reporter: Test result is 'fail', One or more tests failed.
CPAN::Reporter: preparing a CPAN Testers report for IO-Socket-SSL-2.074
Do you want to review or edit the test report? (yes/no) [no] 
Do you want to send the report? (yes/no) [yes] 
CPAN::Reporter: sending test report with 'fail' via Metabase
CPAN::Reporter: Test::Reporter: error from 'Test::Reporter::Transport::Metabase:'
Scheme 'https' is not supported.
IO::Socket::SSL 1.42 must be installed for https support


Terminal does not support GetHistory.
Lockfile removed.
  SULLR/IO-Socket-SSL-2.074.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports SULLR/IO-Socket-SSL-2.074.tar.gz
Failed during this command:
 SULLR/IO-Socket-SSL-2.074.tar.gz             : make_test NO

These results appear similar to what's being reported to CPANtesters.

Can you investigate?

[noxxi]

Unfortunately LibreSSL is a kind of moving target with varying support for TLS 1.3 and different API support in different versions of LibreSSL. This makes it very hard to support IO::Socket::SSL and Net::SSLeay on LibreSSL. The OpenBSD ports for Net::SSLeay and IO::Socket::SSL thus come with their own patches - which sometimes skip tests and sometimes enforce TLS 1.2. Please use therefore the versions from ports instead.

[jkeenan]

Unfortunately LibreSSL is a kind of moving target with varying support for TLS 1.3 and different API support in different versions of LibreSSL. This makes it very hard to support IO::Socket::SSL and Net::SSLeay on LibreSSL. The OpenBSD ports for Net::SSLeay and IO::Socket::SSL thus come with their own patches - which sometimes skip tests and sometimes enforce TLS 1.2. Please use therefore the versions from ports instead.

@noxxi, the problem with the approach you suggest is that it does not appear to account for the fact that when your objective is to report test results to www.cpantesters.org, you need to install Task::CPAN::Reporter, which includes both IO::Socket::SSL and Net::SSLeay in its chain of dependencies.

It so happens that I already have the OpenBSD ports of these libraries in the Vagrantfile via which I provision my OpenBSD virtual machine:

     # Packages which provide CPAN libraries which facilitate jkeenan's
     # programming (including smoke-testing core distribution):
     sudo pkg_add -u \
        p5-Capture-Tiny \
        p5-common-sense \
        p5-Data-Dump \
        p5-Devel-Cover \
        p5-HTTP-Daemon \
        p5-HTTP-Message \
        p5-IO-Socket-SSL \
        p5-JSON \
        p5-JSON-XS \
        p5-List-Compare \
        p5-Net-SSLeay \
...

This works fine when I'm running an application that makes use of the "system" perl. But much of my own work entails building a specific version of perl (often the HEAD of the blead branch), then installing libraries against that perl and sending CPANtesters reports about those libraries. Yesterday, for example, I wanted to install the latest version of DB_File against blead and, in doing so, generate a report for CPANtesters. I therefore had to install Task::CPAN::Reporter against that blead perl, but I couldn't get past the IO::Socket::SSL test failures (at least not without a force install, which I have not yet attempted). (I was somehow able to get Net::SSLeay installed.)

Can you suggest a workaround?

Thank you very much.
Jim Keenan

[noxxi]

Can you suggest a workaround?

Again, I have problems supporting LibreSSL as a quickly moving target. So the workaround is to use a more stable target, i.e. OpenSSL. It is not perfect either but has much better supported both from Net::SSLeay and IO::Socket::SSL. I suggest to stay with OpenSSL 1.1.1 as the more stable version for now though.

[jkeenan]

I should add that, using a combination of OpenBSD packages and CPAN libraries in the Vagrantfile, I have been able to install Task::CPAN::Reporter against the system perl. I can therefore generate CPANtesters reports against that perl, which on OpenBSD-6.9 happens to be 5.32.1. Example from just now:

CPAN::Reporter: Test result is 'pass', All tests successful.
CPAN::Reporter: preparing a CPAN Testers report for Devel-Chitin-0.18
CPAN::Reporter: sending test report with 'pass' via Metabase
  BRUMMETT/Devel-Chitin-0.18.tar.gz
  /usr/bin/make test -- OK

The problem is that I cannot do an install of the reporting setup against any other perl that's locally installed. (The module cited above, for example, is broken in blead.)