Assertion failure in S_find_span_end_mask (regexec.c:689)

[p5pRT]

Migrated from rt.perl.org#133999 (status was 'resolved')

Searchable as RT133999$

[p5pRT]

From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.9-63-g2496d8f3f7 built with afl and run
under libdislocator, I found the following program

use utf8; q!0 æ! =~ m!\b(*COMMIT)(*FAIL)!

to cause an assertion failure

perl​: regexec.c​:6162​: S_regmatch​: Assertion `(((U8*) reginfo->strend)

(S_reghop3((U8*)locinput, -1, (U8*)(reginfo->strbeg))))' failed.

GDB stack trace is following

#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
#1 0x00007ffff7c25535 in __GI_abort () at abort.c​:79
#2 0x00007ffff7c2540f in __assert_fail_base (fmt=0x7ffff7d87ee0
"%s%s%s​:%u​: %s%sAssertion `%s' failed.\n%n",
  assertion=0x555555aeb948 "(((U8*) reginfo->strend) >
(S_reghop3((U8*)locinput, -1, (U8*)(reginfo->strbeg))))",
file=0x555555ae0348 "regexec.c",
  line=6558, function=<optimized out>) at assert.c​:92
#3 0x00007ffff7c330f2 in __GI___assert_fail (assertion=0x555555aeb948
"(((U8*) reginfo->strend) > (S_reghop3((U8*)locinput, -1,
(U8*)(reginfo->strbeg))))",
  file=0x555555ae0348 "regexec.c", line=6558,
function=0x555555af30b0 <__PRETTY_FUNCTION__.19772> "S_regmatch") at
assert.c​:101
#4 0x00005555558b7e99 in S_regmatch (reginfo=0x7fffffffdca0,
startpos=0x555555b777a5 "U", prog=0x555555b80b3c) at regexec.c​:6554
#5 0x00005555558ae0bf in S_regtry (reginfo=0x7fffffffdca0,
startposp=0x7fffffffd7a8) at regexec.c​:3956
#6 0x00005555558a2d8c in S_find_byclass (prog=0x555555b79690,
c=0x555555b80b3c, s=0x555555b777a5 "U", strend=0x555555b777a4 "",
reginfo=0x7fffffffdca0)
  at regexec.c​:2431
#7 0x00005555558ad12a in Perl_regexec_flags (rx=0x555555b72df0,
stringarg=0x555555b777a0 "0 æ", strend=0x555555b777a4 "",
strbeg=0x555555b777a0 "0 æ",
  minend=0, sv=0x555555b72dd8, data=0x0, flags=97) at regexec.c​:3701
#8 0x0000555555769960 in Perl_pp_match () at pp_hot.c​:3018
#9 0x000055555570b640 in Perl_runops_debug () at dump.c​:2537
#10 0x00005555555ed560 in S_run_body (oldscope=1) at perl.c​:2716
#11 0x00005555555ecade in perl_run (my_perl=0x555555b4c260) at perl.c​:2639
#12 0x00005555555a114e in main (argc=4, argv=0x7fffffffe188,
env=0x7fffffffe1b0) at perlmain.c​:127

This is a regression between 5.24 and 5.26, bisect points to

commit 7a20706
Author​: Karl Williamson <khw@​cpan.org>
Date​: Wed Nov 30 09​:53​:17 2016 -0700

  Convert core (except toke.c) to use isFOO_utf8_safe()

  The previous commit added this feature; now this commit uses it in core.
  toke.c is deferred to the next commit to aid in possible future
  bisecting, because some of the changes there seem somewhat more likely
  to expose bugs.

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

From @dur-randir

Created by @dur-randir

While fuzzing perl v5.29.9-63-g2496d8f3f7 built with afl and run
under libdislocator, I found the following program

0 =~ /\p{nv​:(\B(*COMMIT)C+)}/

to cause an assertion failure on debugging builds and triggering
global-buffer-overflow ASAN diagnostics on release builds

perl​: regexec.c​:689​: U8 *S_find_span_end_mask(U8 *, const U8 *, const
U8, const U8)​: Assertion `send >= s' failed.

GDB stack trace is following

#0 __GI_raise (sig=sig@​entry=6) at ../sysdeps/unix/sysv/linux/raise.c​:50
#1 0x00007ffff7c25535 in __GI_abort () at abort.c​:79
#2 0x00007ffff7c2540f in __assert_fail_base (fmt=0x7ffff7d87ee0
"%s%s%s​:%u​: %s%sAssertion `%s' failed.\n%n", assertion=0x555555ae0fe8
"send >= s",
  file=0x555555ae0348 "regexec.c", line=689, function=<optimized
out>) at assert.c​:92
#3 0x00007ffff7c330f2 in __GI___assert_fail (assertion=0x555555ae0fe8
"send >= s", file=0x555555ae0348 "regexec.c", line=689,
  function=0x555555af2d80 <__PRETTY_FUNCTION__.18535>
"S_find_span_end_mask") at assert.c​:101
#4 0x000055555589b48d in S_find_span_end_mask (s=0x555555a7bb68
"1/320", send=0x555555a7bb67 "", span_byte=67 'C', mask=223 '\337') at
regexec.c​:689
#5 0x00005555558c6888 in S_regrepeat (prog=0x555555b797a8,
startposp=0x7fffffffb3a0, p=0x555555b7cd0c, loceol=0x555555a7bb67 "",
reginfo=0x7fffffffbf70,
  max=2147483647, depth=1) at regexec.c​:9515
#6 0x00005555558c193e in S_regmatch (reginfo=0x7fffffffbf70,
startpos=0x555555a7bb68 "1/320", prog=0x555555b7ccfc) at
regexec.c​:8490
#7 0x00005555558ae363 in S_regtry (reginfo=0x7fffffffbf70,
startposp=0x7fffffffba78) at regexec.c​:3956
#8 0x00005555558a3bd1 in S_find_byclass (prog=0x555555b797a8,
c=0x555555b7ccfc, s=0x555555a7bb68 "1/320", strend=0x555555a7bb67 "",
reginfo=0x7fffffffbf70)
  at regexec.c​:2452
#9 0x00005555558ad3ce in Perl_regexec_flags (rx=0x555555b73170,
stringarg=0x555555a7bb63 "-1/2", strend=0x555555a7bb67 "",
strbeg=0x555555a7bb63 "-1/2",
  minend=0, sv=0x555555b73188, data=0x0, flags=1) at regexec.c​:3701
#10 0x000055555589b703 in Perl_pregexec (prog=0x555555b73170,
stringarg=0x555555a7bb63 "-1/2", strend=0x555555a7bb67 "",
strbeg=0x555555a7bb63 "-1/2",
  minend=0, screamer=0x555555b73188, nosave=0) at regexec.c​:765
#11 0x00005555556f2944 in Perl_parse_uniprop_string
(name=0x555555b796c3 "nv​:(\\B(*COMMIT)C+)}", name_len=18,
is_utf8=false, to_fold=false, runtime=false,
  deferrable=true, user_defined_ptr=0x7fffffffc568,
msg=0x555555b73128, level=0) at regcomp.c​:22708
#12 0x00005555556db023 in S_regclass (pRExC_state=0x7fffffffd650,
flagp=0x7fffffffcd14, depth=5, stop_at_1=true,
allow_mutiple_chars=false,
  silence_non_portable=false, strict=false, optimizable=true,
ret_invlist=0x0) at regcomp.c​:17104
#13 0x00005555556c5c79 in S_regatom (pRExC_state=0x7fffffffd650,
flagp=0x7fffffffcd14, depth=4) at regcomp.c​:13647
#14 0x00005555556bc8b9 in S_regpiece (pRExC_state=0x7fffffffd650,
flagp=0x7fffffffce30, depth=3) at regcomp.c​:12457
#15 0x00005555556bc1b9 in S_regbranch (pRExC_state=0x7fffffffd650,
flagp=0x7fffffffced8, first=1, depth=2) at regcomp.c​:12377
#16 0x00005555556b99db in S_reg (pRExC_state=0x7fffffffd650, paren=0,
flagp=0x7fffffffd388, depth=1) at regcomp.c​:12088
#17 0x000055555569d071 in Perl_re_op_compile (patternp=0x0,
pat_count=1, expr=0x555555b79598, eng=0x555555b41d20
<PL_core_reg_engine>, old_re=0x0,
  is_bare_re=0x0, orig_rx_flags=0, pm_flags=0) at regcomp.c​:7705
#18 0x00005555555ba159 in Perl_pmruntime (o=0x555555b795d8,
expr=0x555555b79598, repl=0x0, flags=1, floor=0) at op.c​:7127
#19 0x000055555566ffc3 in Perl_yyparse (gramtype=258) at perly.y​:1234
#20 0x00005555555ec726 in S_parse_body (env=0x0, xsinit=0x5555555a11f8
<xs_init>) at perl.c​:2531
#21 0x00005555555ea9f8 in perl_parse (my_perl=0x555555b4c260,
xsinit=0x5555555a11f8 <xs_init>, argc=2, argv=0x7fffffffe1c8, env=0x0)
at perl.c​:1822
#22 0x00005555555a113b in main (argc=2, argv=0x7fffffffe1c8,
env=0x7fffffffe1e0) at perlmain.c​:126

This is a regression in blead, bisect points to

commit 1532347
Author​: Karl Williamson <khw@​cpan.org>
Date​: Mon Mar 11 17​:16​:34 2019 -0600

  Add Unicode property wildcards

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.29.9:

Configured by dur-randir at Wed Feb 27 14:51:01 MSK 2019.

Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
  Commit id: c1e47bad34ce1d9c84ed57c9b8978bcbd5a02e98
  Platform:
    osname=darwin
    osvers=13.4.0
    archname=darwin-thread-multi-2level
    uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0:
mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64
x86_64 '
    config_args='-de -Dusedevel -DDEBUGGING -Dusethreads'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3 -g'
    cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
    ccversion=''
    gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/usr/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=
    so=dylib
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined
dynamic_lookup -L/usr/local/lib -fstack-protector'



@INC for perl 5.29.9:
    lib
    /usr/local/lib/perl5/site_perl/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/site_perl/5.29.9
    /usr/local/lib/perl5/5.29.9/darwin-thread-multi-2level
    /usr/local/lib/perl5/5.29.9


Environment for perl 5.29.9:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/dur-randir
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin
    PERLBREW_HOME=/Users/dur-randir/.perlbrew
    PERLBREW_MANPATH=/Users/dur-randir/perlbrew/perls/perl-5.22.1/man
    PERLBREW_PATH=/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin
    PERLBREW_PERL=perl-5.22.1
    PERLBREW_ROOT=/Users/dur-randir/perlbrew
    PERLBREW_SHELLRC_VERSION=0.84
    PERLBREW_VERSION=0.84
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

[p5pRT]

@khwilliamson - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

Thanks for finding this. Fixed by
commit b92b270
Author​: Karl Williamson <khw@​cpan.org>
Date​: Tue Apr 9 20​:22​:34 2019 -0600

  PATCH​: [perl #133999] Assertion failure in regex match
 
  This was caused by failing to limit matching to within the bounds of the
  target string. I'm pretty sure this bug has long been there, but was
  exposed by the recently added wildcard property matching feature.
--
Karl Williamson

[p5pRT]

@khwilliamson - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

This was the same fundamental problem as [perl #133999], which this ticket is now merged into
--
Karl Williamson

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'