BBC: blead breaks BKB/Lingua-JA-Moji-0.55.tar.gz #16895

p5pRT · 2019-03-18T20:11:06Z

Migrated from rt.perl.org#133942 (status was 'resolved')

Searchable as RT133942$

p5pRT · 2019-03-18T20:11:06Z

From @eserte

This is a bug report for perl from slaven@rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Flags:
category=core
severity=low

Site configuration information for perl 5.29.8:

Configured by eserte at Thu Feb 21 06:51:23 UTC 2019.

Summary of my perl5 (revision 5 version 29 subversion 8) configuration:

Platform:
osname=linux
osvers=4.15.0-45-generic
archname=x86_64-linux
uname='linux ubuntu18 4.15.0-45-generic #48-ubuntu smp tue jan 29 16:28:13 utc 2019 x86_64 x86_64 x86_64 gnulinux '
config_args='-ds -e -Dprefix=/opt/perl-5.29.8 -Dusedevel -Dusemallocwrap=no -Dcf_email=srezic@cpan.org'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='cc'
ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
optimize='-O2'
cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='7.3.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.27.so
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.27'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

@INC for perl 5.29.8:
/opt/perl-5.29.8/lib/site_perl/5.29.8/x86_64-linux
/opt/perl-5.29.8/lib/site_perl/5.29.8
/opt/perl-5.29.8/lib/5.29.8/x86_64-linux
/opt/perl-5.29.8/lib/5.29.8

Environment for perl 5.29.8:
HOME=/home/eserte
LANG=en_US.UTF-8
LANGUAGE (unset)
LC_ALL=de_DE.UTF-8
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/eserte/bin/linux-gnu:/home/eserte/bin/sh:/home/eserte/bin:/home/eserte/bin/pistachio-perl/bin:/usr/games:/home/eserte/devel
PERLDOC=-MPod::Perldoc::ToTextOverstrike
PERL_BADLANG (unset)
SHELL=/usr/bin/zsh

p5pRT · 2019-03-19T02:24:54Z

From @jkeenan

On Mon, 18 Mar 2019 20:11:06 GMT, slaven@rezic.de wrote:

This is a bug report for perl from slaven@rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl --start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8 --end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua::JA::Moji
#####

... pointed to this commit:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I /home/jkeenan/.cpan -MCPAN::MyConfig -MCPAN -e $CPAN::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e install('Lingua::JA::Moji'); die unless CPAN::Shell->expand(Module => 'Lingua::JA::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author: Karl Williamson <khw@cpan.org>
Date: Mon Aug 20 18:31:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

--
James E Keenan (jkeenan@cpan.org)

p5pRT · 2019-03-19T02:24:54Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2019-03-19T02:38:40Z

From @khwilliamson

On 3/18/19 8:24 PM, James E Keenan via RT wrote:

On Mon, 18 Mar 2019 20:11:06 GMT, slaven@rezic.de wrote:

This is a bug report for perl from slaven@rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl --start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8 --end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua::JA::Moji
#####

... pointed to this commit:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I /home/jkeenan/.cpan -MCPAN::MyConfig -MCPAN -e $CPAN::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e install('Lingua::JA::Moji'); die unless CPAN::Shell->expand(Module => 'Lingua::JA::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author: Karl Williamson <khw@cpan.org>
Date: Mon Aug 20 18:31:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

My guess is that it's the same problem as in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133860

which is the result of the module unsuspectingly relying on a bug in the
former implementation. I may have to come up with a workaround.

p5pRT · 2019-03-24T23:34:45Z

From @khwilliamson

On 3/18/19 8:38 PM, Karl Williamson wrote:

On 3/18/19 8:24 PM, James E Keenan via RT wrote:

On Mon, 18 Mar 2019 20:11:06 GMT, slaven@rezic.de wrote:

This is a bug report for perl from slaven@rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl
--start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8
--end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua::JA::Moji
#####

... pointed to this commit:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from
utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I
/home/jkeenan/.cpan -MCPAN::MyConfig -MCPAN -e
$CPAN::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e
install('Lingua::JA::Moji'); die unless CPAN::Shell->expand(Module =>
'Lingua::JA::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author: Karl Williamson <khw@cpan.org>
Date: Mon Aug 20 18:31:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

My guess is that it's the same problem as in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133860

which is the result of the module unsuspectingly relying on a bug in the
former implementation. I may have to come up with a workaround.

I investigated further and my above guess is wrong.
It turns out that it is a bug in the distro, and I have created a pull
request benkasminbullock/Lingua-JA-Moji#23

The problem is that it had single-quoted '\t' instead of double-quoted
"\t", which created not the intended tab, but backslash-t. The improved
error checking in the new implementation caught it. I did not
investigate to see what the old implementation did; perhaps it compiled
as DWIM, but perhaps it compiled garbage.

But there is a bug in the new implementation which didn't report the
error properly. I will investigate and fix that.

p5pRT · 2019-03-25T16:46:37Z

From @khwilliamson

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged
--
Karl Williamson

p5pRT · 2019-03-25T16:46:38Z

@khwilliamson - Status changed from 'open' to 'pending release'

p5pRT · 2019-03-26T20:50:26Z

From @eserte

Dana Mon, 25 Mar 2019 09:46:37 -0700, khw reče:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the new version still fails with 5.29.9:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

p5pRT · 2019-03-28T19:31:35Z

From @khwilliamson

On 3/26/19 2:50 PM, slaven@rezic.de via RT wrote:

Dana Mon, 25 Mar 2019 09:46:37 -0700, khw reče:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the new version still fails with 5.29.9:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

---
This should be fixed by

commit 766d6d3
Author: Karl Williamson <khw@cpan.org>
Date: Thu Mar 28 12:26:43 2019 -0600

PATCH: [perl #133942] BBC: BKB/Lingua-JA-Moji

This ticket was originally fixed by a PR being issued on the
distribution, which was merged and a new version placed on CPAN.

But before Slaven had a chance to test it, commit
765e6ec came along, and happened to
break it again from a totally independent cause.

So we have two separate bugs on the same ticket, only one of which was
perl's fault. And this commit fixes that. The problematic commit was
still looking in the regnode FLAGS field in one function, whereas that
field has been repurposed for ANYOFH nodes, so should be ignored for
them.

p5pRT · 2019-03-28T19:52:36Z

From @jkeenan

On Thu, 28 Mar 2019 19:31:35 GMT, public@khwilliamson.com wrote:

On 3/26/19 2:50 PM, slaven@rezic.de via RT wrote:

Dana Mon, 25 Mar 2019 09:46:37 -0700, khw reče:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the
new version still fails with 5.29.9:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

---
This should be fixed by

commit 766d6d3
Author: Karl Williamson <khw@cpan.org>
Date: Thu Mar 28 12:26:43 2019 -0600

PATCH: [perl #133942] BBC: BKB/Lingua-JA-Moji

This ticket was originally fixed by a PR being issued on the
distribution, which was merged and a new version placed on CPAN.

But before Slaven had a chance to test it, commit
765e6ec came along, and happened to
break it again from a totally independent cause.

So we have two separate bugs on the same ticket, only one of which was
perl's fault. And this commit fixes that. The problematic commit was
still looking in the regnode FLAGS field in one function, whereas that
field has been repurposed for ANYOFH nodes, so should be ignored for
them.

Lingua::JA::Moji now PASS for me at v5.29.9-38-gca21d46d3d in the FreeBSD environment cited earlier.

Thank you very much.
--
James E Keenan (jkeenan@cpan.org)

p5pRT · 2019-05-22T15:07:22Z

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

p5pRT · 2019-05-22T15:07:22Z

@khwilliamson - Status changed from 'pending release' to 'resolved'

p5pRT closed this as completed May 22, 2019

p5pRT added Severity Low labels Oct 19, 2019

toddr added this to the 5.30.0 milestone Oct 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BBC: blead breaks BKB/Lingua-JA-Moji-0.55.tar.gz #16895

BBC: blead breaks BKB/Lingua-JA-Moji-0.55.tar.gz #16895

p5pRT commented Mar 18, 2019

p5pRT commented Mar 18, 2019

p5pRT commented Mar 19, 2019

p5pRT commented Mar 19, 2019

p5pRT commented Mar 19, 2019

p5pRT commented Mar 24, 2019

p5pRT commented Mar 25, 2019

p5pRT commented Mar 25, 2019

p5pRT commented Mar 26, 2019

p5pRT commented Mar 28, 2019

p5pRT commented Mar 28, 2019

p5pRT commented May 22, 2019

p5pRT commented May 22, 2019

BBC: blead breaks BKB/Lingua-JA-Moji-0.55.tar.gz #16895

BBC: blead breaks BKB/Lingua-JA-Moji-0.55.tar.gz #16895

Comments

p5pRT commented Mar 18, 2019

p5pRT commented Mar 18, 2019

From @eserte

p5pRT commented Mar 19, 2019

From @jkeenan

p5pRT commented Mar 19, 2019

p5pRT commented Mar 19, 2019

From @khwilliamson

p5pRT commented Mar 24, 2019

From @khwilliamson

p5pRT commented Mar 25, 2019

From @khwilliamson

p5pRT commented Mar 25, 2019

p5pRT commented Mar 26, 2019

From @eserte

p5pRT commented Mar 28, 2019

From @khwilliamson

p5pRT commented Mar 28, 2019

From @jkeenan

p5pRT commented May 22, 2019

From @khwilliamson

p5pRT commented May 22, 2019