Apparent failure to localize %^H #13514

p5pRT · 2014-01-07T21:58:34Z

Migrated from rt.perl.org#120950 (status was 'resolved')

Searchable as RT120950$

p5pRT · 2014-01-07T21:58:34Z

From @khwilliamson

This is a bug report for perl from khw@karl.(none),
generated with the help of perlbug 1.39 running under perl 5.19.8.

Thanks to Matthew Horsfall for helping diagnose this

This program:

use Data::Dumper;
my $x;
#my $y = uc(chr(256));
BEGIN { $x = \%^H; }
print Dumper($x);

sub { tr/\x{345}/\x{370}/ }

prints out

$VAR1 = {
'reflags' => 0,
'reflags_charset' => 4
};

If the commented line is uncommented-out, it instead prints

$VAR1 = {};

What is happening is that $x points to the hints hash at compile time,
and tr/\x{100}/\x{200} happens at compile time, but after the BEGIN
block has executed. So $x continues to hold a reference and the tr//
modifies what it points to. What it points to is items from the 'use re
"/aa"' called by utf8_heavy.pl which is called by the tr///.

The 'uc', if included, causes utf8_heavy to run before the tr/// calls
it. And subsequent utf8_heavy calls don't exhibit the problem. Thus
there appears to be some initialization that is skipped on subsequent
utf8_heavy calls.

It seems to me that the hints hash is not getting localized properly.
(Matthew allows for the possibly that it is not a bug, but bizarre
action at a distance.)

Reordering the tests in B::Deparse.t causes this to show up. The order
they are currently in in blead masks the issue, just as the 'uc' in the
program above does.

Flags:
category=core
severity=medium

Site configuration information for perl 5.19.8:

Configured by khw at Tue Jan 7 07:43:12 MST 2014.

Summary of my perl5 (revision 5 version 19 subversion 8) configuration:
Commit id: 6efa4e8
Platform:
osname=linux, osvers=2.6.35-32-generic-pae,
archname=i686-linux-thread-multi-64int-ld
uname='linux karl 2.6.35-32-generic-pae #67-ubuntu smp mon mar 5
21:23:19 utc 2012 i686 gnulinux '
config_args='-des -Dprefix=/home/khw/blead -Dusedevel
-D'optimize=-ggdb3' -A'optimize=-ggdb3' -A'optimize=-O0'
-Accflags='-DPERL_BOOL_AS_CHAR' -Dman1dir=none -Dman3dir=none
-DDEBUGGING -Dcc=g++ -Dusemorebits -Dusethreads'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=undef, uselongdouble=define
usemymalloc=n, bincompat5005=undef
Compiler:
cc='g++', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_BOOL_AS_CHAR
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize=' -ggdb3 -O0',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_BOOL_AS_CHAR
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include'
ccversion='', gccversion='4.4.5', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='long double', nvsize=12,
Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='g++', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/../lib /usr/lib/../lib /lib /usr/lib
/usr/lib/i686-linux-gnu
libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.12.1.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.12'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -ggdb3 -ggdb3 -O0
-L/usr/local/lib -fstack-protector'

@INC for perl 5.19.8:

/home/khw/blead/lib/perl5/site_perl/5.19.8/i686-linux-thread-multi-64int-ld
/home/khw/blead/lib/perl5/site_perl/5.19.8
/home/khw/blead/lib/perl5/5.19.8/i686-linux-thread-multi-64int-ld
/home/khw/blead/lib/perl5/5.19.8
/home/khw/blead/lib/perl5/site_perl
.

Environment for perl 5.19.8:
HOME=/home/khw
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LD_LIBRARY_PATH (unset)
LOGDIR (unset)

PATH=/home/khw/bin:/home/khw/perl5/perlbrew/bin:/home/khw/print/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/usr/games:/home/khw/cxoffice/bin
PERL5OPT=-w
PERL_BADLANG (unset)
PERL_POD_PEDANTIC=1
SHELL=/bin/ksh

p5pRT · 2014-01-19T05:46:04Z

From @cpansprout

On Tue Jan 07 13:58:34 2014, public@khwilliamson.com wrote:

It seems to me that the hints hash is not getting localized properly.
(Matthew allows for the possibly that it is not a bug, but bizarre
action at a distance.)

Knowing how %^H works, I think what you see is actually correct behaviour. %^H is not localised unless the HINT_LOCALIZE_HH bit is set in $^H. Unlike local(), this localisation is different in that exiting to an outer compilation scope that did not localise %^H causes the outer scope to get a new %^H, so it appears that what was originally the outer scope’s %^H is ‘stolen’ by the inner scope that localises it.

Reordering the tests in B::Deparse.t causes this to show up. The
order
they are currently in in blead masks the issue, just as the 'uc' in
the
program above does.

I think the bug is in B::Deparse. It should not be deparsing %^H unless the hint in $^H is set.

--

Father Chrysostomos

p5pRT · 2014-01-19T05:46:04Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2015-04-03T21:31:38Z

From @iabyn

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via RT wrote:

On Tue Jan 07 13:58:34 2014, public@khwilliamson.com wrote:
This program:
 use Data&#8203;::Dumper;
 my $x;
 \#my $y = uc$chr\(256$\);
 BEGIN \{ $x = \\%^H; \}
 print Dumper$$x$;

 sub \{ tr/\\x\{345\}/\\x\{370\}/ \}
prints out
 $VAR1 = \{
           'reflags' => 0\,
           'reflags\_charset' => 4
 \};
If the commented line is uncommented-out, it instead prints
 $VAR1 = \{\};
It seems to me that the hints hash is not getting localized properly.
(Matthew allows for the possibly that it is not a bug, but bizarre
action at a distance.)
Knowing how %^H works, I think what you see is actually correct
behaviour. %^H is not localised unless the HINT_LOCALIZE_HH bit is set
in $^H. Unlike local(), this localisation is different in that exiting
to an outer compilation scope that did not localise %^H causes the outer
scope to get a new %^H, so it appears that what was originally the outer
scope’s %^H is ‘stolen’ by the inner scope that localises it.

Reordering the tests in B::Deparse.t causes this to show up. The
order they are currently in in blead masks the issue, just as the 'uc'
in the program above does.

I think the bug is in B::Deparse. It should not be deparsing %^H unless
the hint in $^H is set.

On the other hand, setting HINT_LOCALIZE_HH in Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.

Opinions?

I looked at this issue to try and fix the Deparse.t TODO test accidentally
passing in smokes (it passes if PERL_UNICODE="").

Inline Patch

diff --git a/regcomp.c b/regcomp.c
index 5d5332d..aa532c3 100644
--- a/regcomp.c
+++ b/regcomp.c
@@ -17749,6 +17749,7 @@ Perl_save_re_context(pTHX)
                 save_scalar(gv);
         }
     }
+    PL_hints |= HINT_LOCALIZE_HH;
 }
 #endif
 


--

"Procrastination grows to fill the available time"
-- Mitchell's corollary to Parkinson's Law

p5pRT · 2015-04-04T15:47:41Z

From @khwilliamson

On 04/03/2015 03:31 PM, Dave Mitchell wrote:

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via RT wrote:
On Tue Jan 07 13:58:34 2014, public@khwilliamson.com wrote:
This program:
  use Data&#8203;::Dumper;
  my $x;
  \#my $y = uc$chr\(256$\);
  BEGIN \{ $x = \\%^H; \}
  print Dumper$$x$;

  sub \{ tr/\\x\{345\}/\\x\{370\}/ \}
prints out
  $VAR1 = \{
            'reflags' => 0\,
            'reflags\_charset' => 4
  \};
If the commented line is uncommented-out, it instead prints
  $VAR1 = \{\};
It seems to me that the hints hash is not getting localized properly.
(Matthew allows for the possibly that it is not a bug, but bizarre
action at a distance.)
Knowing how %^H works, I think what you see is actually correct
behaviour. %^H is not localised unless the HINT_LOCALIZE_HH bit is set
in $^H. Unlike local(), this localisation is different in that exiting
to an outer compilation scope that did not localise %^H causes the outer
scope to get a new %^H, so it appears that what was originally the outer
scope’s %^H is ‘stolen’ by the inner scope that localises it.

Reordering the tests in B::Deparse.t causes this to show up. The
order they are currently in in blead masks the issue, just as the 'uc'
in the program above does.

I think the bug is in B::Deparse. It should not be deparsing %^H unless
the hint in $^H is set.
On the other hand, setting HINT_LOCALIZE_HH in Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.

Opinions?

Not knowing the nuances, it sounds like a good idea to me.

I looked at this issue to try and fix the Deparse.t TODO test accidentally
passing in smokes (it passes if PERL_UNICODE="").

diff --git a/regcomp.c b/regcomp.c
index 5d5332d..aa532c3 100644
--- a/regcomp.c
+++ b/regcomp.c
@@ -17749,6 +17749,7 @@ Perl_save_re_context(pTHX)
save_scalar(gv);
}
}
+ PL_hints |= HINT_LOCALIZE_HH;
}
#endif

p5pRT · 2015-04-07T21:42:49Z

From @cpansprout

On Fri Apr 03 14:31:38 2015, davem wrote:

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via RT wrote:
On Tue Jan 07 13:58:34 2014, public@khwilliamson.com wrote:
This program:
 use Data&#8203;::Dumper;
 my $x;
 \#my $y = uc$chr\(256$\);
 BEGIN \{ $x = \\%^H; \}
 print Dumper$$x$;

 sub \{ tr/\\x\{345\}/\\x\{370\}/ \}
prints out
 $VAR1 = \{
           'reflags' => 0\,
           'reflags\_charset' => 4
 \};
If the commented line is uncommented-out, it instead prints
 $VAR1 = \{\};
It seems to me that the hints hash is not getting localized properly.
(Matthew allows for the possibly that it is not a bug, but bizarre
action at a distance.)
Knowing how %^H works, I think what you see is actually correct
behaviour. %^H is not localised unless the HINT_LOCALIZE_HH bit is set
in $^H. Unlike local(), this localisation is different in that exiting
to an outer compilation scope that did not localise %^H causes the outer
scope to get a new %^H, so it appears that what was originally the outer
scope’s %^H is ‘stolen’ by the inner scope that localises it.

Reordering the tests in B::Deparse.t causes this to show up. The
order they are currently in in blead masks the issue, just as the 'uc'
in the program above does.

I think the bug is in B::Deparse. It should not be deparsing %^H unless
the hint in $^H is set.
On the other hand, setting HINT_LOCALIZE_HH in Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.

Opinions?

I have not looked into this deeply, but I suspect that will simply cover up the problem, which it may be possible to trigger another way. I do think the bug is in B::Deparse.

I looked at this issue to try and fix the Deparse.t TODO test accidentally
passing in smokes (it passes if PERL_UNICODE="").

diff --git a/regcomp.c b/regcomp.c
index 5d5332d..aa532c3 100644
--- a/regcomp.c
+++ b/regcomp.c
@@ -17749,6 +17749,7 @@ Perl_save_re_context(pTHX)
save_scalar(gv);
}
}
+ PL_hints |= HINT_LOCALIZE_HH;
}
#endif

--

Father Chrysostomos

p5pRT · 2015-04-08T15:47:23Z

From @iabyn

On Tue, Apr 07, 2015 at 02:42:49PM -0700, Father Chrysostomos via RT wrote:

On Fri Apr 03 14:31:38 2015, davem wrote:

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via RT wrote:

I think the bug is in B::Deparse. It should not be deparsing %^H unless
the hint in $^H is set.

On the other hand, setting HINT_LOCALIZE_HH in Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.
I have not looked into this deeply, but I suspect that will simply cover
up the problem, which it may be possible to trigger another way. I do
think the bug is in B::Deparse.

Just to be clear - what particular flag in $^H do you feel should be set
in order for Deparse to not skip processing the contents of %^H?

--
But Pity stayed his hand. "It's a pity I've run out of bullets",
he thought. -- "Bored of the Rings"

p5pRT · 2015-12-10T02:33:00Z

From @jkeenan

On Wed Apr 08 08:47:23 2015, davem wrote:

On Tue, Apr 07, 2015 at 02:42:49PM -0700, Father Chrysostomos via RT
wrote:

On Fri Apr 03 14:31:38 2015, davem wrote:

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via
RT wrote:

I think the bug is in B::Deparse. It should not be deparsing %^H
unless
the hint in $^H is set.

On the other hand, setting HINT_LOCALIZE_HH in
Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.
I have not looked into this deeply, but I suspect that will simply
cover
up the problem, which it may be possible to trigger another way. I
do
think the bug is in B::Deparse.

Just to be clear - what particular flag in $^H do you feel should be
set
in order for Deparse to not skip processing the contents of %^H?

Father C, Dave, khw: Do we know how we should move forward on this ticket?

Thank you very much.

--
James E Keenan (jkeenan@cpan.org)

p5pRT · 2017-02-10T04:01:38Z

From @khwilliamson

On Wed, 09 Dec 2015 18:33:00 -0800, jkeenan wrote:

On Wed Apr 08 08:47:23 2015, davem wrote:

On Tue, Apr 07, 2015 at 02:42:49PM -0700, Father Chrysostomos via RT
wrote:

On Fri Apr 03 14:31:38 2015, davem wrote:

On Sat, Jan 18, 2014 at 09:46:04PM -0800, Father Chrysostomos via
RT wrote:

I think the bug is in B::Deparse. It should not be deparsing %^H
unless
the hint in $^H is set.

On the other hand, setting HINT_LOCALIZE_HH in
Perl_save_re_context()
fixes the above issue, and the TODO test in Deparse.t.
I have not looked into this deeply, but I suspect that will simply
cover
up the problem, which it may be possible to trigger another way. I
do
think the bug is in B::Deparse.

Just to be clear - what particular flag in $^H do you feel should be
set
in order for Deparse to not skip processing the contents of %^H?

Father C, Dave, khw: Do we know how we should move forward on this ticket?

Thank you very much.

I don't know how to move forward on this, and no one else has responded in over a year. How about now?
--
Karl Williamson

p5pRT · 2019-03-28T14:23:03Z

From @khwilliamson

It turns out that this was fixed by
commit 8946fcd
Author: Karl Williamson <khw@cpan.org>
Date: Fri Mar 23 13:43:56 2018 -0600

Move UTF-8 case changing data into core
--
Karl Williamson

p5pRT · 2019-03-28T14:23:04Z

@khwilliamson - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Mar 28, 2019

p5pRT added the Severity Low label Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Apparent failure to localize %^H #13514

Apparent failure to localize %^H #13514

p5pRT commented Jan 7, 2014

p5pRT commented Jan 7, 2014

Uh oh!

p5pRT commented Jan 19, 2014

Uh oh!

p5pRT commented Jan 19, 2014

Uh oh!

p5pRT commented Apr 3, 2015

Uh oh!

p5pRT commented Apr 4, 2015

Uh oh!

p5pRT commented Apr 7, 2015

Uh oh!

p5pRT commented Apr 8, 2015

Uh oh!

p5pRT commented Dec 10, 2015

Uh oh!

p5pRT commented Feb 10, 2017

Uh oh!

p5pRT commented Mar 28, 2019

Uh oh!

p5pRT commented Mar 28, 2019

Uh oh!

Apparent failure to localize %^H #13514

Apparent failure to localize %^H #13514

Comments

p5pRT commented Jan 7, 2014

p5pRT commented Jan 7, 2014

From @khwilliamson

Uh oh!

p5pRT commented Jan 19, 2014

From @cpansprout

Uh oh!

p5pRT commented Jan 19, 2014

Uh oh!

p5pRT commented Apr 3, 2015

From @iabyn

Uh oh!

p5pRT commented Apr 4, 2015

From @khwilliamson

Uh oh!

p5pRT commented Apr 7, 2015

From @cpansprout

Uh oh!

p5pRT commented Apr 8, 2015

From @iabyn

Uh oh!

p5pRT commented Dec 10, 2015

From @jkeenan

Uh oh!

p5pRT commented Feb 10, 2017

From @khwilliamson

Uh oh!

p5pRT commented Mar 28, 2019

From @khwilliamson

Uh oh!

p5pRT commented Mar 28, 2019

Uh oh!