segfault (dead loop) with Encoding, use open :locale, print STDERR

[p5pRT]

Migrated from rt.perl.org#41442 (status was 'resolved')

Searchable as RT41442$

[p5pRT]

From aa29@mail.ru

To​: perlbug@​perl.org
Subject​: [PerlIO] Warning with char that does not map to STDERR encoding
leads to infinite cycle.
Reply-To​: aa29@​mail.ru
Message-Id​: <5.8.4_3176_1084980326@​INFORMED>

This is a bug report for perl from aa29@​aa29xp,
generated with the help of perlbug 1.35 running under perl v5.8.4.

PerlIO​: Warning with char that does not map to STDERR encoding leads
to infinite cycle.

binmode(STDERR, "​:encoding(cp866)");
warn "\302\253"; # no char in cp866 for this

Then Perl tries to encode to cp866 while flushing STDERR buffer, it
finds out that this char cannot be mapped. Default check-mode is
ENCODE_WARN_ON_ERR | ENCODE_PERLQQ, so it generates warning​:

ext\Encode\Encode.xs, line 153​:

if (check & ENCODE_WARN_ON_ERR){
  Perl_warner(aTHX_ packWARN(WARN_UTF8),
  ERR_ENCODE_NOMAP, (UV)ch, enc->name[0]);

But "\302\253" already is in the STDERR buffer, and Perl_warner just
adds ERR_ENCODE_NOMAP to this buffer and flushes, and then it comes
right to the same line, and so on.

---

Flags​:
  category=core
  severity=medium

---

Site configuration information for perl v5.8.4​:

Configured by aa29 at Mon May 17 17​:59​:46 2004.

Summary of my perl5 (revision 5 version 8 subversion 4) configuration​:
  Platform​:
  osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread
  uname=''
  config_args='undef'
  hint=recommended, useposix=true, d_sigaction=undef
  usethreads=undef use5005threads=undef useithreads=define
usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cl', ccflags
='-nologo -Gf -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE
_DES_FCRYPT -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL
_MSVCRT_READFIX',
  optimize='-MD -Zi -DNDEBUG -O1',
  cppflags='-DWIN32'
  ccversion='', gccversion='', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='link', ldflags
'-nologo -nodefaultlib -debug -opt​:ref,icf -libpath​:"c​:\perl\lib\CORE" -ma
chine​:x86'
  libpth=\lib
  libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib
uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib
msvcrt.lib
  perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib
uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib
msvcrt.lib
  libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib
  gnulibc_version='undef'
  Dynamic Linking​:
  dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ',
lddlflags='-dll -nologo -nodefaultlib -debug -opt​:ref,icf -libpath​:"c​:\perl
\lib\CORE" -machine​:x86'

Locally applied patches​:

---

@​INC for perl v5.8.4​:
  C​:/Perl/lib
  C​:/Perl/site/lib
  .

---

Environment for perl v5.8.4​:
  HOME (unset)
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=C​:\cygwin\bin;C​:\Tcl\bin;C​:\WINDOWS\system32;C​:\WINDOWS;C​:\WINDOWS\Syst
em32\Wbem;C​:\Perl\bin;C​:\Program Files\Support Tools;D​:\src\lib;C​:\Program
Files\Microsoft Visual Studio\Common\Tools\WinNT;C​:\Program Files\Microsoft
Visual Studio\Common\MSDev98\Bin;C​:\Program Files\Microsoft Visual
Studio\Common\Tools;C​:\Program Files\Microsoft Visual
Studio\VC98\bin;C​:\Arc;C​:\Program Files\Utils;C​:\Mysql\bin;C​:\Program
Files\Debugging Tools for Windows;C​:\Tcl\bin;D​:\Linda\XML\fop;C​:\Program
Files\GNU\WinCvs 1.2;D​:\src\bin;C​:\Program Files\Far
  PERL_BADLANG (unset)
  SHELL (unset)

aa29

[p5pRT]

From bulb@ucw.cz

Created by bulb@ucw.cz

Hello,

I have recently seen perl crashing with segfault for certain combinations of
locale and output. I have finally managed to narrow the problem down to the
following snippet (note, that it is utf-8 encoded and contains
a non-iso-8859-* character)​:

#!/usr/bin/perl
use utf8;
$string = "�\n";
binmode STDERR, "​:encoding(iso-8859-2)";
print STDERR $string;
__END__

This script crashes on 3 different systems. One is SuSE 9.0 with perl 5.8.1,
another is Debian Etch on i686 with perl 5.8.7 and this one which is Debian
Sid on amd64 with perl 5.8.7.

The script does NOT crash when STDERR is replaced by STDOUT, but rather
prints​:

"\x{8336}" does not map to iso-8859-2 at perlio-bug.pl line 9.
\x{8336}

as expected. Since the contents of the perlio layer stack might be important
here, these are dumps of default perlio stacs (with commands that produced
them)​:

bulb@​efreet​:~$ perl -MData​::Dumper -MPerlIO -e 'print Dumper(PerlIO​::get_layers(STDERR, details => 1))'
$VAR1 = 'unix';
$VAR2 = undef;
$VAR3 = 13177344;
$VAR4 = 'perlio';
$VAR5 = undef;
$VAR6 = 13177344;

bulb@​efreet​:~$ perl -MData​::Dumper -MPerlIO -e 'print Dumper(PerlIO​::get_layers(STDOUT, details => 1))'
$VAR1 = 'unix';
$VAR2 = undef;
$VAR3 = 13111808;
$VAR4 = 'perlio';
$VAR5 = undef;
$VAR6 = 13111808;

bulb@​efreet​:~$ perl -MData​::Dumper -MPerlIO -e 'binmode(STDERR, "​:encoding(iso-8859-2)"); print Dumper(PerlIO​::get_layers(STDERR, details => 1))'
$VAR1 = 'unix';
$VAR2 = undef;
$VAR3 = 13144576;
$VAR4 = 'perlio';
$VAR5 = undef;
$VAR6 = 13144576;
$VAR7 = 'encoding';
$VAR8 = 'iso-8859-2';
$VAR9 = 13144576;

bulb@​efreet​:~$ perl -MData​::Dumper -MPerlIO -e 'binmode(STDOUT, "​:encoding(iso-8859-2)"); print Dumper(PerlIO​::get_layers(STDOUT, details => 1))'
$VAR1 = 'unix';
$VAR2 = undef;
$VAR3 = 13144576;
$VAR4 = 'perlio';
$VAR5 = undef;
$VAR6 = 13144576;
$VAR7 = 'encoding';
$VAR8 = 'iso-8859-2';
$VAR9 = 13144576;

This is a distribution perl, which is stripped, so the backtrace won't be
exactly useful, but here it is (or rather part of it) anyway​:

(gdb) bt
#0 0x00002aaaab20c844 in realloc () from /lib/libc.so.6
#1 0x00002aaaaac3f68f in Perl_safesysrealloc () from /usr/lib/libperl.so.5.8
#2 0x00002aaaaac5ed8b in Perl_sv_grow () from /usr/lib/libperl.so.5.8
#3 0x00002aaaaac6c2ce in Perl_sv_catpvn_flags () from /usr/lib/libperl.so.5.8
#4 0x00002aaaaac69807 in Perl_sv_vcatpvfn () from /usr/lib/libperl.so.5.8
#5 0x00002aaaaac409f8 in Perl_vmess () from /usr/lib/libperl.so.5.8
#6 0x00002aaaaac41955 in Perl_vwarn () from /usr/lib/libperl.so.5.8
#7 0x00002aaaaac41fd3 in Perl_vwarner () from /usr/lib/libperl.so.5.8
#8 0x00002aaaaac42183 in Perl_warner () from /usr/lib/libperl.so.5.8
#9 0x00002aaaab8125cd in call_failure () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#10 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#11 0x00002aaaaac59625 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#12 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#13 0x00002aaaaabf90c8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#14 0x00002aaaab70d285 in PerlIOEncode_flush () from /usr/lib/perl/5.8/auto/PerlIO/encoding/encoding.so
#15 0x00002aaaaacbab5f in Perl_PerlIO_flush () from /usr/lib/libperl.so.5.8
#16 0x00002aaaaacbf0ed in PerlIOBuf_write () from /usr/lib/libperl.so.5.8
#17 0x00002aaaaac40d1e in Perl_write_to_stderr () from /usr/lib/libperl.so.5.8
#18 0x00002aaaaac41baf in Perl_vwarn () from /usr/lib/libperl.so.5.8
#19 0x00002aaaaac41fd3 in Perl_vwarner () from /usr/lib/libperl.so.5.8
#20 0x00002aaaaac42183 in Perl_warner () from /usr/lib/libperl.so.5.8
#21 0x00002aaaab8125cd in call_failure () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#22 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#23 0x00002aaaaac59625 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#24 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#25 0x00002aaaaabf90c8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#26 0x00002aaaab70d285 in PerlIOEncode_flush () from /usr/lib/perl/5.8/auto/PerlIO/encoding/encoding.so
#27 0x00002aaaaacbab5f in Perl_PerlIO_flush () from /usr/lib/libperl.so.5.8
#28 0x00002aaaaacbf0ed in PerlIOBuf_write () from /usr/lib/libperl.so.5.8
#29 0x00002aaaaac40d1e in Perl_write_to_stderr () from /usr/lib/libperl.so.5.8
#30 0x00002aaaaac41baf in Perl_vwarn () from /usr/lib/libperl.so.5.8
#31 0x00002aaaaac41fd3 in Perl_vwarner () from /usr/lib/libperl.so.5.8
#32 0x00002aaaaac42183 in Perl_warner () from /usr/lib/libperl.so.5.8
#33 0x00002aaaab8125cd in call_failure () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#34 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#35 0x00002aaaaac59625 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#36 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#37 0x00002aaaaabf90c8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#38 0x00002aaaab70d285 in PerlIOEncode_flush () from /usr/lib/perl/5.8/auto/PerlIO/encoding/encoding.so
#39 0x00002aaaaacbab5f in Perl_PerlIO_flush () from /usr/lib/libperl.so.5.8
#40 0x00002aaaaacbf0ed in PerlIOBuf_write () from /usr/lib/libperl.so.5.8
#41 0x00002aaaaac40d1e in Perl_write_to_stderr () from /usr/lib/libperl.so.5.8
#42 0x00002aaaaac41baf in Perl_vwarn () from /usr/lib/libperl.so.5.8
#43 0x00002aaaaac41fd3 in Perl_vwarner () from /usr/lib/libperl.so.5.8
#44 0x00002aaaaac42183 in Perl_warner () from /usr/lib/libperl.so.5.8
#45 0x00002aaaab8125cd in call_failure () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#46 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#47 0x00002aaaaac59625 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#48 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#49 0x00002aaaaabf90c8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#50 0x00002aaaab70d285 in PerlIOEncode_flush () from /usr/lib/perl/5.8/auto/PerlIO/encoding/encoding.so
#51 0x00002aaaaacbab5f in Perl_PerlIO_flush () from /usr/lib/libperl.so.5.8
#52 0x00002aaaaacbf0ed in PerlIOBuf_write () from /usr/lib/libperl.so.5.8
#53 0x00002aaaaac40d1e in Perl_write_to_stderr () from /usr/lib/libperl.so.5.8
#54 0x00002aaaaac41baf in Perl_vwarn () from /usr/lib/libperl.so.5.8
#55 0x00002aaaaac41fd3 in Perl_vwarner () from /usr/lib/libperl.so.5.8
#56 0x00002aaaaac42183 in Perl_warner () from /usr/lib/libperl.so.5.8
#57 0x00002aaaab8125cd in call_failure () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#58 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so

And this goes on for THOUSANDS of lines. It ends LOOOONG way down by​:

#66106 0x00002aaaab813a27 in XS_Encode__XS_encode () from /usr/lib/perl/5.8/auto/Encode/Encode.so
#66107 0x00002aaaaac59625 in Perl_pp_entersub () from /usr/lib/libperl.so.5.8
#66108 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#66109 0x00002aaaaabf90c8 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#66110 0x00002aaaab70d285 in PerlIOEncode_flush () from /usr/lib/perl/5.8/auto/PerlIO/encoding/encoding.so
#66111 0x00002aaaaacbab5f in Perl_PerlIO_flush () from /usr/lib/libperl.so.5.8
#66112 0x00002aaaaacbf073 in PerlIOBuf_write () from /usr/lib/libperl.so.5.8
#66113 0x00002aaaaaca2b4f in Perl_do_print () from /usr/lib/libperl.so.5.8
#66114 0x00002aaaaac534c8 in Perl_pp_print () from /usr/lib/libperl.so.5.8
#66115 0x00002aaaaac51b0e in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#66116 0x00002aaaaabf9dbb in perl_run () from /usr/lib/libperl.so.5.8
#66117 0x000000000040180c in main ()

Feel free to ask for more information.

Regards,

Jan Hudec <bulb@​ucw.cz>

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.8.7:

Configured by Debian Project at Tue Sep 13 02:36:46 CEST 2005.

Summary of my perl5 (revision 5 version 8 subversion 7) configuration:
  Platform:
    osname=linux, osvers=2.6.12-1-amd64-k8, archname=x86_64-linux-gnu-thread-multi
    uname='linux athlon.lowpingbastards.de 2.6.12-1-amd64-k8 #1 sun jul 17 15:04:25 cest 2005 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.7 -Dsitearch=/usr/local/lib/perl/5.8.7 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.7 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.0.2 20050821 (prerelease) (Debian 4.0.1-6)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.5.so, so=so, useshrplib=true, libperl=libperl.so.5.8.7
    gnulibc_version='2.3.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.7:
    /home/bulb/lib/perl/5.8.7
    /home/bulb/share/perl/5.8.7
    /etc/perl
    /usr/local/lib/perl/5.8.7
    /usr/local/share/perl/5.8.7
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.7:
    HOME=/home/bulb
    LANG (unset)
    LANGUAGE=en_GB:en_US:en:cs_CZ
    LC_CTYPE=cs_CZ.UTF-8
    LC_TIME=cs_CZ.UTF-8
    LD_LIBRARY_PATH=/home/bulb/lib/
    LOGDIR (unset)
    PATH=/home/bulb/bin:/home/bulb/shellstuff//bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
    PERL5LIB=/home/bulb/lib/perl/5.8.7:/home/bulb/share/perl/5.8.7
    PERL_BADLANG (unset)
    SHELL=/bin/zsh-beta

[p5pRT]

From bulb@ucw.cz

Hello Again,

I've played with the bug a little more and I include several more scripts,
that should help pinning down the bug.

First, let's get rid of the dependence on UTF-8 in the mail. This is the
original testcase, with escape sequence instead of the non-ascii character.
Also it now says :encoding(ascii) instead of iso-8859-2.

#!/usr/bin/perl
use utf8;
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

Now the next script does NOT crash. It works normally​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

As you can see, the only difference is the line​:

$SIG{__WARN__} = sub { };

So it shows, that the problem is caused by printing to STDERR from a warning
generated inside print to STDERR. The next script crashes again​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print STDERR "BUG\n"; };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

Now if the print is not to STDERR, it works. The following script does not
crash​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print "BUG\n"; };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

The most surprising is this script though. The next script replaces STDERR
with STDOUT​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print STDOUT "BUG\n"; };
$string = "\x{8336}\n";
binmode STDOUT, "​:encoding(ascii)";
print STDOUT $string;
__END__

Now this script does crash, but in a DIFFERENT WAY. Instead of immediately
dying of SIGSEGV, it prints the following​:

"\x{8336}" does not map to ascii at perlio-bug4.pl line 3.
\x{8336}
BUG
panic​: sv_setpvn called with negative strlen at perlio-bug4.pl line 6.

And exits with status 2. Note, that this was on a terminal, where STDOUT is
line-buffered (but not unbuffered).

--
  Jan 'Bulb' Hudec <bulb@​ucw.cz>

[p5pRT]

From @smpeters

[bulb@​ucw.cz - Fri Sep 23 10​:44​:07 2005]​:

Hello Again,

I've played with the bug a little more and I include several more scripts,
that should help pinning down the bug.

First, let's get rid of the dependence on UTF-8 in the mail. This is the
original testcase, with escape sequence instead of the non-ascii
character.
Also it now says :encoding(ascii) instead of iso-8859-2.

#!/usr/bin/perl
use utf8;
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

Now the next script does NOT crash. It works normally​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

As you can see, the only difference is the line​:

$SIG{__WARN__} = sub { };

So it shows, that the problem is caused by printing to STDERR from a
warning
generated inside print to STDERR. The next script crashes again​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print STDERR "BUG\n"; };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

Now if the print is not to STDERR, it works. The following script does not
crash​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print "BUG\n"; };
$string = "\x{8336}\n";
binmode STDERR, "​:encoding(ascii)";
print STDERR $string;
__END__

The most surprising is this script though. The next script replaces STDERR
with STDOUT​:

#!/usr/bin/perl
use utf8;
$SIG{__WARN__} = sub { print STDOUT "BUG\n"; };
$string = "\x{8336}\n";
binmode STDOUT, "​:encoding(ascii)";
print STDOUT $string;
__END__

Now this script does crash, but in a DIFFERENT WAY. Instead of immediately
dying of SIGSEGV, it prints the following​:

"\x{8336}" does not map to ascii at perlio-bug4.pl line 3.
\x{8336}
BUG
panic​: sv_setpvn called with negative strlen at perlio-bug4.pl line 6.

And exits with status 2. Note, that this was on a terminal, where
STDOUT is
line-buffered (but not unbuffered).

The code is essentially going into an infinite loop. The code discovers
the problem with the encoding and when attempting to flush the IO in
ext/PerlIO/encoding/encoding.xs calls Encode​::encode(), starting the
cycle all over again. Since Perl_warner() is not capable of handling
anything but char* data, is there a way to turn off encoding while it
warns, then turn it back on? A sort of "no utf-8;" within the core code
itself?

Here's the fatal end of the stack trace.

#0 0x080fc619 in Perl_sv_vcatpvfn (sv=0x933e5d8,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", patlen=30,
  args=0xbf184324, svargs=0x0, svmax=0, maybe_tainted=0x0) at sv.c​:8832
8832 bool has_utf8 = DO_UTF8(sv); /* has the result utf8? */
(gdb) bt
#0 0x080fc619 in Perl_sv_vcatpvfn (sv=0x933e5d8,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", patlen=30,
  args=0xbf184324, svargs=0x0, svmax=0, maybe_tainted=0x0) at sv.c​:8832
#1 0x080fc425 in Perl_sv_vsetpvfn (sv=0x933e5d8,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", patlen=30,
  args=0xbf184324, svargs=0x0, svmax=0, maybe_tainted=0x0) at sv.c​:8754
#2 0x080c7d3e in Perl_vmess (
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", args=0xbf184324)
  at util.c​:991
#3 0x080c8c89 in Perl_vwarn (
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", args=0xbf184324)
  at util.c​:1260
#4 0x080c943f in Perl_vwarner (err=44,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", args=0xbf184324)
  at util.c​:1377
#5 0x080c9199 in Perl_warner (err=44,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s") at util.c​:1350
#6 0xb7f65537 in encode_method (enc=0xb7c749d8, dir=0xb7c713e0,
  src=0x819e0c8, check=2306, offset=0x0, term=0x0, retcode=0x0)
  at Encode.xs​:162
#7 0xb7f6735b in XS_Encode__XS_encode (cv=0x821f548) at Encode.xs​:564
#8 0x080e8bbf in Perl_pp_entersub () at pp_hot.c​:2788
#9 0x080c5d55 in Perl_runops_debug () at dump.c​:1597
#10 0x08064d6b in S_call_body (myop=0xbf184640, is_eval=0 '\0') at
perl.c​:2617
#11 0x080645d7 in Perl_call_sv (sv=0x933e5b8, flags=64) at perl.c​:2515
#12 0x0806433a in Perl_call_method (methname=0xb7c7a5b0 "encode", flags=0)
  at perl.c​:2448
#13 0xb7c78e60 in PerlIOEncode_flush (f=0x81abd24) at encoding.xs​:420
#14 0x08164983 in Perl_PerlIO_flush (f=0x81abd24) at perlio.c​:1615
#15 0x081678c5 in PerlIOBuf_write (f=0x81abd24, vbuf=0x935ef00, count=61)
  at perlio.c​:3776
#16 0xb7c79bca in PerlIOEncode_write (f=0x81abd24, vbuf=0x935ef00, count=61)
  at encoding.xs​:584
#17 0x08164826 in Perl_PerlIO_write (f=0x81abd24, vbuf=0x935ef00, count=61)
  at perlio.c​:1592
#18 0x080c845c in Perl_write_to_stderr (
  message=0x935ef00 "\"\\x{00e8}\" does not map to iso-8859-2 at
rt_37234.pl line 6.\n", msgl en=61) at util.c​:1060
#19 0x080c9153 in Perl_vwarn (
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", args=0xbf184944)
  at util.c​:1300
#20 0x080c943f in Perl_vwarner (err=44,
  pat=0xb7f6a348 "\"\\x{%04lx}\" does not map to %s", args=0xbf184944)
  at util.c​:1377

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From pixel@mandriva.com

This is a bug report for perl from pixel@​mandriva.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

this segfaults if the locale charset is not utf8​:

use open '​:locale';
print STDERR "\x{201e}"; # „

---

Flags​:
  category=core
  severity=medium

---

Site configuration information for perl v5.8.8​:

Configured by Mandriva at Fri Sep 8 20​:00​:54 CEST 2006.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration​:
  Platform​:
  osname=linux, osvers=2.6.12-12mdksmp, archname=i386-linux
  uname='linux n4.mandriva.com 2.6.12-12mdksmp #1 smp fri sep 9 17​:43​:23 cest 2005 i686 intel(r) xeon(tm) cpu 2.80ghz gnulinux '
  config_args='-des -Dinc_version_list=5.8.7 5.8.7/i386-linux 5.8.6 5.8.6/i386-linux 5.8.5 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0 5.6.1 5.6.0 -Darchname=i386-linux -Dcc=gcc -Doptimize=-O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -march=i586 -mtune=pentiumpro -fasynchronous-unwind-tables -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr -Dsitebin=/usr/local/bin -Dsiteman1dir=/usr/local/share/man/man1 -Dsiteman3dir=/usr/local/share/man/man3 -Dman3ext=3pm -Dcf_by=Mandriva -Dmyhostname=localhost -Dperladmin=root@​localhost -Dcf_email=root@​localhost -Dd_dosuid -Ud_csh -Duseshrplib'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
  optimize='-O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -march=i586 -mtune=pentiumpro -fasynchronous-unwind-tables',
  cppflags='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm'
  ccversion='', gccversion='4.1.1 20060724 (prerelease) (4.1.1-3mdk)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
  libc=/lib/libc-2.4.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.4'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches​:
  Mandriva Linux patches

---

@​INC for perl v5.8.8​:
  /home/pixel/etc/perl
  /home/pixel/etc/perl/i386-linux
  /home/pixel/etc/perl/site_perl
  /usr/lib/perl5/5.8.8/i386-linux
  /usr/lib/perl5/5.8.8
  /usr/lib/perl5/site_perl/5.8.8/i386-linux
  /usr/lib/perl5/site_perl/5.8.8
  /usr/lib/perl5/site_perl
  /usr/lib/perl5/vendor_perl/5.8.8/i386-linux
  /usr/lib/perl5/vendor_perl/5.8.8
  /usr/lib/perl5/vendor_perl/5.8.7
  /usr/lib/perl5/vendor_perl/5.8.7/i386-linux
  /usr/lib/perl5/vendor_perl/5.8.6
  /usr/lib/perl5/vendor_perl/5.8.6/i386-linux
  /usr/lib/perl5/vendor_perl/5.8.5
  /usr/lib/perl5/vendor_perl/5.8.4
  /usr/lib/perl5/vendor_perl/5.8.3
  /usr/lib/perl5/vendor_perl/5.8.2
  /usr/lib/perl5/vendor_perl/5.8.1
  /usr/lib/perl5/vendor_perl
  .

---

Environment for perl v5.8.8​:
  HOME=/home/pixel
  LANG (unset)
  LANGUAGE (unset)
  LC_SOURCED=1
  LD_LIBRARY_PATH=/home/pixel/lib/i386​:/usr/X11R6/lib​:/usr/X11R6/lib/modules
  LOGDIR (unset)
  PATH=/home/pixel/bin​:/home/pixel/bin/oneliner​:/home/pixel/bin/perl​:/home/pixel/bin/ruby​:/home/pixel/bin/python​:/home/pixel/bin/tcl​:/home/pixel/bin/i386​:/usr/freeware/bin​:/usr/etc​:/usr/local/bin​:/usr/X11R6/bin​:/sbin​:/usr/sbin​:/sbin​:/usr/sbin​:/usr/X11R6/bin​:/bin​:/usr/bin​:/usr/local/bin​:/root/bin​:/root/bin​:/usr/X11R6/bin
  PERL5LIB=/home/pixel/etc/perl​:/home/pixel/etc/perl/i386-linux​:/home/pixel/etc/perl/site_perl
  PERL_BADLANG (unset)
  SHELL=/bin/bash

[p5pRT]

From @eserte

"pixel@​mandriva.com (via RT)" <perlbug-followup@​perl.org> writes​:

# New Ticket Created by pixel@​mandriva.com
# Please include the string​: [perl #41442]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=41442 >

This is a bug report for perl from pixel@​mandriva.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

this segfaults if the locale charset is not utf8​:

use open '​:locale';
print STDERR "\x{201e}"; # „

This seems to be an endless loop. This part is repeated over and over
again (tried with bleedperl)​:

#852 0x282f9fc8 in PerlIOEncode_flush (f=0x81c0014) at encoding.xs​:419
#853 0x0813b79c in Perl_PerlIO_flush (f=0x81c0014) at perlio.c​:1655
#854 0x0813eba8 in PerlIOBuf_write (f=0x81c0014, vbuf=0x188f25c8, count=39)
  at perlio.c​:3927
#855 0x282fb9de in PerlIOEncode_write (f=0x81c0014, vbuf=0x188f25c8, count=39)
  at encoding.xs​:582
#856 0x0813b64a in Perl_PerlIO_write (f=0x81c0014, vbuf=0x188f25c8, count=39)
  at perlio.c​:1631
#857 0x0809a7fe in Perl_write_to_stderr (
  message=0x188f25c8 "\"\\x{201e}\" does not map to iso-8859-1.\n",
  msglen=39) at util.c​:1211
#858 0x0809b16e in Perl_vwarn (
  pat=0x282f24a4 "\"\\x{%04lx}\" does not map to %s", args=0xbbc103c4)
  at util.c​:1431
---Type <return> to continue, or q <return> to quit---
#859 0x0809b44b in Perl_vwarner (err=44,
  pat=0x282f24a4 "\"\\x{%04lx}\" does not map to %s", args=0xbbc103c4)
  at util.c​:1508
#860 0x0809b1b2 in Perl_warner (err=44,
  pat=0x282f24a4 "\"\\x{%04lx}\" does not map to %s") at util.c​:1481
#861 0x282eb060 in encode_method (enc=0x282f3160, dir=0x282f30e0,
  src=0x81a71c8, check=2306, offset=0x0, term=0x0, retcode=0x0)
  at Encode.xs​:187
#862 0x282ee82a in XS_Encode__XS_encode (cv=0x825b6a8) at Encode.xs​:620
#863 0x080c13b3 in Perl_pp_entersub () at pp_hot.c​:2802
#864 0x0809859a in Perl_runops_debug () at dump.c​:1902
#865 0x080b252f in Perl_call_sv (sv=0x188fa128, flags=64) at perl.c​:2647
#866 0x080b22d3 in Perl_call_method (methname=0x282fcb08 "encode", flags=0)
  at perl.c​:2580

Regards,
  Slaven

---
Flags​:
category=core
severity=medium
---
Site configuration information for perl v5.8.8​:

Configured by Mandriva at Fri Sep 8 20​:00​:54 CEST 2006.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration​:
Platform​:
osname=linux, osvers=2.6.12-12mdksmp, archname=i386-linux
uname='linux n4.mandriva.com 2.6.12-12mdksmp #1 smp fri sep 9 17​:43​:23 cest 2005 i686 intel(r) xeon(tm) cpu 2.80ghz gnulinux '
config_args='-des -Dinc_version_list=5.8.7 5.8.7/i386-linux 5.8.6 5.8.6/i386-linux 5.8.5 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0 5.6.1 5.6.0 -Darchname=i386-linux -Dcc=gcc -Doptimize=-O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -march=i586 -mtune=pentiumpro -fasynchronous-unwind-tables -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr -Dsitebin=/usr/local/bin -Dsiteman1dir=/usr/local/share/man/man1 -Dsiteman3dir=/usr/local/share/man/man3 -Dman3ext=3pm -Dcf_by=Mandriva -Dmyhostname=localhost -Dperladmin=root@​localhost -Dcf_email=root@​localhost -Dd_dosuid -Ud_csh -Duseshrplib'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='gcc', ccflags ='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
optimize='-O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -march=i586 -mtune=pentiumpro -fasynchronous-unwind-tables',
cppflags='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='4.1.1 20060724 (prerelease) (4.1.1-3mdk)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries​:
ld='gcc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/libc-2.4.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.4'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux/CORE'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches​:
Mandriva Linux patches

---
@​INC for perl v5.8.8​:
/home/pixel/etc/perl
/home/pixel/etc/perl/i386-linux
/home/pixel/etc/perl/site_perl
/usr/lib/perl5/5.8.8/i386-linux
/usr/lib/perl5/5.8.8
/usr/lib/perl5/site_perl/5.8.8/i386-linux
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.8/i386-linux
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl/5.8.7
/usr/lib/perl5/vendor_perl/5.8.7/i386-linux
/usr/lib/perl5/vendor_perl/5.8.6
/usr/lib/perl5/vendor_perl/5.8.6/i386-linux
/usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl/5.8.4
/usr/lib/perl5/vendor_perl/5.8.3
/usr/lib/perl5/vendor_perl/5.8.2
/usr/lib/perl5/vendor_perl/5.8.1
/usr/lib/perl5/vendor_perl
.

---
Environment for perl v5.8.8​:
HOME=/home/pixel
LANG (unset)
LANGUAGE (unset)
LC_SOURCED=1
LD_LIBRARY_PATH=/home/pixel/lib/i386​:/usr/X11R6/lib​:/usr/X11R6/lib/modules
LOGDIR (unset)
PATH=/home/pixel/bin​:/home/pixel/bin/oneliner​:/home/pixel/bin/perl​:/home/pixel/bin/ruby​:/home/pixel/bin/python​:/home/pixel/bin/tcl​:/home/pixel/bin/i386​:/usr/freeware/bin​:/usr/etc​:/usr/local/bin​:/usr/X11R6/bin​:/sbin​:/usr/sbin​:/sbin​:/usr/sbin​:/usr/X11R6/bin​:/bin​:/usr/bin​:/usr/local/bin​:/root/bin​:/root/bin​:/usr/X11R6/bin
PERL5LIB=/home/pixel/etc/perl​:/home/pixel/etc/perl/i386-linux​:/home/pixel/etc/perl/site_perl
PERL_BADLANG (unset)
SHELL=/bin/bash

--
Slaven Rezic - slaven <at> rezic <dot> de
  babybike - routeplanner for cyclists in Berlin
  handheld (e.g. Compaq iPAQ with Linux) version of bbbike
  http​://bbbike.sourceforge.net

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @eserte

"pixel@​mandriva.com (via RT)" <perlbug-followup@​perl.org> writes​:

# New Ticket Created by pixel@​mandriva.com
# Please include the string​: [perl #41442]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=41442 >

This is a bug report for perl from pixel@​mandriva.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

this segfaults if the locale charset is not utf8​:

use open '​:locale';
print STDERR "\x{201e}"; # „

Attached is a patch which provides a workaround for the problem by
trapping recursive calls to the encode() method in
PerlIO/encoding/encoding.xs. However, I am far from sure that this is
solved properly.

[p5pRT]

From @eserte

perlio-encoding.patch

# 
# 
# To apply this patch:
# STEP 1: Chdir to the source directory.
# STEP 2: Run the 'applypatch' program with this patch file as input.
#
# If you do not have 'applypatch', it is part of the 'makepatch' package
# that you can fetch from the Comprehensive Perl Archive Network:
# http://www.perl.com/CPAN/authors/Johan_Vromans/makepatch-x.y.tar.gz
# In the above URL, 'x' should be 2 or higher.
#
# To apply this patch without the use of 'applypatch':
# STEP 1: Chdir to the source directory.
# STEP 2: Run the 'patch' program with this file as input.
#
#### End of Preamble ####

#### Patch data follows ####
diff -up 'bleedperl/ext/PerlIO/encoding/encoding.xs' 'bleedperl3/ext/PerlIO/encoding/encoding.xs'
Index: ./ext/PerlIO/encoding/encoding.xs
Prereq:  0.3 
--- ./ext/PerlIO/encoding/encoding.xs	Tue Jun 13 21:29:10 2006
+++ ./ext/PerlIO/encoding/encoding.xs	Mon Feb  5 22:41:59 2007
@@ -48,6 +48,7 @@ typedef struct {
     SV *enc;			/* the encoding object */
     SV *chk;                    /* CHECK in Encode methods */
     int flags;			/* Flags currently just needs lines */
+    int inEncodeCall;		/* trap recursive encode calls */
 } PerlIOEncode;
 
 #define NEEDS_LINES	1
@@ -147,6 +148,7 @@ PerlIOEncode_pushed(pTHX_ PerlIO * f, co
     }
 
     e->chk = newSVsv(get_sv("PerlIO::encoding::fallback", 0));
+    e->inEncodeCall = 0;
 
     FREETMPS;
     LEAVE;
@@ -404,6 +406,7 @@ PerlIOEncode_flush(pTHX_ PerlIO * f)
 	STRLEN len;
 	SSize_t count = 0;
 	if ((PerlIOBase(f)->flags & PERLIO_F_WRBUF) && (e->base.ptr > e->base.buf)) {
+	    if (e->inEncodeCall) return 0;
 	    /* Write case - encode the buffer and write() to layer below */
 	    PUSHSTACKi(PERLSI_MAGIC);
 	    SPAGAIN;
@@ -416,9 +419,12 @@ PerlIOEncode_flush(pTHX_ PerlIO * f)
 	    XPUSHs(e->bufsv);
 	    XPUSHs(e->chk);
 	    PUTBACK;
+	    e->inEncodeCall = 1;
 	    if (call_method("encode", G_SCALAR) != 1) {
+		e->inEncodeCall = 0;
 		Perl_die(aTHX_ "panic: encode did not return a value");
 	    }
+	    e->inEncodeCall = 0;
 	    SPAGAIN;
 	    str = POPs;
 	    PUTBACK;
@@ -453,6 +459,7 @@ PerlIOEncode_flush(pTHX_ PerlIO * f)
 	    }
 	    /* See if there is anything left in the buffer */
 	    if (e->base.ptr < e->base.end) {
+		if (e->inEncodeCall) return 0;
 		/* Bother - have unread data.
 		   re-encode and unread() to layer below
 		 */
@@ -472,9 +479,12 @@ PerlIOEncode_flush(pTHX_ PerlIO * f)
 		XPUSHs(str);
 		XPUSHs(e->chk);
 		PUTBACK;
+		e->inEncodeCall = 1;
 		if (call_method("encode", G_SCALAR) != 1) {
-		     Perl_die(aTHX_ "panic: encode did not return a value");
+		    e->inEncodeCall = 0;
+		    Perl_die(aTHX_ "panic: encode did not return a value");
 		}
+		e->inEncodeCall = 0;
 		SPAGAIN;
 		str = POPs;
 		PUTBACK;
#### End of Patch data ####

#### ApplyPatch data follows ####
# Data version        : 1.0
# Date generated      : Mon Feb  5 22:58:23 2007
# Generated by        : makepatch 2.03
# Recurse directories : Yes
# Excluded files      : (\A|/).*\~\Z
#                       (\A|/).*\.a\Z
#                       (\A|/).*\.bak\Z
#                       (\A|/).*\.BAK\Z
#                       (\A|/).*\.elc\Z
#                       (\A|/).*\.exe\Z
#                       (\A|/).*\.gz\Z
#                       (\A|/).*\.ln\Z
#                       (\A|/).*\.o\Z
#                       (\A|/).*\.obj\Z
#                       (\A|/).*\.olb\Z
#                       (\A|/).*\.old\Z
#                       (\A|/).*\.orig\Z
#                       (\A|/).*\.rej\Z
#                       (\A|/).*\.so\Z
#                       (\A|/).*\.Z\Z
#                       (\A|/)\.del\-.*\Z
#                       (\A|/)\.make\.state\Z
#                       (\A|/)\.nse_depinfo\Z
#                       (\A|/)core\Z
#                       (\A|/)tags\Z
#                       (\A|/)TAGS\Z
# v 'patchlevel.h' 4732 1160388693 33056
# p 'ext/PerlIO/encoding/encoding.xs' 16562 1170711719 0100440
#### End of ApplyPatch data ####

#### End of Patch kit [created: Mon Feb  5 22:58:23 2007] ####
#### Patch checksum: 99 3441 30793 ####
#### Checksum: 117 4065 16793 ####

[p5pRT]

From @eserte

--
Slaven Rezic - slaven <at> rezic <dot> de

  tkrevdiff - graphical display of diffs between revisions (RCS, CVS or SVN)
  http​://ptktools.sourceforge.net/#tkrevdiff

[p5pRT]

From @rgarcia

On 05 Feb 2007 23​:04​:07 +0100, Slaven Rezic <slaven@​rezic.de> wrote​:

use open '​:locale';
print STDERR "\x{201e}"; # „

Attached is a patch which provides a workaround for the problem by
trapping recursive calls to the encode() method in
PerlIO/encoding/encoding.xs. However, I am far from sure that this is
solved properly.

The patch might be interesting to prevent similar segfaults in the
future, but it doesn't fix the problem. I haven't looked at the code
yet, but it seems that Encode tries to print a warning, but STDERR
being already in use, somehow that clashes. Maybe disable re-encoding
when printing warnings ?

[p5pRT]

From @eserte

"Rafael Garcia-Suarez" <rgarciasuarez@​gmail.com> writes​:

On 05 Feb 2007 23​:04​:07 +0100, Slaven Rezic <slaven@​rezic.de> wrote​:

use open '​:locale';
print STDERR "\x{201e}"; # „

Attached is a patch which provides a workaround for the problem by
trapping recursive calls to the encode() method in
PerlIO/encoding/encoding.xs. However, I am far from sure that this is
solved properly.

The patch might be interesting to prevent similar segfaults in the
future, but it doesn't fix the problem. I haven't looked at the code
yet, but it seems that Encode tries to print a warning, but STDERR
being already in use, somehow that clashes. Maybe disable re-encoding
when printing warnings ?

Yes, this is the problem, and this is what the patch tries to fix. The
problematic path looks as following​: PerlIOEncode_flush is called
while there are some (utf-8) bytes in the e->bufsv buffer (which seems
to be the same content like e->base.buf, as it looks from a gdb
session). Then encode() is called with this buffer, detects that the
character in the buffer does not fit into iso-8859-1 (or whatever
target encoding is causing the problem) and issues a warning. This
warning will be written to STDERR (again!), meaning the warning string
is appended to the above mentioned buffer. Eventually
PerlIOEncode_flush is called again, still with the problem-causing
bytes at the beginning, which would cause another warning and so on,
until all memory is eaten up.

With the inEncodeCall flag I tried to prevent calling encode() again
recursively.

I am not sure if the patch is right, because
* Is it OK to add another element to the PerlIOEncode struct (because
  of binary compatibility or so?)=
* Is it guaranteed that the call to the encode() method will always
  return? Otherwise there could be a pending inEncodeCall flag never
  reset again.
* Are there legitimate cases where there could be recursive encode()
  calls?

Regards,
  Slaven

--
Slaven Rezic - slaven <at> rezic <dot> de

  tksm - Perl/Tk program for searching and replacing in multiple files
  http​://ptktools.sourceforge.net/#tksm

[p5pRT]

From @rgarcia

On 06 Feb 2007 21​:59​:53 +0100, Slaven Rezic <slaven@​rezic.de> wrote​:

Yes, this is the problem, and this is what the patch tries to fix. The
problematic path looks as following​: PerlIOEncode_flush is called
while there are some (utf-8) bytes in the e->bufsv buffer (which seems
to be the same content like e->base.buf, as it looks from a gdb
session). Then encode() is called with this buffer, detects that the
character in the buffer does not fit into iso-8859-1 (or whatever
target encoding is causing the problem) and issues a warning. This
warning will be written to STDERR (again!), meaning the warning string
is appended to the above mentioned buffer. Eventually
PerlIOEncode_flush is called again, still with the problem-causing
bytes at the beginning, which would cause another warning and so on,
until all memory is eaten up.

With the inEncodeCall flag I tried to prevent calling encode() again
recursively.

Thanks, applied as change #30213, adding also Pixel's test case. (and
bumping $VERSION)
Certainly not as elegant as having a fully reentrant thing, but works.

I am not sure if the patch is right, because
* Is it OK to add another element to the PerlIOEncode struct (because
of binary compatibility or so?)=

I don't see why it wouldn't be.

* Is it guaranteed that the call to the encode() method will always
return? Otherwise there could be a pending inEncodeCall flag never
reset again.
* Are there legitimate cases where there could be recursive encode()
calls?

I don't think so, since that wasn't designed for that apparently.

[p5pRT]

@rgs - Status changed from 'open' to 'resolved'