foreach spuriously autovivifies

[p5pRT]

Migrated from rt.perl.org#2166 (status was 'open')

Searchable as RT2166$

[p5pRT]

From [email protected]

  foreach (@​h{a, b}) { }

autovivifies $h{a} and $h{b}.

Yes, I understand why it happens.

Mike Guy

% perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration​:
  Platform​:
  osname=sunos, osvers=4.1.3, archname=sun4-sunos
  uname=''
  hint=previous, useposix=true, d_sigaction=define
  usethreads=undef useperlio=undef d_sfio=undef
  Compiler​:
  cc='gcc', optimize='-O', gccversion=2.7.2.3
  cppflags='-I/usr/local/include -DREG_INFTY=22790'
  ccflags ='-I/usr/local/include -DREG_INFTY=22790'
  stdchar='unsigned char', d_stdstdio=define, usevfork=true
  intsize=4, longsize=4, ptrsize=4, doublesize=8
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
  alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries​:
  ld='ld', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib /usr/ucblib
  libs=-ldbm -ldl -lm -lc -lposix
  libc=/lib/libc.so.1.8, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
  cccdlflags='-fpic', lddlflags='-assert nodefinitions -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Locally applied patches​:
  pos.list
  IO.file.return
  int.delimiter
  bigfloat
  Built under sunos
  Compiled at Jul 7 1999 16​:09​:54
  @​INC​:
  /home/mjtg/perl5.005_03/lib
  /home/mjtg/perl5.005_03/lib
  /home/mjtg/perl5.005_03/lib
  .

[p5pRT]

From @schwern

[mjtg <!--c--> <i>at</i> <!--a--> cus.cam.ac.uk - Thu Feb 17 18​:57​:54 2000]​:

       foreach \(@​h\{a\, b\}\) \{ \}

autovivifies $h{a} and $h{b}.

Issue still exists in 5.8.1 RC 2

[p5pRT]

From [email protected]

Is it really a bug? I would expect these elements to be autovivified...

[p5pRT]

From @smpeters

On Sun, Jun 15, 2008 at 6​:02 AM, reneeb via RT
<perlbug-followup@​perl.org> wrote​:

Is it really a bug? I would expect these elements to be autovivified...

Unfortunately, it has pretty much agreed that it is a bug, but due to
the regular use of this bug as a feature, it cannot be fixed. See
http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2006-02/msg00823.html
for additional examples of this bug.

Steve Peters
steve@​fisharerojo.org

[p5pRT]

From @davidnicol

On 6/20/08, Steve Peters <steve@​fisharerojo.org> wrote​:

Unfortunately, it has pretty much agreed that it is a bug, but due to

but not universally. The opposing POV holds that the non-autoviv of
nonexistent hash elements used as subroutine arguments was misguided
due to the fact that it greatly confuses the issues and creates false
expectations and is therefore the aspect (more general term used to
avoid either "feature" or "bug") which should be deprecated and backed
out, reaplced with instructions to doublequote your hash element
look-ups if you want to pass them to subroutines without having them
autovivified.

  my %empty;
  some_function($empty{hooha}); # once autovivved, doesn't now, until
$_[0] assigned to
  some_function("$empty{hooha}"); # never autovivved, still doesn't

[p5pRT]

From @iabyn

On Fri, Jun 20, 2008 at 01​:03​:31PM -0500, David Nicol wrote​:

reaplced with instructions to doublequote your hash element
look-ups if you want to pass them to subroutines without having them
autovivified.

my %empty;
some_function($empty{hooha}); # once autovivved, doesn't now, until
$_[0] assigned to
some_function("$empty{hooha}"); # never autovivved, still doesn't

Which would be pretty unfortunate if $empty{hooha} was a ref.

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

[p5pRT]

From [email protected]

In article <20080622132012.GM12186@​iabyn.com>, davem@​iabyn.com (Dave Mitchell)
wrote​:

On Fri, Jun 20, 2008 at 01​:03​:31PM -0500, David Nicol wrote​:

reaplced with instructions to doublequote your hash element
look-ups if you want to pass them to subroutines without having them
autovivified.

my %empty;
some_function($empty{hooha}); # once autovivved, doesn't now, until
$_[0] assigned to
some_function("$empty{hooha}"); # never autovivved, still doesn't

Which would be pretty unfortunate if $empty{hooha} was a ref.

Which is why you use do-BLOCK instead​:

  some_function(do { $empty{hooha}; });
  # never autovivifies, preserves references

I don't think this behaviour is documented anywhere, though.

/Bo Lindbergh

[p5pRT]

From @ysth

On Sun, June 22, 2008 12​:25 pm, Bo Lindbergh wrote​:

In article <20080622132012.GM12186@​iabyn.com>, davem@​iabyn.com (Dave
Mitchell) wrote​:

On Fri, Jun 20, 2008 at 01​:03​:31PM -0500, David Nicol wrote​:

reaplced with instructions to doublequote your hash element look-ups
if you want to pass them to subroutines without having them
autovivified.

my %empty; some_function($empty{hooha}); # once autovivved, doesn't
now, until $_[0] assigned to
some_function("$empty{hooha}"); # never autovivved, still doesn't

Which would be pretty unfortunate if $empty{hooha} was a ref.

Which is why you use do-BLOCK instead​:

some_function(do { $empty{hooha}; }); # never autovivifies, preserves
references

Or some_function(scalar $empty{hooha})

[p5pRT]

From [email protected]

Created by [email protected]

  This code​:

  my %a;
  die "Bad 1\n" if exists $a{'FOO'};
  foreach my $b ( $a{'FOO'}, $a{'BAR'} ) {}
  die "Bad 2\n" if exists $a{'FOO'};

  will die with "Bad 2". I don't believe the aliasing aspect
  of the "foreach" loop should actually autovivify hash elements...

  This is particularly bad for tied hashes​:

  foreach my $x ( $ENV{'TMPDIR'}, $ENV{'TMP'}, ... ) {}

  That will actually cause environment variables to spring to life.
  Aliasing should not have this side-effect.

  It is not good enough to create the element and delete it later;
  no element should be created at any point in this code.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl v5.8.8:

Configured by Mandriva at Fri May  9 03:51:40 EDT 2008.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.24.4-server-1mnb, archname=i386-linux
    uname='linux titan.mandriva.com 2.6.24.4-server-1mnb #1 smp thu mar 27
15:24:36 cet 2008 i686 intel(r) xeon(r) cpu x3210 @ 2.13ghz gnulinux '
    config_args='-des -Dinc_version_list=5.8.7 5.8.7/i386-linux 5.8.6
5.8.6/i386-linux 5.8.5 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0 5.6.1 5.6.0
-Darchname=i386-linux -Dcc=gcc -Doptim$
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-fno-strict-aliasing -pipe
-Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fomit-frame-pointer -march=i586 -mtune=generic
-fasynchronous-unwind-tables',
    cppflags='-fno-strict-aliasing -pipe -Wdeclaration-after-statement
-I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='4.1.2 20070302 (prerelease)
(4.1.2-1mdv2007.1)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.4.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    Mandriva Linux patches


@INC for perl v5.8.8:
    /usr/lib/perl5/5.8.8/i386-linux
    /usr/lib/perl5/5.8.8
    /usr/lib/perl5/site_perl/5.8.8/i386-linux
    /usr/lib/perl5/site_perl/5.8.8
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.8/i386-linux
    /usr/lib/perl5/vendor_perl/5.8.8
    /usr/lib/perl5/vendor_perl/5.8.7
    /usr/lib/perl5/vendor_perl/5.8.7/i386-linux
    /usr/lib/perl5/vendor_perl/5.8.6
    /usr/lib/perl5/vendor_perl
    .


Environment for perl v5.8.8:
    HOME=/home/jj
    LANG=C
    LANGUAGE=en_US:en
    LC_ADDRESS=C
    LC_COLLATE=C
    LC_CTYPE=C
    LC_IDENTIFICATION=C
    LC_MEASUREMENT=C
    LC_MESSAGES=C
    LC_MONETARY=C
    LC_NAME=C
    LC_NUMERIC=C
    LC_PAPER=C
    LC_SOURCED=1
    LC_TELEPHONE=C
    LC_TIME=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/bin:/usr/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @demerphq

2009/3/26 Jason Noakes <perlbug-followup@​perl.org>​:

# New Ticket Created by  Jason Noakes
# Please include the string​:  [perl #64228]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=64228 >

This is a bug report for perl from jjnoakes@​gmail.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

-----------------------------------------------------------------
[Please enter your report here]

 This code​:

   my %a;
   die "Bad 1\n" if exists $a{'FOO'};
   foreach my $b ( $a{'FOO'}, $a{'BAR'} ) {}
   die "Bad 2\n" if exists $a{'FOO'};

 will die with "Bad 2". I don't believe the aliasing aspect
 of the "foreach" loop should actually autovivify hash elements...

 This is particularly bad for tied hashes​:

   foreach my $x ( $ENV{'TMPDIR'}, $ENV{'TMP'}, ... ) {}

 That will actually cause environment variables to spring to life.
 Aliasing should not have this side-effect.

 It is not good enough to create the element and delete it later;
 no element should be created at any point in this code.

No, thats wrong. Aliasing SHOULD have this effect.

Or else this wouldn't work​:

perl -e'my %x=(a=>1); $_='muhahaha' for $x{b}; use Data​::Dumper;
print Dumper(\%x)'
$VAR1 = {
  'a' => 1,
  'b' => 'muhahaha'
  };

Its irritating i agree, but the list provided to foreach() is in
LVALUE context because of the aliasing, and thus it autovivifies the
slot.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @davidnicol

On Fri, Mar 27, 2009 at 12​:23 PM, demerphq <demerphq@​gmail.com> wrote​:

No, thats wrong. Aliasing SHOULD have this effect.

Or else this wouldn't work​:

 perl -e'my %x=(a=>1); $_='muhahaha' for $x{b}; use Data​::Dumper;
print Dumper(\%x)'
$VAR1 = {
         'a' => 1,
         'b' => 'muhahaha'
       };

Its irritating i agree, but the list provided to foreach() is in
LVALUE context because of the aliasing, and thus it autovivifies the
slot.

Yves

Yet, in Mr. Noakes' defense,

perl -e'my %x=(a=>1); sub xy {$_[1]="muhahaha" }; xy $x{b},
$x{bwahaha}; use Data​::Dumper; print Dumper(\%x)'
$VAR1 = {
  'bwahaha' => 'muhahaha',
  'a' => 1
  };

does not autoviv $x{b}, demonstrating non-autivivving aliasing and
creating a reasonable expectation that for aliases won't autoviv
except when actually used as lvalues, just like arg aliases don't.

[p5pRT]

From @smpeters

On Fri, Mar 27, 2009 at 1​:52 PM, David Nicol <davidnicol@​gmail.com> wrote​:

On Fri, Mar 27, 2009 at 12​:23 PM, demerphq <demerphq@​gmail.com> wrote​:

No, thats wrong. Aliasing SHOULD have this effect.

Or else this wouldn't work​:

 perl -e'my %x=(a=>1); $_='muhahaha' for $x{b}; use Data​::Dumper;
print Dumper(\%x)'
$VAR1 = {
         'a' => 1,
         'b' => 'muhahaha'
       };

Its irritating i agree, but the list provided to foreach() is in
LVALUE context because of the aliasing, and thus it autovivifies the
slot.

Yves

Yet, in Mr. Noakes' defense,

 perl -e'my %x=(a=>1); sub xy {$_[1]="muhahaha" }; xy $x{b},
$x{bwahaha}; use Data​::Dumper; print Dumper(\%x)'
$VAR1 = {
         'bwahaha' => 'muhahaha',
         'a' => 1
       };

does not autoviv $x{b}, demonstrating non-autivivving aliasing and
creating a reasonable expectation that for aliases won't autoviv
except when actually used as lvalues, just like arg aliases don't.

This appears to be another case of "RT #2166​: foreach spuriously
autovivifies". It is widely considered to be a bug, but is also
regularly used as a feature. It is well documented, but is unlikely
to ever be fixed due to its extensive use.

Steve Peters
steve@​fisharerojo.org

[p5pRT]

From @gbarr

On Mar 27, 2009, at 1​:52 PM, David Nicol wrote​:

On Fri, Mar 27, 2009 at 12​:23 PM, demerphq <demerphq@​gmail.com> wrote​:

No, thats wrong. Aliasing SHOULD have this effect.

Or else this wouldn't work​:

perl -e'my %x=(a=>1); $_='muhahaha' for $x{b}; use Data​::Dumper;
print Dumper(\%x)'
$VAR1 = {
'a' => 1,
'b' => 'muhahaha'
};

Its irritating i agree, but the list provided to foreach() is in
LVALUE context because of the aliasing, and thus it autovivifies the
slot.

Yves

Yet, in Mr. Noakes' defense,

perl -e'my %x=(a=>1); sub xy {$_[1]="muhahaha" }; xy $x{b},
$x{bwahaha}; use Data​::Dumper; print Dumper(\%x)'
$VAR1 = {
'bwahaha' => 'muhahaha',
'a' => 1
};

does not autoviv $x{b}, demonstrating non-autivivving aliasing and
creating a reasonable expectation that for aliases won't autoviv
except when actually used as lvalues, just like arg aliases don't.

subs did at one time also autoviv but it was changed during 5.005
development (I think)

There are many places that auto-viv due to being in LVALUE (map and
grep for
example), but sub args were special cased out (I cannot remember why)

IMO, the fact that subs do not autoviv is the bug as they are the
exception

In the past I have used reverse to avoid the autoviv as it does not
place its
arguments in an LVALUE context

Graham.

[p5pRT]

From @rgs

2009/3/31 Graham Barr <gbarr@​pobox.com>​:

On Mar 27, 2009, at 1​:52 PM, David Nicol wrote​:

On Fri, Mar 27, 2009 at 12​:23 PM, demerphq <demerphq@​gmail.com> wrote​:

No, thats wrong. Aliasing SHOULD have this effect.

Or else this wouldn't work​:

 perl -e'my %x=(a=>1); $_='muhahaha' for $x{b}; use Data​::Dumper;
print Dumper(\%x)'
$VAR1 = {
        'a' => 1,
        'b' => 'muhahaha'
      };

Its irritating i agree, but the list provided to foreach() is in
LVALUE context because of the aliasing, and thus it autovivifies the
slot.

Yves

Yet, in Mr. Noakes' defense,

perl -e'my %x=(a=>1); sub xy {$_[1]="muhahaha" }; xy $x{b},
$x{bwahaha}; use Data​::Dumper; print Dumper(\%x)'
$VAR1 = {
        'bwahaha' => 'muhahaha',
        'a' => 1
      };

does not autoviv $x{b}, demonstrating non-autivivving aliasing and
creating a reasonable expectation that for aliases won't autoviv
except when actually used as lvalues, just like arg aliases don't.

subs did at one time also autoviv but it was changed during 5.005
development (I think)

There are many places that auto-viv due to being in LVALUE (map and grep for
example), but sub args were special cased out (I cannot remember why)

IMO, the fact that subs do not autoviv is the bug as they are the exception

In the past I have used reverse to avoid the autoviv as it does not place
its
arguments in an LVALUE context

On that subject, and work to be done, see
http​://consttype.blogspot.com/2009/03/autovivification-in-perl-5.html
(blogged to -maybe- get a wider audience)

[p5pRT]

From @iabyn

I'm marking this ticket as rejected/wontfix, because no-one can agree on
whether for() should auto-vivify, and if there was agreement, changing
it would break backwards compatibility.

[p5pRT]

@iabyn - Status changed from 'open' to 'rejected'

[p5pRT]

From @cpansprout

On Mon Sep 06 05​:57​:18 2010, davem wrote​:

I'm marking this ticket as rejected/wontfix, because no-one can agree on
whether for() should auto-vivify, and if there was agreement, changing
it would break backwards compatibility.

I have just discovered this ticket, as there is a to-do test for it, which I found when about to add a test for #115194 (a duplicate).

I did not realise this was so controversial.

The problem here is that I broke compatibility (and App​::JobLog) when making foreach(pos $x || pos $y) work (by propagating the lvalue context to allow $_ assignment to set pos). Now foreach($h{k} || $h{l}) vivifies the elements, whereas it did not before.

The obvious solution seemed to be to use defelem magic in foreach, as we already do for sub calls.

Come to think of it, this has already been inconsistent, as constant folding will make for(1||$h{k}) vivify even before my changes.

So I think we need to bite the bullet and decide that this is a bug that needs fixing.

--

Father Chrysostomos

[p5pRT]

From [email protected]

I wrote​:

Come to think of it, this has already been inconsistent, as constant
folding will make for(1||$h{k}) vivify even before my changes.

I meant 0, not 1​:

$ perl5.10 -MData​::Dumper -e 'for(${\0}||$h{k}) {} print Dumper \%h'
$VAR1 = {};
$ perl5.10 -MData​::Dumper -e 'for(0||$h{k}) {} print Dumper \%h'
$VAR1 = {
  'k' => undef
  };

[p5pRT]

From @iabyn

On Thu, Jan 23, 2014 at 06​:19​:53PM -0800, Father Chrysostomos via RT wrote​:

The obvious solution seemed to be to use defelem magic in foreach, as we
already do for sub calls.

I don't like defelem magic​: I think it introduces unnecessary overhead to
function calls. So I wouldn't be keen for it to be extended to for loops.

--
Red sky at night - gerroff my land!
Red sky at morning - gerroff my land!
  -- old farmers' sayings #14

[p5pRT]

From [email protected]

Dave Mitchel wrote​:

On Thu, Jan 23, 2014 at 06​:19​:53PM -0800, Father Chrysostomos via RT wrote​:

The obvious solution seemed to be to use defelem magic in foreach, as we
already do for sub calls.

I don't like defelem magic​: I think it introduces unnecessary overhead to
function calls.

Is the overhead much greater than that occasioned by unnecessary viv-
ification?

So I wouldn't be keen for it to be extended to for loops.

How else can we fix the disparity between for(0||$h{k}) and
for(${\0}||$h{k) without breaking code that does not expect the latter
to autovivify?

[p5pRT]

From @iabyn

On Fri, Jan 24, 2014 at 02​:42​:35PM -0000, Father Chrysostomos wrote​:

Dave Mitchel wrote​:

On Thu, Jan 23, 2014 at 06​:19​:53PM -0800, Father Chrysostomos via RT wrote​:

The obvious solution seemed to be to use defelem magic in foreach, as we
already do for sub calls.

I don't like defelem magic​: I think it introduces unnecessary overhead to
function calls.

Is the overhead much greater than that occasioned by unnecessary viv-
ification?

Yes. Look at the timings for the following code (each time is for that
line only uncommented)​:

  sub f {}

  for (1..10_000_000) {
  my %h = qw(a 1 b 2 c3);
  # empty loop 6.17 sec
  # f($h{a}, $h{b}, $h{c}); # 10.94 sec
  # f($h{d}, $h{e}, $h{f}); # 15.53 sec
  # f(@​h{d}, @​h{e}, @​h{f}); # 10.40 sec
  }

Note that 1-element hash slices like @​h{d} still autovivify when used as
function arguments.

From the above, subtracting out the empty loop time​:

  4.77 s passing elements that already exist
  9.36 s creating and passing PVLV proxies
  4.23 s autovivifing, then passing

(and no, I don't understand why autovivifying is slightly faster than them
already existing.)

How else can we fix the disparity between for(0||$h{k}) and
for(${\0}||$h{k) without breaking code that does not expect the latter
to autovivify?

Do we even need to worry about it? It looks to me to be an obscure edge
case of autovivifying, where people would be hard-pressed to agree on what
they would expect the semantics to be. Doesn't seem worth making the
general case slower for it.

For the record, it think

  for ($h{k}) should auto-vivify $h{k} (and indeed it does)
  for ($h{k}||$h{l}) should ideally auto-vivify $h{l} if $h{k} is false,
  although I'm not too bothered either way
  (it didn't use to, but does now)

but more generally, I'm in favour the behaviour not changing from what it
has traditionally been, due to the risk of code breakage.

--
Modern art​:
  "That's easy, I could have done that!"
  "Ah, but you didn't!"

[p5pRT]

From @cpansprout

On Mon Jan 27 03​:35​:27 2014, davem wrote​:

4.77 s passing elements that already exist
9.36 s creating and passing PVLV proxies
4.23 s autovivifing, then passing

(and no, I don't understand why autovivifying is slightly faster than
them
already existing.)

If I could change that (make PVLVs fast), would you be willing to change
your mind?

--

Father Chrysostomos

[p5pRT]

@cpansprout - Status changed from 'rejected' to 'stalled'

[p5pRT]

From @iabyn

On Sun, Feb 09, 2014 at 11​:16​:03AM -0800, Father Chrysostomos via RT wrote​:

On Mon Jan 27 03​:35​:27 2014, davem wrote​:

4.77 s passing elements that already exist
9.36 s creating and passing PVLV proxies
4.23 s autovivifing, then passing

(and no, I don't understand why autovivifying is slightly faster than
them
already existing.)

If I could change that (make PVLVs fast), would you be willing to change
your mind?

Yes.

--
In economics, the exam questions are the same every year.
They just change the answers.

[p5pRT]

The RT System itself - Status changed from 'stalled' to 'open'