Devel::Peek: misleading documentation for Dump($hashref)

[p5pRT]

Migrated from rt.perl.org#133499 (status was 'resolved')

Searchable as RT133499$

[p5pRT]

From @jkeenan

While looking at
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132764 I had
occasion to work through the examples in the Devel​::Peek
documentation. I found one case -- Dump-ing a hashref --
where the actual output of Dump() does not match the
predicted output.

The relevant documentation is attached as
'devel-peek-dump-hash-ref.txt'. Note that in the example in
the docs there are a total of 19 lines in the (STDERR)
output and that those lines include​:

#####
  RITER = -1
  EITER = 0x0
#####

... which are explained as​: "Fields RITER and EITER are used
by perlfunc/each."

Next, I ran the attached program, 'peek-hashref.pl' (again,
at blead). Note that its output -- attached as
peek-hashref-dump.txt -- now contains only 17 lines. The
lines beginning with 'RITER' and 'EITER' are missing.

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

Thank you very much.
Jim Keenan

[p5pRT]

From @jkeenan

=head2 A reference to a hash

The following shows the raw form of a reference to a hash.

  use Devel​::Peek;
  $a = {hello=>42};
  Dump $a;

The output​:

  SV = IV(0x8177858) at 0x816a618
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x814fc10
  SV = PVHV(0x8167768) at 0x814fc10
  REFCNT = 1
  FLAGS = (SHAREKEYS)
  ARRAY = 0x816c5b8 (0​:7, 1​:1)
  hash quality = 100.0%
  KEYS = 1
  FILL = 1
  MAX = 7
  RITER = -1
  EITER = 0x0
  Elt "hello" HASH = 0xc8fd181b
  SV = IV(0x816c030) at 0x814fcf4
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 42

This shows C<$a> is a reference pointing to an SV. That SV is a PVHV, a
hash. Fields RITER and EITER are used by C<L<perlfunc/each>>.

[p5pRT]

From @jkeenan

peek-hashref.pl

[p5pRT]

From @jkeenan

SV = IV(0x55cb50b50fb0) at 0x55cb50b50fc0
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x55cb50b2b758
  SV = PVHV(0x55cb50b319c0) at 0x55cb50b2b758
  REFCNT = 1
  FLAGS = (SHAREKEYS)
  ARRAY = 0x55cb50b941a0 (0​:7, 1​:1)
  hash quality = 100.0%
  KEYS = 1
  FILL = 1
  MAX = 7
  Elt "hello" HASH = 0x3128ece4
  SV = IV(0x55cb50b464f8) at 0x55cb50b46508
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 42

[p5pRT]

From @jkeenan

Summary of my perl5 (revision 5 version 29 subversion 3) configuration​:
  Commit id​: 3809fbe
  Platform​:
  osname=linux
  osvers=4.15.0-33-generic
  archname=x86_64-linux
  uname='linux zareason 4.15.0-33-generic #36-ubuntu smp wed aug 15 16​:00​:05 utc 2018 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=undef
  usemultiplicity=undef
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize='-O2'
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='7.3.0'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.27.so
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version='2.27'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  HAS_TIMES
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  Built under linux
  Compiled at Sep 4 2018 10​:33​:18
  %ENV​:
  PERL2DIR="/home/jkeenan/gitwork/perl2"
  PERLBREW_BASHRC_VERSION="0.78"
  PERLBREW_HOME="/home/jkeenan/.perlbrew"
  PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.28.0/man"
  PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin​:/home/jkeenan/perl5/perlbrew/perls/perl-5.28.0/bin"
  PERLBREW_PERL="perl-5.28.0"
  PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"
  PERLBREW_VERSION="0.78"
  PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.29.3/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.29.3
  /usr/local/lib/perl5/5.29.3/x86_64-linux
  /usr/local/lib/perl5/5.29.3

[p5pRT]

From @tonycoz

On Tue, 04 Sep 2018 07​:44​:00 -0700, jkeenan@​pobox.com wrote​:

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

No, the behaviour changed in e1a7ec8.

Note that you can get RITER/EITER if the HV is iterated over​:

$ ./perl -Ilib -MDevel​::Peek -le '$a = { hello => 42 }; keys(%$a); Dump($a)'
SV = IV(0x5639e6ab1a50) at 0x5639e6ab1a60
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x5639e6a8f508
  SV = PVHV(0x5639e6a95880) at 0x5639e6a8f508
  REFCNT = 1
  FLAGS = (OOK,SHAREKEYS)
  AUX_FLAGS = 0
  ARRAY = 0x5639e6aae330 (0​:7, 1​:1)
  hash quality = 100.0%
  KEYS = 1
  FILL = 1
  MAX = 7
  RITER = -1
  EITER = 0x0
  RAND = 0xeea6f6ff
  Elt "hello" HASH = 0x30ca77f3
  SV = IV(0x5639e6aa84a8) at 0x5639e6aa84b8
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 42

So perhaps the example could be updated.

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jkeenan

On Thu, 06 Sep 2018 00​:55​:38 GMT, tonyc wrote​:

On Tue, 04 Sep 2018 07​:44​:00 -0700, jkeenan@​pobox.com wrote​:

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

No, the behaviour changed in e1a7ec8.

Note that you can get RITER/EITER if the HV is iterated over​:

$ ./perl -Ilib -MDevel​::Peek -le '$a = { hello => 42 }; keys(%$a); Dump($a)'
SV = IV(0x5639e6ab1a50) at 0x5639e6ab1a60
REFCNT = 1
FLAGS = (ROK)
RV = 0x5639e6a8f508
SV = PVHV(0x5639e6a95880) at 0x5639e6a8f508
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0x5639e6aae330 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0xeea6f6ff
Elt "hello" HASH = 0x30ca77f3
SV = IV(0x5639e6aa84a8) at 0x5639e6aa84b8
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42

So perhaps the example could be updated.

Tony, thanks for that explanation. The updating could proceed in (at least) two directions​:

1. We could retain the code being dumped and omit (i) omit the RITER and EITER lines from the output; and (ii) the sentence, "Fields RITER and EITER are used by perlfunc/each."

2. We could do (1) and then additionally document your example with 'keys'. However, in that case we would not only have to explain the RITER and EITER lines; we would have to explain some additional lines as well. This is what I get​:

#####
$ perl -MDevel​::Peek -e '$a = { hello => 42 }; keys %{$a}; Dump $a;'
SV = IV(0xeb0dc0) at 0xeb0dd0
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0xe8a578
  SV = PVHV(0xe909c0) at 0xe8a578
  REFCNT = 1
  FLAGS = (OOK,SHAREKEYS)
  AUX_FLAGS = 0
  ARRAY = 0xeeb1d0 (0​:7, 1​:1)
  hash quality = 100.0%
  KEYS = 1
  FILL = 1
  MAX = 7
  RITER = -1
  EITER = 0x0
  RAND = 0x9a8c1a4e
  Elt "hello" HASH = 0xe2eb8e64
  SV = IV(0xea5f58) at 0xea5f68
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 42
#####

The FLAGS line now has​:

#####
FLAGS = (OOK,SHAREKEYS)
#####

... instead of just (SHAREKEYS).

This line is not previously documented​:

#####
AUX_FLAGS = 0
#####

This line is also not previously documented​:

#####
  RAND = 0x9a8c1a4e
#####

Given that the behavior changed back in 2013, the documentation has been misleading for several years. Hence, any change we make should be backportable to 5.26-maint and 5.28-maint.

Approach (1) above is easy; I could do it today. But I don't know enough about hash structures to do approach (2).

How shall we proceed?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @tonycoz

On Fri, 07 Sep 2018 07​:24​:27 -0700, jkeenan wrote​:

Approach (1) above is easy; I could do it today. But I don't know
enough about hash structures to do approach (2).

How shall we proceed?

The output of Devel​::Peek's Dump() is very much an internals thing, I don't think it's a big deal that it doesn't exactly match.

If it bothers you that we don't get quite the same output, please update the documentation to match.

Tony

[p5pRT]

From @jkeenan

On Fri, 07 Sep 2018 14​:24​:27 GMT, jkeenan wrote​:

On Thu, 06 Sep 2018 00​:55​:38 GMT, tonyc wrote​:

On Tue, 04 Sep 2018 07​:44​:00 -0700, jkeenan@​pobox.com wrote​:

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

No, the behaviour changed in
e1a7ec8.

Note that you can get RITER/EITER if the HV is iterated over​:

$ ./perl -Ilib -MDevel​::Peek -le '$a = { hello => 42 }; keys(%$a);
Dump($a)'
SV = IV(0x5639e6ab1a50) at 0x5639e6ab1a60
REFCNT = 1
FLAGS = (ROK)
RV = 0x5639e6a8f508
SV = PVHV(0x5639e6a95880) at 0x5639e6a8f508
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0x5639e6aae330 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0xeea6f6ff
Elt "hello" HASH = 0x30ca77f3
SV = IV(0x5639e6aa84a8) at 0x5639e6aa84b8
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42

So perhaps the example could be updated.

Tony, thanks for that explanation. The updating could proceed in (at
least) two directions​:

1. We could retain the code being dumped and omit (i) omit the RITER
and EITER lines from the output; and (ii) the sentence, "Fields RITER
and EITER are used by perlfunc/each."

I decided to go with the simpler option​: #1. Pushed to blead in commit 7bf15fe.

2. We could do (1) and then additionally document your example with
'keys'. However, in that case we would not only have to explain the
RITER and EITER lines; we would have to explain some additional lines
as well. This is what I get​:

#####
$ perl -MDevel​::Peek -e '$a = { hello => 42 }; keys %{$a}; Dump $a;'
SV = IV(0xeb0dc0) at 0xeb0dd0
REFCNT = 1
FLAGS = (ROK)
RV = 0xe8a578
SV = PVHV(0xe909c0) at 0xe8a578
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0xeeb1d0 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0x9a8c1a4e
Elt "hello" HASH = 0xe2eb8e64
SV = IV(0xea5f58) at 0xea5f68
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42
#####

The FLAGS line now has​:

#####
FLAGS = (OOK,SHAREKEYS)
#####

... instead of just (SHAREKEYS).

This line is not previously documented​:

#####
AUX_FLAGS = 0
#####

This line is also not previously documented​:

#####
RAND = 0x9a8c1a4e
#####

Given that the behavior changed back in 2013, the documentation has
been misleading for several years. Hence, any change we make should
be backportable to 5.26-maint and 5.28-maint.

Approach (1) above is easy; I could do it today. But I don't know
enough about hash structures to do approach (2).

How shall we proceed?

Thank you very much.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

@jkeenan - Status changed from 'open' to 'pending release'

[p5pRT]

From @demerphq

On Fri, 7 Sep 2018, 16​:24 James E Keenan via RT, <perlbug-followup@​perl.org>
wrote​:

On Thu, 06 Sep 2018 00​:55​:38 GMT, tonyc wrote​:

On Tue, 04 Sep 2018 07​:44​:00 -0700, jkeenan@​pobox.com wrote​:

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

No, the behaviour changed in e1a7ec8.

Note that you can get RITER/EITER if the HV is iterated over​:

$ ./perl -Ilib -MDevel​::Peek -le '$a = { hello => 42 }; keys(%$a);
Dump($a)'
SV = IV(0x5639e6ab1a50) at 0x5639e6ab1a60
REFCNT = 1
FLAGS = (ROK)
RV = 0x5639e6a8f508
SV = PVHV(0x5639e6a95880) at 0x5639e6a8f508
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0x5639e6aae330 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0xeea6f6ff
Elt "hello" HASH = 0x30ca77f3
SV = IV(0x5639e6aa84a8) at 0x5639e6aa84b8
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42

So perhaps the example could be updated.

Tony, thanks for that explanation. The updating could proceed in (at
least) two directions​:

1. We could retain the code being dumped and omit (i) omit the RITER and
EITER lines from the output; and (ii) the sentence, "Fields RITER and EITER
are used by perlfunc/each."

2. We could do (1) and then additionally document your example with
'keys'. However, in that case we would not only have to explain the RITER
and EITER lines; we would have to explain some additional lines as well.
This is what I get​:

#####
$ perl -MDevel​::Peek -e '$a = { hello => 42 }; keys %{$a}; Dump $a;'
SV = IV(0xeb0dc0) at 0xeb0dd0
REFCNT = 1
FLAGS = (ROK)
RV = 0xe8a578
SV = PVHV(0xe909c0) at 0xe8a578
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0xeeb1d0 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0x9a8c1a4e
Elt "hello" HASH = 0xe2eb8e64
SV = IV(0xea5f58) at 0xea5f68
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42
#####

The FLAGS line now has​:

#####
FLAGS = (OOK,SHAREKEYS)
#####

... instead of just (SHAREKEYS).

Nicholas used the OOK flag to signal that the HVAUX structure is included
at the end of the bucket array. It is only set when a hash has been
iterated over at least once in its lifetime. Eiter and riter are it's main
contents.

This line is not previously documented​:

#####
AUX_FLAGS = 0
#####

These are flags in the HVAUX structure.

This line is also not previously documented​:

#####
RAND = 0x9a8c1a4e
#####

This is a mask used during iteration over the keys which makes it a bit
harder to attack the hash, and which is used to detect insert during each.

Given that the behavior changed back in 2013, the documentation has been
misleading for several years. Hence, any change we make should be
backportable to 5.26-maint and 5.28-maint.

Approach (1) above is easy; I could do it today. But I don't know enough
about hash structures to do approach (2).

How shall we proceed?

Up to you. These are optional fields and flags that only exist if the has
has been iterated over at least once. They are optional as they save a lot
of memory

Yves

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133499

[p5pRT]

From @jkeenan

On Mon, 15 Oct 2018 16​:35​:50 GMT, demerphq wrote​:

On Fri, 7 Sep 2018, 16​:24 James E Keenan via RT, <perlbug-
followup@​perl.org>
wrote​:

On Thu, 06 Sep 2018 00​:55​:38 GMT, tonyc wrote​:

On Tue, 04 Sep 2018 07​:44​:00 -0700, jkeenan@​pobox.com wrote​:

Has the behavior of Devel​::Peek changed since the
documentation was written? Or was the documentation simply
wrong from the onset?

No, the behaviour changed in
e1a7ec8.

Note that you can get RITER/EITER if the HV is iterated over​:

$ ./perl -Ilib -MDevel​::Peek -le '$a = { hello => 42 }; keys(%$a);
Dump($a)'
SV = IV(0x5639e6ab1a50) at 0x5639e6ab1a60
REFCNT = 1
FLAGS = (ROK)
RV = 0x5639e6a8f508
SV = PVHV(0x5639e6a95880) at 0x5639e6a8f508
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0x5639e6aae330 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0xeea6f6ff
Elt "hello" HASH = 0x30ca77f3
SV = IV(0x5639e6aa84a8) at 0x5639e6aa84b8
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42

So perhaps the example could be updated.

Tony, thanks for that explanation. The updating could proceed in (at
least) two directions​:

1. We could retain the code being dumped and omit (i) omit the RITER
and
EITER lines from the output; and (ii) the sentence, "Fields RITER and
EITER
are used by perlfunc/each."

2. We could do (1) and then additionally document your example with
'keys'. However, in that case we would not only have to explain the
RITER
and EITER lines; we would have to explain some additional lines as
well.
This is what I get​:

#####
$ perl -MDevel​::Peek -e '$a = { hello => 42 }; keys %{$a}; Dump $a;'
SV = IV(0xeb0dc0) at 0xeb0dd0
REFCNT = 1
FLAGS = (ROK)
RV = 0xe8a578
SV = PVHV(0xe909c0) at 0xe8a578
REFCNT = 1
FLAGS = (OOK,SHAREKEYS)
AUX_FLAGS = 0
ARRAY = 0xeeb1d0 (0​:7, 1​:1)
hash quality = 100.0%
KEYS = 1
FILL = 1
MAX = 7
RITER = -1
EITER = 0x0
RAND = 0x9a8c1a4e
Elt "hello" HASH = 0xe2eb8e64
SV = IV(0xea5f58) at 0xea5f68
REFCNT = 1
FLAGS = (IOK,pIOK)
IV = 42
#####

The FLAGS line now has​:

#####
FLAGS = (OOK,SHAREKEYS)
#####

... instead of just (SHAREKEYS).

Nicholas used the OOK flag to signal that the HVAUX structure is
included
at the end of the bucket array. It is only set when a hash has been
iterated over at least once in its lifetime. Eiter and riter are it's
main
contents.

This line is not previously documented​:

#####
AUX_FLAGS = 0
#####

These are flags in the HVAUX structure.

This line is also not previously documented​:

#####
RAND = 0x9a8c1a4e
#####

This is a mask used during iteration over the keys which makes it a
bit
harder to attack the hash, and which is used to detect insert during
each.

Given that the behavior changed back in 2013, the documentation has
been
misleading for several years. Hence, any change we make should be
backportable to 5.26-maint and 5.28-maint.

Approach (1) above is easy; I could do it today. But I don't know
enough
about hash structures to do approach (2).

How shall we proceed?

Up to you. These are optional fields and flags that only exist if the
has
has been iterated over at least once. They are optional as they save a
lot
of memory

Yves

Yves,

Since I didn't hear anything for several weeks, I pushed a patch to blead yesterday. If you would like to make revisions along the lines of your comments above, could you propose a patch to blead?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'