In certain conditions in a podman container the link count in a directory may be inaccurate, confusing File::Find

[djerius]

Module: File::Find

Description

tldr;

• Adding a file to a directory built into a Debian 10 container running under podman may decrease the directory link count to an incorrect(?) value which leads File::Find to believe either that there are no subdirectories (if nlinks==2) or that there are fewer subdirectories present, producing incorrect results.
• This does not seem to affect directories mounted into the container, only directories which were built into the container.

Caveats

• I have seen this behavior on a Debian 10 host running podman 2.1.1.
• I cannot run docker on this host, so cannot verify that the same issue appears under docker
• I cannot reproduce this on another Debian 10 host running podman 2.1.1
• I cannot reproduce this on a CentOS7 host running docker
• I do not know if there are any guarantees as to what the link count in a directory entry means, so I don't know if this is a bug in podman.

Steps to Reproduce
As mentioned above, I've found it hard to duplicate, but here's my setup. The Dockerfile:

FROM    debian:10

RUN     apt-get update &&                       \
        apt-get -y upgrade &&                   \
        apt-get -y install                      \
            perl                                \
            perl-modules

At the command line:

% podman build -t file-find .

% podman run --rm -it file-find

% cd /var/log

% stat .
  File: .
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 3fh/63d Inode: 14945612    Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-11-10 05:54:11.026700311 +0000
Modify: 2020-11-10 05:53:51.000000000 +0000
Change: 2020-11-10 05:54:10.966700370 +0000
 Birth: -

The link count is 3.
`

% perl -MFile::Find=find -E 'find( sub { say $File::Find::name },q[.])'
.
./lastlog
./dpkg.log
./faillog
./btmp
./wtmp
./apt
./apt/history.log
./apt/term.log
./apt/eipp.log.xz

Note that File::Find descends into the apt directory. Now, modify the directory

% touch foo

% stat .
  File: .
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 3fh/63d Inode: 14945612    Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-11-10 05:54:11.026700311 +0000
Modify: 2020-11-10 15:16:09.549100015 +0000
Change: 2020-11-10 15:16:09.549100015 +0000
 Birth: -

The link count has decreased from 3 to 2.

% perl -MFile::Find=find -E 'find( sub { say $File::Find::name },q[.])'
.
./lastlog
./dpkg.log
./faillog
./foo
./btmp
./apt
./wtmp

and File::Find no longer descends into apt

Analysis

File::Find may use the link count of a directory to determine the number of subdirectories. This is controlled by the $File::Find::dont_use_nlink variable, which depends upon a configuration variable:

790 # Set dont_use_nlink in your hint file if your system's stat doesn't
791 # report the number of links in a directory as an indication
792 # of the number of files.
793 # See e.g. hints/haiku.sh for Haiku.
794 unless ($File::Find::dont_use_nlink) {
795     require Config;
796     $File::Find::dont_use_nlink = 1 if ($Config::Config{'dont_use_nlink'});
797 }

(From File::Find version 1.34). On Debian 10,

% perl -V:dont_use_nlink
dont_use_nlink='undef';

Later, in _find_dir, the link count to the directory is used to determine if there are any subdirectories, as well as the number of subdirectories. As Perl is configured to trust the link count, the revised link count of 2 after adding the file to the directory convinces File::Find that there are no subdirectories, when in fact there are.

Expected behavior

I expect File::Find to traverse all of the subdirectories

Perl configuration

Site configuration information for perl 5.28.1:

Configured by Debian at Tue Jul 21 19:27:00 UTC 2020.

Summary of my perl5 (revision 5 version 28 subversion 1) configuration:

  Platform:
    osname=linux
    osvers=4.9.0
    archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 4.9.0 #1 smp debian 4.9.0 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-voFw8F/perl-5.28.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.28 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.28 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.28 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.28.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.28.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.28.1'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='x86_64-linux-gnu-gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion=''
    gccversion='8.3.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.28.so
    so=so
    useshrplib=true
    libperl=libperl.so.5.28
    gnulibc_version='2.28'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.28.1-6+deb10u1 in patchlevel.h
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068.
    DEBPKG:debian/hppa_opmini_optimize_workaround - https://bugs.debian.org/869122 Lower the optimization level of opmini.c on hppa
    DEBPKG:debian/sh4_op_optimize_workaround - https://bugs.debian.org/869373 Also lower the optimization level of op.c and opmini.c on sh4
    DEBPKG:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/disable-stack-check - https://bugs.debian.org/902779 [perl #133327] Disable debugperl stack extension checks for binary compatibility with perl
    DEBPKG:debian/gdbm-fatal - [perl #133295] https://bugs.debian.org/904005 Temporarily skip GDBM_File fatal.t for gdbm >= 1.15 compatibility
    DEBPKG:fixes/storable-recursion - https://bugs.debian.org/912900 [perl #133326] [120060c] (perl #133326) fix and clarify handling of recurs_sv.
    DEBPKG:fixes/caretx-fallback - https://bugs.debian.org/913347 [perl #133573] [03b94aa] RT#133573: $^X fallback when platform-specific technique fails
    DEBPKG:fixes/eumm-usrmerge - https://bugs.debian.org/913637 Avoid mangling /bin non-perl shebangs on merged-/usr systems
    DEBPKG:fixes/errno-include-path - [6c5080f] [perl #133662] https://bugs.debian.org/875921 Make Errno_pm.PL compatible with /usr/include/<ARCH>/errno.h
    DEBPKG:fixes/kfreebsd-renameat - [a3c63a9] https://bugs.debian.org/912521 [perl #133668] Also work around renameat() kernel bug on GNU/kFreeBSD
    DEBPKG:fixes/time-local-2020 - https://bugs.debian.org/915209 [rt.cpan.org #124787] Fix Time::Local tests
    DEBPKG:fixes/inplace-editing-bugfix/part1 - https://bugs.debian.org/914651 (perl #133659) move argvout cleanup to a new function
    DEBPKG:fixes/inplace-editing-bugfix/part2 - https://bugs.debian.org/914651 (perl #133659) tests for global destruction handling of inplace editing
    DEBPKG:fixes/inplace-editing-bugfix/part3 - https://bugs.debian.org/914651 (perl #133659) make an in-place edit successful if the exit status is zero
    DEBPKG:fixes/fix-manifest-failures - https://bugs.debian.org/914962 Fix t/porting/manifest.t failures when run in a foreign git checkout
    DEBPKG:fixes/pipe-open-bugfix/part1 - [perl #133726] https://bugs.debian.org/916313 Always mark pipe in pipe-open as inherit-on-exec
    DEBPKG:fixes/pipe-open-bugfix/part2 - [perl #133726] https://bugs.debian.org/916313 Always mark pipe in list pipe-open as inherit-on-exec
    DEBPKG:fixes/storable-probing/prereq1 - [3f4cad1] Storable: fix for strawberry build failures:
    DEBPKG:fixes/storable-probing/prereq2 - [perl #133411] [edf639f] (perl #133411) don't try to load Storable with -Dusecrosscompile
    DEBPKG:fixes/storable-probing/disable-probing - https://bugs.debian.org/914133 [perl #133708] [2a0bbd3] (perl #133708) remove build-time probing for stack limits for Storable
    DEBPKG:debian/perlbug-editor - https://bugs.debian.org/922609 Use "editor" as the default perlbug editor, as per Debian policy
    DEBPKG:fixes/posix-mbrlen - [25d7b7a] https://bugs.debian.org/924517 [perl #133928] Fix POSIX::mblen mbstate_t initialization on threaded perls with glibc
    DEBPKG:fixes/CVE-2020-10543 - https://bugs.debian.org/962005 regcomp.c: Prevent integer overflow from nested regex quantifiers.
    DEBPKG:fixes/CVE-2020-10878 - https://bugs.debian.org/962005 study_chunk: extract rck_elide_nothing
    DEBPKG:fixes/CVE-2020-12723 - https://bugs.debian.org/962005 study_chunk: avoid mutating regexp program within GOSUB
    DEBPKG:fixes/io-socket-ip-nov4 - https://bugs.debian.org/962019 Fix test failures in IO::Socket::IP with an IPv6-only host

---
@INC for perl 5.28.1:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.28.1
    /usr/local/share/perl/5.28.1
    /usr/lib/x86_64-linux-gnu/perl5/5.28
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.28
    /usr/share/perl/5.28
    /usr/local/lib/site_perl
    /usr/lib/x86_64-linux-gnu/perl-base

---
Environment for perl 5.28.1:
    HOME=/root
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    PERL_BADLANG (unset)
    SHELL (unset)

[djerius]

Further note, since I can't replicate this outside of the one host, I can only raise this as a concern, so that if you do see other bug reports there's some added history.

Unless File::Find does a run-time test to see if directory link counts are accurate indications of the number of directories, I don't see much that can be done.

[Grinnz]

$File::Find::dont_use_nlink always defaults to 1 since Perl 5.30. On older Perls you can set this manually. See https://perldoc.perl.org/File::Find#CAVEAT and #16759.

[Leont]

$File::Find::dont_use_nlink always defaults to 1 since Perl 5.30. On older Perls you can set this manually. See https://perldoc.perl.org/File::Find#CAVEAT and #16759.

Yeah, it was an optimization that kept biting us in the tail

[djerius]

The incorrect directory link count was caused by a mis-configured system.

Anyway, glad to hear that had I upgraded to 5.30 I wouldn't have had this problem (but then I wouldn't have fixed the underlying problem).

[jkeenan]

The incorrect directory link count was caused by a mis-configured system.

Anyway, glad to hear that had I upgraded to 5.30 I wouldn't have had this problem (but then I wouldn't have fixed the underlying problem).

If I understand the comments in this ticket correctly, the original problem is not a problem in supported versions of Perl. Hence, this ticket should be closable. Comments?

Thank you very much.
Jim Keenan

[jkeenan]

The incorrect directory link count was caused by a mis-configured system.
Anyway, glad to hear that had I upgraded to 5.30 I wouldn't have had this problem (but then I wouldn't have fixed the underlying problem).

If I understand the comments in this ticket correctly, the original problem is not a problem in supported versions of Perl. Hence, this ticket should be closable. Comments?

No one has indicated this is not closable and no one has commented further. Closing ticket.

Thank you very much.
Jim Keenan

[karenetheridge]

Anyway, glad to hear that had I upgraded to 5.30 I wouldn't have had this problem (but then I wouldn't have fixed the underlying problem).

You don't need to upgrade to perl 5.30 -- all you need to do is update File::Find to at least version 1.35 (it's available separately on cpan), or set local $File​::Find​::dont_use_link = 1; before using File::Find functions.