Fatal Python error: could not acquire lock for <_io.BufferedReader name='<stdin>'> at interpreter shutdown, possibly due to daemon threads

[brianjmurrell]

I'm seeing this with python3-clustershell-1.8.3-5.fc33.noarch/python3-3.9.2-1.fc33.x86_64:

$ clush -B -l vagrant -R ssh -S -w vm1,vm2,vm3,vm4,vm5,vm6,vm7,vm8,vm9 'set -ex
declare -a ftest_mounts
mapfile -t ftest_mounts < <(grep '\''added by ftest.sh'\'' /etc/fstab)
for n_mnt in "${ftest_mounts[@]}"; do
    mpnt=("${n_mnt}")
    sudo umount "${mpnt[1]}"
done
sudo sed -i -e "/added by ftest.sh/d" /etc/fstab'
...
Fatal Python error: could not acquire lock for <_io.BufferedReader name='<stdin>'> at interpreter shutdown, possibly due to daemon threads

Thread 0x00007fda5b020700 (most recent call first):
  File "/usr/local/lib/python3.6/site-packages/ClusterShell/CLI/Clush.py", line 620 in _stdin_thread_start
  File "/usr/lib64/python3.6/threading.py", line 864 in run
  File "/usr/lib64/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib64/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fda67de3740 (most recent call first):
/usr/lib/daos/TESTING/ftest/ftest.sh: line 69: 10381 Aborted                 clush "${CLUSH_ARGS[@]}" -B -l "${REMOTE_ACCT:-jenkins}" -R ssh -S -w "$(IFS=','; echo "${nodes[*]}")" "$(sed -e '1,/^$/d' "$SCRIPT_LOC"/pre_clean_nodes.sh)"

I found this: https://bugs.python.org/issue26037

Any thoughts?

[martinetd]

As a workaround you probably won't get this with --nostdin, while that limits some usage quite a few commands would still work.

If the problem is we can't be in a read() when clustershell exits then a proper fix is going to be annoying... switching to asyncio as they suggest might be the least painful solution but sounds like overkill to me...

[brianjmurrell]

I think the work-around works. But as you say, it limits what can be done.

Is this something that can/will be ultimately fixed in a way that the work-around is not needed?

[martinetd]

This definitely needs to get fixed, yes; just not sure of how yet so it might take some time.

I've also got a machine with the same versions (python3-3.9.2-1.fc33.x86_64 / python3-clustershell-1.8.3-5.fc33.noarch) and cannot reproduce the problem though, how often does this happen?
You did provide a command but maybe there are conditions around how it's started (e.g. from a script, non-interactively?) or something that would help trim this down.

[brianjmurrell]

It is running in a CI framework (i.e. so from a script).

It was happening 100% of the time before I used the work-around.

[degremont]

Hi Brian,

I remember catching strange issue when using Jenkins. The kind of file descriptor/tty Jenkins(Java?) is simulating could have sometimes unexpected behavior.

Is it a problem that appears when switching to Fedora 33 or this specific Python version?

• Could try sending something on stdin like an empty file to see if that makes a difference (compared to using --nostdin)
• --nostdin change the way Clush manages threads. Actually it is much simpler when --nostdin is used. That could explain why the bug happens only when using 2 threads.

[martinetd]

Ah, thanks for the hint (weird file for jenkins), I could reproduce with a fifo:

$ mkfifo /tmp/fifo
(need to open the fifo for writing or open for reading will block)
$ cat > /tmp/fifo &
$ clush -w localhost uname -r < /tmp/fifo > /dev/null
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedReader name='<stdin>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x96b3c0)

Current thread 0x00007fa76c36d740 (most recent call first):
<no Python frame>
Aborted (core dumped)

I guess this won't be the first time fifos don't work like normal pipes..

[brianjmurrell]

That's great news that you were able to reproduce. Does that make it easier to create and land a PR to fix it?