clush: line buffering broken with Python 3.9.16

[thiell]

With Python 3.9.16, clush performs full buffering on stdout/stderr instead of line buffering.

Environment

# cat /etc/redhat-release 
Rocky Linux release 9.2 (Blue Onyx)
# which python
/usr/bin/python
# ls -l /usr/bin/python
lrwxrwxrwx. 1 root root 9 Jun 21 23:53 /usr/bin/python -> ./python3
# python --version
Python 3.9.16

Problem

Lines are not displayed live anymore, console or not. Example with a pipe to pv:

# clush -w @oss 'for i in {1..3}; do date; sleep 1; done' | pv -nlt
1.0811 0
2.0722 0
3.0633 0
4.0543 0
elm-rcf-io1-s2: Sat Sep 16 01:25:57 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:57 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:25:58 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:58 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:25:59 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:59 PM PDT 2023
4.2953 2

Same for stderr:

# clush -w @oss 'for i in {1..3}; do date >&2; sleep 1; done' |& pv -nlt
1.0812 0
2.0722 0
3.0632 0
elm-rcf-io1-s2: Sat Sep 16 01:28:21 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:21 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:28:22 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:22 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:28:23 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:23 PM PDT 2023
3.4652 2

We should have the following instead:

# clush -w @oss 'for i in {1..3}; do date; sleep 1; done' | pv -nlt
elm-rcf-io1-s2: Sat Sep 16 01:41:30 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:30 PM PDT 2023
1.0797 2
elm-rcf-io1-s2: Sat Sep 16 01:41:31 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:31 PM PDT 2023
2.0814 4
elm-rcf-io1-s2: Sat Sep 16 01:41:32 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:32 PM PDT 2023
3.0838 6
3.4693 8

This is an issue with the Python 3 port because we use sys.stdout.buffer (_io.BufferedWriter) instead of sys.stdout (_io.TextIOWrapper).

With Python 3.9.16:

• if we use _io.BufferedWriter (eg. sys.stdout.buffer), we always get full buffering
• if stdout is a console, _io.TextIOWrapper will perform line buffering by default (we want that)
• if stdout is NOT a console, _io.TextIOWrapper will perform full buffering, but we want line buffering! Thankfully there is a way to enable line buffering with sys.stdout.reconfigure(line_buffering=True)

  If line_buffering is True, flush() is implied when a call to write contains a newline character or a carriage return.

Resolution

We could add explicit flush() calls – it would be easy thanks to CLI/Display.py – but I don't love it, because they will potentially be called a lot of time. It's better to let Python handle that. So let's try to avoid using sys.stdout.buffer completely + io.TextIOWrapper.reconfigure() if available in Python 3 and see how it goes. I will submit a patch shortly.