Output appears in "Captured stderr call" but is unavailable in capsys or capfd

[butla]

Environment

Pytest version: 5.2.1
Operating system: Kubuntu 18.04.3 LTS
Pip list:

Package            Version
------------------ -------
atomicwrites       1.3.0
attrs              19.3.0
importlib-metadata 0.23
more-itertools     7.2.0
packaging          19.2
pip                19.3.1
pluggy             0.13.0
py                 1.8.0
pyparsing          2.4.2
pytest             5.2.1
setuptools         41.4.0
six                1.12.0
wcwidth            0.1.7
wheel              0.33.6
zipp               0.6.0

Description

It seems that output can appear in "Captured stderr call" but be unavailable to capsys or capfd, which can be a bug? It's definitely weird.

I came accross this issue while trying to test my logging setup. This is a minimal example I came up with (available at https://github.com/butla/experiments/tree/master/pytest_capture_log_error):

# a_file.py
import logging

logging.basicConfig(level=logging.INFO)

LOG_MESSAGE = "the message we'll look for in the test"

_log = logging.getLogger(__name__)


def bla():
    _log.info(LOG_MESSAGE)
    return 5

# test_file.py
import a_file

def test_a(capsys):
    assert a_file.bla() == 5
    assert a_file.LOG_MESSAGE in capsys.readouterr().err

The test fails and it shows

------------------------------------------ Captured stderr call ------------------------------------------
INFO:a_file:the message we'll look for in the test

but both capsys and capfd show out and err to be empty.

I would use caplog, but the messages I get from it don't conform to the log configuration that I have set up in my original app, so I decided to just use stderr.

[asottile]

this is pretty strange, it probably should work with capfd (I could see why it wouldn't with capsys due the way the logging module works)

the test passes with -s -- definitely feels like some sort of bug

[butla]

Oh yeah, passing -s and using capfd fixture makes it pass...

[blueyed]

Related (I think): #3448

[MarcoGorelli]

This is just for stderr - if I modify a_file.py to have

def bla():
    print("foo")
    _log.info(LOG_MESSAGE)
    return 5

and test_file.py to have

def test_a(capsys):
    assert a_file.bla() == 5
    assert "foo" in capsys.readouterr().out
    assert a_file.LOG_MESSAGE in capsys.readouterr().err

then only the third assertion fails.

Currently looking into this

[blueyed]

@MarcoGorelli
This is due to logging using stderr, isn't it?
It picks up the mocked away stderr from pytest's capturing likely, i.e. it would also work if you'd do the logging setup in the test.

[MarcoGorelli]

Just as an update - I've been studying capture.py, but still haven't figured out where the issue might lie. If this is urgent, anyone more experienced following along is welcome to take over.

Else I'll keep hacking away :)

[ahal]

I'm seeing this issue with stdout logged via a custom logging module we have. Our logger is basically just calling sys.stdout.write(); sys.stdout.flush() at the end of the day.