Doctests fail when another test uses the fixture between runs.

[kevincox]

Here is a weird issue to track down and run. When you have a file with multiple doctests the tests will start failing if another test runs in between them. See the example below.

# code.py
def a():
    """
    >>> getfixture("foo")
    """
def b():
    """
    >>> getfixture("foo")
    """

# conftest.py
import pytest

@pytest.fixture
def foo():
    pass

# test.py
def test_func(foo):
    pass

% py.test --version                                                                         
This is pytest version 2.7.2, imported from /Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pytest-ipdb-0.1-prerelease2 at /Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/pytestipdb/ptipdb.pyc
  pytest-random-0.02 at /Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/random_plugin.pyc
% py.test --random --random-seed 368666649299 -vv --doctest-modules test.py code.py
================================================= test session starts =================================================
platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2 -- /Users/kevincox/.virtualenvs/starscream/bin/python2.7
Tests are shuffled using seed number 368666649299.
rootdir: /Users/kevincox, inifile: 
plugins: ipdb, random
collected 3 items 

code.py::code.a PASSED
test.py::test_func PASSED
code.py::code.b FAILED

====================================================== FAILURES =======================================================
__________________________________________________ [doctest] code.b ___________________________________________________
006     """
007     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.b[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:7: UnexpectedException
========================================= 1 failed, 2 passed in 0.02 seconds ==========================================
% echo $?
1

[kevincox]

It's worth noting that all tests after the first "interruption" will fall. See the below example with more tests in code.py.

% py.test --random -vv --doctest-modules test.py code.py
=================================================================================================================================== test session starts ===================================================================================================================================
platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2 -- /Users/kevincox/.virtualenvs/starscream/bin/python2.7
Tests are shuffled using seed number 368666913960.
rootdir: /Users/kevincox, inifile: 
plugins: ipdb, random
collected 8 items 

code.py::code.b PASSED
code.py::code.a PASSED
code.py::code.e PASSED
test.py::test_func PASSED
code.py::code.f FAILED
code.py::code.g FAILED
code.py::code.c FAILED
code.py::code.d FAILED

======================================================================================================================================== FAILURES =========================================================================================================================================
____________________________________________________________________________________________________________________________________ [doctest] code.f _____________________________________________________________________________________________________________________________________
022     """
023     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.f[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:23: UnexpectedException
____________________________________________________________________________________________________________________________________ [doctest] code.g _____________________________________________________________________________________________________________________________________
026     """
027     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.g[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:27: UnexpectedException
____________________________________________________________________________________________________________________________________ [doctest] code.c _____________________________________________________________________________________________________________________________________
010     """
011     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.c[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:11: UnexpectedException
____________________________________________________________________________________________________________________________________ [doctest] code.d _____________________________________________________________________________________________________________________________________
014     """
015     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.d[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:15: UnexpectedException
=========================================================================================================================== 4 failed, 4 passed in 0.03 seconds ============================================================================================================================ 

[kevincox]

Note, this also happens with multiple doctest files. Once another test starts running in between they start failing.

py.test --random -vv --doctest-modules code.py code2.py
===================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2 -- /Users/kevincox/.virtualenvs/starscream/bin/python2.7
Tests are shuffled using seed number 369149711971.
rootdir: /Users/kevincox, inifile: 
plugins: ipdb, random, xdist, xdist, xdist
collected 14 items 

code.py::code.a PASSED
code.py::code.e PASSED
code2.py::code2.f PASSED
code2.py::code2.a PASSED
code2.py::code2.g PASSED
code2.py::code2.e PASSED
code2.py::code2.b PASSED
code2.py::code2.c PASSED
code.py::code.g FAILED
code2.py::code2.d FAILED
code.py::code.c FAILED
code.py::code.d FAILED
code.py::code.f FAILED
code.py::code.b FAILED

========================================================================== FAILURES ===========================================================================
______________________________________________________________________ [doctest] code.g _______________________________________________________________________
026     """
027     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.g[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:27: UnexpectedException
______________________________________________________________________ [doctest] code2.d ______________________________________________________________________
014     """
015     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code2.d[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code2.py:15: UnexpectedException
______________________________________________________________________ [doctest] code.c _______________________________________________________________________
010     """
011     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.c[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:11: UnexpectedException
______________________________________________________________________ [doctest] code.d _______________________________________________________________________
014     """
015     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.d[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:15: UnexpectedException
______________________________________________________________________ [doctest] code.f _______________________________________________________________________
022     """
023     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.f[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:23: UnexpectedException
______________________________________________________________________ [doctest] code.b _______________________________________________________________________
006     """
007     >>> getfixture("foo")
UNEXPECTED EXCEPTION: AttributeError("FixtureDef instance has no attribute 'cached_result'",)
Traceback (most recent call last):

  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs

  File "<doctest code.b[0]>", line 1, in <module>

  File "/Users/kevincox/.virtualenvs/starscream/lib/python2.7/site-packages/_pytest/python.py", line 1385, in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]

AttributeError: FixtureDef instance has no attribute 'cached_result'

/Users/kevincox/code.py:7: UnexpectedException
============================================================= 6 failed, 8 passed in 0.03 seconds ==============================================================

[kgashok]

I am not sure whether this is a related issue, but cyber-dojo/web#38 demonstrates an odd behavior when a totally unrelated file named code.py exists in the same directory in which tests are executed.

See the output file in the http://cyber-dojo.org/kata/edit/24A724BD24?avatar=kingfisher

Should I post this as a separate issue?

[bluetech]

This doesn't reproduce anymore.