gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage

[sobolevn]

Instead of this coverage (which is almost 20 years old https://github.com/python/cpython/blame/78e6d72e38ef4b490f0098b644454031f20ae361/Lib/test/test_doctest.py#L3365-L3373) we should work on proper coverage.py integration in our test suite.

• Issue: test_doctest cannot be run directly: let's remove this option #111348

[skirpichev]

That's what I got in the mastermain and on this branch with ./python -m test -T test_doctest:

Using random seed: 3478682052
0:00:00 load avg: 0.96 Run 1 test sequentially
0:00:00 load avg: 0.96 [1/1] test_doctest
test_doctest passed in 38.9 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 38.9 sec
Total tests: run=63
Total test files: run=1/1
Result: SUCCESS
Not printing coverage data for '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py': [Errno 2] No such file or directory: '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/sk/src/cpython/Lib/test/__main__.py", line 2, in <module>
    main(_add_python_opts=True)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 665, in main
    Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 657, in main
    exitcode = self.run_tests(selected, tests)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 494, in run_tests
    return self._run_tests(selected, tests)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 474, in _run_tests
    self.finalize_tests(tracer)
  File "/home/sk/src/cpython/Lib/test/libregrtest/main.py", line 374, in finalize_tests
    results.write_results(show_missing=True, summary=True,
  File "/home/sk/src/cpython/Lib/trace.py", line 273, in write_results
    with open(filename, 'rb') as fp:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp7k4z8ing/doctest_testpkg/__init__.py'

After removing problem doctest (on this branch)

diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
index e51f54ef35..72f49a6834 100644
--- a/Lib/test/test_doctest.py
+++ b/Lib/test/test_doctest.py
@@ -2843,39 +2843,6 @@ def test_lineendings(): r"""
     TestResults(failed=0, attempted=1)
     >>> os.remove(fn)
 
-Now we test with a package loader that has a get_data method, since that
-bypasses the standard universal newline handling so doctest has to do the
-newline conversion itself; let's make sure it does so correctly (issue 1812).
-We'll write a file inside the package that has all three kinds of line endings
-in it, and use a package hook to install a custom loader; on any platform,
-at least one of the line endings will raise a ValueError for inconsistent
-whitespace if doctest does not correctly do the newline conversion.
-
-    >>> from test.support import os_helper
-    >>> import shutil
-    >>> dn = tempfile.mkdtemp()
-    >>> pkg = os.path.join(dn, "doctest_testpkg")
-    >>> os.mkdir(pkg)
-    >>> os_helper.create_empty_file(os.path.join(pkg, "__init__.py"))
-    >>> fn = os.path.join(pkg, "doctest_testfile.txt")
-    >>> with open(fn, 'wb') as f:
-    ...     f.write(
-    ...         b'Test:\r\n\r\n'
-    ...         b'  >>> x = 1 + 1\r\n\r\n'
-    ...         b'Done.\r\n'
-    ...         b'Test:\n\n'
-    ...         b'  >>> x = 1 + 1\n\n'
-    ...         b'Done.\n'
-    ...         b'Test:\r\r'
-    ...         b'  >>> x = 1 + 1\r\r'
-    ...         b'Done.\r'
-    ...     )
-    95
-    >>> with test_hook(dn):
-    ...     doctest.testfile("doctest_testfile.txt", package="doctest_testpkg", verbose=False)
-    TestResults(failed=0, attempted=3)
-    >>> shutil.rmtree(dn)
-
 """
 
 def test_testmod(): r"""

I got

$ ./python -m test -T test_doctest | grep doctest
0:00:00 load avg: 0.44 [1/1] test_doctest
test_doctest passed in 38.3 sec
 1092    69%   doctest   (/home/sk/src/cpython/Lib/doctest.py)
    5    80%   test.doctest_aliases   (/home/sk/src/cpython/Lib/test/doctest_aliases.py)
   17    82%   test.doctest_lineno   (/home/sk/src/cpython/Lib/test/doctest_lineno.py)
   13   100%   test.sample_doctest   (/home/sk/src/cpython/Lib/test/sample_doctest.py)
    3    66%   test.sample_doctest_no_docstrings   (/home/sk/src/cpython/Lib/test/sample_doctest_no_docstrings.py)
    5    80%   test.sample_doctest_no_doctests   (/home/sk/src/cpython/Lib/test/sample_doctest_no_doctests.py)
  184    80%   test.test_doctest   (/home/sk/src/cpython/Lib/test/test_doctest.py)
Warning -- Unraisable exception
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError: 

So, at least we have some coverage for the doctest module.

With coveragepy:

python -m coverage run --branch --source=doctest Lib/test/regrtest.py test_doctest
0:00:00 load avg: 1.33 Run 1 test sequentially
0:00:00 load avg: 1.33 [1/1] test_doctest

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.0 sec
Total tests: run=59
Total test files: run=1/1
Result: SUCCESS
Warning -- Unraisable exception
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError: 
$ python -m coverage report
Name             Stmts   Miss Branch BrPart  Cover
--------------------------------------------------
Lib/doctest.py     983    136    448     64    84%
--------------------------------------------------
TOTAL              983    136    448     64    84%
Exception ignored in: <module 'threading' from '/home/sk/src/cpython/Lib/threading.py'>
Traceback (most recent call last):
  File "/home/sk/src/cpython/Lib/threading.py", line 1588, in _shutdown
    assert tlock.locked()
AssertionError: 

Apparently, we have some coverage statistics for this module. (And few new bugs...)

[sobolevn]

@skirpichev I think that -T is not related to this issue. This one is about direct module invocation. However, this does look like a bug. Can you please open a new issue about it?

[sobolevn]

Because the failure that is still there is rather complex:

.....................................F...........................
======================================================================
FAIL: basics (test.test_doctest.test_DocTestFinder)
Doctest: test.test_doctest.test_DocTestFinder.basics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_doctest.test_DocTestFinder.basics
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 445, in basics

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 469, in test.test_doctest.test_DocTestFinder.basics
Failed example:
    print(tests)  # doctest: +ELLIPSIS
Expected:
    [<DocTest sample_func from test_doctest.py:32 (1 example)>]
Got:
    [<DocTest sample_func from /Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py:32 (1 example)>]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 576, in test.test_doctest.test_DocTestFinder.basics
Failed example:
    for t in tests:
        print('%2s  %s' % (len(t.examples), t.name))
Expected:
     1  some_module
     3  some_module.SampleClass
     3  some_module.SampleClass.NestedClass
     1  some_module.SampleClass.NestedClass.__init__
     1  some_module.SampleClass.__init__
     1  some_module.SampleClass.a_cached_property
     2  some_module.SampleClass.a_classmethod
     1  some_module.SampleClass.a_classmethod_property
     1  some_module.SampleClass.a_property
     1  some_module.SampleClass.a_staticmethod
     1  some_module.SampleClass.double
     1  some_module.SampleClass.get
     1  some_module.__test__.c
     2  some_module.__test__.d
     1  some_module.sample_func
Got:
     1  some_module
     1  some_module.__test__.c
     2  some_module.__test__.d

We would need to have some compat code there as well, while it does not bring much value (while making the test more complex and brittle)

[skirpichev]

I think that -T is not related to this issue

Hmm. That's an alias to --coverage (documented as a stdlib alternative to coveragepy).

Can you please open a new issue about it?

#111355

Because the failure that is still there is rather complex

Yeah, but what we gain raising a RuntimeError? Someone someday will open a bug if #111348 will be closed.

[AlexWaygood]

After merging in main, all tests pass for me locally with this PR branch when I run python Lib/test/test_doctest.py. Nice work @sobolevn and @skirpichev!

[miss-islington-app]

Thanks @sobolevn for the PR, and @AlexWaygood for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

[bedevere-app]

GH-111359 is a backport of this pull request to the 3.12 branch.

[bedevere-app]

GH-111360 is a backport of this pull request to the 3.11 branch.