Improve test coverage for idlelib

[tonybaloney]

BPO	35668
Nosy	@terryjreedy, @taleinat, @csabella, @tonybaloney, @tonybaloney
PRs	gh-79871: IDLE - Fix and test debugger module #11451 gh-79871: IDLE - Fix and test debugger module #11451

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2019-01-05.21:44:20.194>
labels = ['3.7', '3.8', 'expert-IDLE', 'type-feature', 'library']
title = 'Improve test coverage for idlelib'
updated_at = <Date 2019-05-09.04:54:19.385>
user = 'https://github.com/tonybaloney'

bugs.python.org fields:

activity = <Date 2019-05-09.04:54:19.385>
actor = 'anthonypjshaw'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE', 'Library (Lib)']
creation = <Date 2019-01-05.21:44:20.194>
creator = 'anthony shaw'
dependencies = []
files = []
hgrepos = []
issue_num = 35668
keywords = ['patch', 'patch', 'patch', 'patch']
message_count = 10.0
messages = ['333077', '333083', '333084', '333085', '333086', '333089', '333113', '333124', '333138', '333269']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'taleinat', 'cheryl.sabella', 'anthonypjshaw', 'anthony shaw']
pr_nums = ['11451', '11451']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35668'
versions = ['Python 3.7', 'Python 3.8']

[tonybaloney]

idlelib is one of the lesser-tested libraries in cpython:
https://codecov.io/gh/python/cpython/tree/master/Lib/idlelib

Raising this issue and also volunteering to extend the test module to get coverage across major behaviours and functions that are missing tests.

[terryjreedy]

I (we) agree that idlelib needs even better test coverage. Some history:

1. I added idle_test/ 5-1/2 years ago. Only the calltip module had automated tests easily converted to unittests (test_calltip).
2. Other exiting tests, requiring human judgement, were debugged, completed, and converted to 'htests', driven by idle_test.htest. Properly measuring idlelib coverage requires excluding htest code (up to 20% of a module).
3. idle_test/README.txt has information on testing IDLE and local conventions. Perhaps more should be added about mocking.
4. We should now be using ttk widgets whenever possible. I would like to add an up-to-date widget-testing doc.

Some modules need doctests and a bit of refactoring before adding tests. Tests for a specific module should be a separate issue.

Testing tkinter is hard. Can you tell us a bit about your python, tkinter, and testing experience?

Cheryl Sabella wrote the majority of tests added in the last year+, and Tal Einat wrote test_squeezer. I suggest you review some of their recent work. They should be able to answer most questions as well as me.

[tonybaloney]

thanks terry,

Some great pointers there, I'll review the exiting work and the README doc.

With regards to my experience, I have quite extensive experience with python testing. Most of which would be open-source on my Github profile https://github.com/tonybaloney
Some of the larger Python projects I've contributed test suites to would be SaltStack, Apache Libcloud and StackStorm.
I've contributed to the tox project and pytest.
I also write tutorials on Python testing (mostly for beginners) like this one https://realpython.com/python-testing/

I admit I don't have much experience with tkinter.

It does sound like a challenge, but definitely one that I'm willing to research and approach responsibly.

[terryjreedy]

Great. I suggest then that you start with untested, normal, non-GUI (non-tkinter) code, assuming that you can find some. This can mean segregating functional code from gui code if they are currently intertwined. As point 1 above suggested, IDLE, which dates from about 2000, was originally written without automated testing in mind.

I should have added above
5. 7 months ago, finished #78036 'Minimally test every implementation module'.
In a few cases, that meant import the file, create an instance of the main module, and make a couple of minimal assertions. The 'coverage' of such files mostly means no syntax errors and X% ran without an exception.

Such code still needs 'does the right thing' tests. Part of my intention was to make adding those easier by removing the initial boilerplate as a barrier.

[tonybaloney]

I was looking at the debugger.py module as being a good place the start, writing test cases for the Idb, Debugger, StackViewer and NamespaceViewer by patching out the dependant components (bdb, Idb, etc.

I might start there, raise a PR against it and do a module at a time, then work my way up to some of the trickier, tkinter-based modules.

[terryjreedy]

I like that choice. There are 15 open issues for debugger and I have notes on a few possible enhancements.

[csabella]

Welcome Anthony!

Please ask me any questions you may have. My only suggestion at this point would be for you to feel free to add any docstrings/comments to the module as you go. Having the docstrings really helps in understanding the tests and vice versa.

[tonybaloney]

thanks Cheryl,

here's my branch https://github.com/tonybaloney/cpython/tree/idlelib_tests

I've already seen some code which is 17 years old!

[terryjreedy]

I like having multiple commits with explanatory messages. They get squashed when merging into master. Unless I get distracted and forget, I will squash the messages.

[terryjreedy]

I moved the debugger tests to bpo-35690. I want to keep this issue for general discussion of testing IDLE, and possibly related PRs improving the documentation thereof.

Serhiy Storchaka, the current tkinter maintainer, and I, have decided that IDLE should run with the implicit default root mechanism disabled. (It allows bugs unless there are no explicit Tk() calls in the process.)

Therefore, tests should run the same way. But when I put 'tk.NoDefaultRoot()' in the always-run part of test.test_idle, which is usually run with test.regrtest, there were problems. So I moved it to the 'if main' clause that runs the IDLE suite directly with unittest. But this clause only executes when test_idle is the main module, such as when one runs 'python -m test.test_idle' instead of 'python -m test -ugui', with or without 'test_idle added to restrict testing to test_idle.

I will add the consequence for the debugger test to the PR.

[terryjreedy]

With the movement of issues to github, we have IDLE Issues with 'Tests' and 'Test module' sections.