Improve test coverage for idlelib

[tonybaloney]

BPO	35668
Nosy	@terryjreedy, @taleinat, @csabella, @tonybaloney, @tonybaloney
PRs	gh-79871: IDLE - Fix and test debugger module #11451 gh-79871: IDLE - Fix and test debugger module #11451

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2019-01-05.21:44:20.194>
labels = ['3.7', '3.8', 'expert-IDLE', 'type-feature', 'library']
title = 'Improve test coverage for idlelib'
updated_at = <Date 2019-05-09.04:54:19.385>
user = 'https://github.com/tonybaloney'

bugs.python.org fields:

activity = <Date 2019-05-09.04:54:19.385>
actor = 'anthonypjshaw'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE', 'Library (Lib)']
creation = <Date 2019-01-05.21:44:20.194>
creator = 'anthony shaw'
dependencies = []
files = []
hgrepos = []
issue_num = 35668
keywords = ['patch', 'patch', 'patch', 'patch']
message_count = 10.0
messages = ['333077', '333083', '333084', '333085', '333086', '333089', '333113', '333124', '333138', '333269']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'taleinat', 'cheryl.sabella', 'anthonypjshaw', 'anthony shaw']
pr_nums = ['11451', '11451']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35668'
versions = ['Python 3.7', 'Python 3.8']

[tonybaloney]

idlelib is one of the lesser-tested libraries in cpython:
https://codecov.io/gh/python/cpython/tree/master/Lib/idlelib

Raising this issue and also volunteering to extend the test module to get coverage across major behaviours and functions that are missing tests.

[terryjreedy]

I (we) agree that idlelib needs even better test coverage. Some history:

1. I added idle_test/ 5-1/2 years ago. Only the calltip module had automated tests easily converted to unittests (test_calltip).
2. Other exiting tests, requiring human judgement, were debugged, completed, and converted to 'htests', driven by idle_test.htest. Properly measuring idlelib coverage requires excluding htest code (up to 20% of a module).
3. idle_test/README.txt has information on testing IDLE and local conventions. Perhaps more should be added about mocking.
4. We should now be using ttk widgets whenever possible. I would like to add an up-to-date widget-testing doc.

Some modules need doctests and a bit of refactoring before adding tests. Tests for a specific module should be a separate issue.

Testing tkinter is hard. Can you tell us a bit about your python, tkinter, and testing experience?

Cheryl Sabella wrote the majority of tests added in the last year+, and Tal Einat wrote test_squeezer. I suggest you review some of their recent work. They should be able to answer most questions as well as me.

[tonybaloney]

thanks terry,

Some great pointers there, I'll review the exiting work and the README doc.

With regards to my experience, I have quite extensive experience with python testing. Most of which would be open-source on my Github profile https://github.com/tonybaloney
Some of the larger Python projects I've contributed test suites to would be SaltStack, Apache Libcloud and StackStorm.
I've contributed to the tox project and pytest.
I also write tutorials on Python testing (mostly for beginners) like this one https://realpython.com/python-testing/

I admit I don't have much experience with tkinter.

It does sound like a challenge, but definitely one that I'm willing to research and approach responsibly.

[terryjreedy]

Great. I suggest then that you start with untested, normal, non-GUI (non-tkinter) code, assuming that you can find some. This can mean segregating functional code from gui code if they are currently intertwined. As point 1 above suggested, IDLE, which dates from about 2000, was originally written without automated testing in mind.

I should have added above
5. 7 months ago, finished #78036 'Minimally test every implementation module'.
In a few cases, that meant import the file, create an instance of the main module, and make a couple of minimal assertions. The 'coverage' of such files mostly means no syntax errors and X% ran without an exception.

Such code still needs 'does the right thing' tests. Part of my intention was to make adding those easier by removing the initial boilerplate as a barrier.

[tonybaloney]

I was looking at the debugger.py module as being a good place the start, writing test cases for the Idb, Debugger, StackViewer and NamespaceViewer by patching out the dependant components (bdb, Idb, etc.

I might start there, raise a PR against it and do a module at a time, then work my way up to some of the trickier, tkinter-based modules.

[terryjreedy]

I like that choice. There are 15 open issues for debugger and I have notes on a few possible enhancements.

[csabella]

Welcome Anthony!

Please ask me any questions you may have. My only suggestion at this point would be for you to feel free to add any docstrings/comments to the module as you go. Having the docstrings really helps in understanding the tests and vice versa.

[tonybaloney]

thanks Cheryl,

here's my branch https://github.com/tonybaloney/cpython/tree/idlelib_tests

I've already seen some code which is 17 years old!