From 4580a0914a42eee17088621a769e2cc05684a00a Mon Sep 17 00:00:00 2001 From: AlexWaygood Date: Sat, 30 Jul 2022 17:01:01 +0100 Subject: [PATCH 1/4] `test_cases`: Improve README --- test_cases/README.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/test_cases/README.md b/test_cases/README.md index d2265d2cb51d..36785767a133 100644 --- a/test_cases/README.md +++ b/test_cases/README.md @@ -10,15 +10,38 @@ get right.** 100% test coverage for typeshed is neither necessary nor desirable, as it would lead to code duplication. Moreover, typeshed has multiple other mechanisms for spotting errors in the stubs. +### The purpose of these tests + +Different tests in this directory serve different purposes. For some stubs in +typeshed, the type annotations are complex enough that it's useful to have +basic sanity checks that test whether a type checker understands the intent of +the annotations correctly. Examples of tests like these are +`stdlib/builtins/test_pow.py` and `stdlib/asyncio/test_gather.py`. + +Other tests, such as the tests for `ExitStack` in `stdlib/test_contextlib.py` +and the tests for `LogRecord` in `stdlib/test_logging.py`, do not relate to +stubs where the annotations are particularly complex, but they *do* relate to +stubs where decisions have been taken that might be slightly unusual. These +tests serve a different purpose: to check that type checkers do not emit +false-positive errors for idiomatic usage of these classes. + +### Differences to the rest of typeshed + Unlike the rest of typeshed, this directory largely contains `.py` files. This is because the purpose of this folder is to test the implications of typeshed changes for end users. Another difference to the rest of typeshed is that the test cases in this -directory cannot always use modern syntax for type hints. For example, PEP 604 +directory cannot always use modern syntax for type hints. + +For example, PEP 604 syntax (unions with a pipe `|` operator) is new in Python 3.10. While this syntax can be used on older Python versions in a `.pyi` file, code using this syntax will fail at runtime on Python <=3.9. Since the test cases all use `.py` extensions, and since the tests need to pass on all Python versions >=3.7, PEP 604 syntax cannot be used in a test case. Use `typing.Union` and `typing.Optional` instead. + +PEP 585 syntax can also not be used in the `test_cases` directory. Use +`typing.Tuple` instead of `tuple`, `typing.Callable` instead of +`collections.abc.Callable`, and `re.Match` instead of `typing.Match` (etc.). From 0043c6ea8e59e21d345b1e4b6177a386cc148897 Mon Sep 17 00:00:00 2001 From: AlexWaygood Date: Sat, 30 Jul 2022 17:26:15 +0100 Subject: [PATCH 2/4] Info on using `assert_type` etc. --- test_cases/README.md | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/test_cases/README.md b/test_cases/README.md index 36785767a133..1b9ef6ad2c68 100644 --- a/test_cases/README.md +++ b/test_cases/README.md @@ -1,8 +1,7 @@ ## Regression tests for typeshed This directory contains regression tests for the stubs found elsewhere in the -typeshed repo. Each file contains a number of test cases, all of which should -pass a type checker without error. +typeshed repo. **This directory should *only* contain tests for functions and classes which are known to have caused problems in the past, where the stubs are difficult to @@ -25,6 +24,34 @@ stubs where decisions have been taken that might be slightly unusual. These tests serve a different purpose: to check that type checkers do not emit false-positive errors for idiomatic usage of these classes. +### How the tests work + +Some files in this directory simply contain samples of idiomatic Python, which +should not (if the stubs are correct) cause a type checker to emit any errors. + +Many test cases also make use of +[`assert_type`](https://docs.python.org/3.11/library/typing.html#typing.assert_type), +a function which allows us to test whether a type checker's inferred type of an +expression is what we'd like it be. + +Finally, some stubs make use of `# type: ignore` comments (in combination with +mypy's +[`--warn-unused-ignore` setting](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores) +and pyright's +[`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings) +setting) to test instances where a type checker *should* emit some kind of +error, if the stubs are correct. Note that mypy's `--warn-unused-ignore` +setting is enabled for the entire subdirectory; however, the pyright setting +must be enabled on a per-file basis with +`# pyright: reportUnnecessaryTypeIgnoreComment=true` at the the top of the +file. + +For more information on using `assert_type` and +`--warn-unused-ignore`/`reportUnnecessaryTypeIgnoreComment` to test type +annotations, +[this page](https://typing.readthedocs.io/en/latest/source/quality.html#testing-using-assert-type-and-warn-unused-ignores) +provides a useful guide. + ### Differences to the rest of typeshed Unlike the rest of typeshed, this directory largely contains `.py` files. This From 83aa4085f5ef1cfeae5a8fd1d655ab8a705cfc2d Mon Sep 17 00:00:00 2001 From: Alex Waygood Date: Sat, 30 Jul 2022 17:28:02 +0100 Subject: [PATCH 3/4] Update test_cases/README.md --- test_cases/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test_cases/README.md b/test_cases/README.md index 1b9ef6ad2c68..f0c9dc571687 100644 --- a/test_cases/README.md +++ b/test_cases/README.md @@ -36,8 +36,8 @@ expression is what we'd like it be. Finally, some stubs make use of `# type: ignore` comments (in combination with mypy's -[`--warn-unused-ignore` setting](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores) -and pyright's +[`--warn-unused-ignore`](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores) +setting and pyright's [`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings) setting) to test instances where a type checker *should* emit some kind of error, if the stubs are correct. Note that mypy's `--warn-unused-ignore` From 41f01cec40dcc9773905e65804ecd0aafd65130f Mon Sep 17 00:00:00 2001 From: Alex Waygood Date: Sat, 30 Jul 2022 17:43:17 +0100 Subject: [PATCH 4/4] `--warn-unused-ignores`, not `--warn-unused-ignore` --- test_cases/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/test_cases/README.md b/test_cases/README.md index f0c9dc571687..f8db5bb5b0d9 100644 --- a/test_cases/README.md +++ b/test_cases/README.md @@ -36,18 +36,18 @@ expression is what we'd like it be. Finally, some stubs make use of `# type: ignore` comments (in combination with mypy's -[`--warn-unused-ignore`](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores) +[`--warn-unused-ignores`](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores) setting and pyright's [`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings) setting) to test instances where a type checker *should* emit some kind of -error, if the stubs are correct. Note that mypy's `--warn-unused-ignore` +error, if the stubs are correct. Note that mypy's `--warn-unused-ignores` setting is enabled for the entire subdirectory; however, the pyright setting must be enabled on a per-file basis with `# pyright: reportUnnecessaryTypeIgnoreComment=true` at the the top of the file. For more information on using `assert_type` and -`--warn-unused-ignore`/`reportUnnecessaryTypeIgnoreComment` to test type +`--warn-unused-ignores`/`reportUnnecessaryTypeIgnoreComment` to test type annotations, [this page](https://typing.readthedocs.io/en/latest/source/quality.html#testing-using-assert-type-and-warn-unused-ignores) provides a useful guide.