Use native SupportsIndex in builtins

[hatal175]

As requested by @Akuli in #5228.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore.git)
- tests/wasm/json2mc.py:103: note:     def __getitem__(self, _SupportsIndex) -> Any
+ tests/wasm/json2mc.py:103: note:     def __getitem__(self, SupportsIndex) -> Any
- tests/wasm/json2mc.py:159: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "_SupportsIndex"
+ tests/wasm/json2mc.py:159: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "SupportsIndex"
- tests/wasm/json2mc.py:161: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "_SupportsIndex"
+ tests/wasm/json2mc.py:161: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "SupportsIndex"

scipy (https://github.com/scipy/scipy.git)
+ scipy/linalg/_cython_signature_generator.py:198: error: Unpacking a string is disallowed  [misc]
+ scipy/linalg/_cython_signature_generator.py:199: error: Cannot determine type of 'libname'  [has-type]
+ scipy/linalg/_cython_signature_generator.py:200: error: Cannot determine type of 'outfile'  [has-type]
+ scipy/linalg/_cython_signature_generator.py:200: error: Cannot determine type of 'src_dir'  [has-type]
+ scipy/linalg/_cython_signature_generator.py:201: error: Cannot determine type of 'libname'  [has-type]
+ scipy/linalg/_cython_signature_generator.py:202: error: Cannot determine type of 'outfile'  [has-type]
+ scipy/linalg/_cython_signature_generator.py:202: error: Cannot determine type of 'src_dir'  [has-type]
+ scipy/io/matlab/tests/test_mio.py:224: error: "Dict[str, Collection[str]]" has no attribute "append"  [attr-defined]

sphinx (https://github.com/sphinx-doc/sphinx.git)
+ sphinx/util/logging.py: note: In function "pending_warnings":
+ sphinx/util/logging.py:200:24: error: "Handler" has no attribute "__iter__" (not iterable)
+ sphinx/util/logging.py: note: In function "suppress_logging":
+ sphinx/util/logging.py:232:24: error: "Handler" has no attribute "__iter__" (not iterable)
+ sphinx/util/logging.py: note: In function "setup":
+ sphinx/util/logging.py:573:20: error: "Handler" has no attribute "__iter__" (not iterable)
+ sphinx/util/logging.py: note: At top level:
+ sphinx/pycode/parser.py: note: In member "get_qualname_for" of class "VariableCommentPicker":
+ sphinx/pycode/parser.py:243:24: error: Incompatible return value type (got "str", expected "Optional[List[str]]")
+ sphinx/pycode/parser.py:243:44: error: Unsupported operand types for + ("str" and "List[str]")
+ sphinx/pycode/parser.py: note: At top level:
+ sphinx/pycode/parser.py: note: In member "visit_Assign" of class "VariableCommentPicker":
+ sphinx/pycode/parser.py:368:37: error: Unsupported operand types for + ("List[str]" and "str")
+ sphinx/pycode/parser.py: note: At top level:
+ sphinx/events.py: note: In member "disconnect" of class "EventManager":
+ sphinx/events.py:84:20: error: "object" has no attribute "id"
+ sphinx/events.py:85:38: error: Argument 1 to "remove" of "list" has incompatible type "object"; expected "EventListener"
+ sphinx/environment/adapters/toctree.py: note: In member "_toctree_prune" of class "TocTree":
+ sphinx/environment/adapters/toctree.py:281:24: error: "Node" has no attribute "__iter__" (not iterable)
+ sphinx/environment/adapters/toctree.py: note: In member "get_toctree_for" of class "TocTree":
+ sphinx/environment/adapters/toctree.py:334:9: error: Incompatible types in assignment (expression has type "Node", variable has type "Element")
+ sphinx/ext/autodoc/__init__.py: note: In member "_find_signature" of class "DocstringSignatureMixin":
+ sphinx/ext/autodoc/__init__.py:1184:32: error: Incompatible types in assignment (expression has type "List[str]", variable has type "List[List[str]]")
+ sphinx/ext/autodoc/__init__.py: note: In member "format_signature" of class "ClassDocumenter":
+ sphinx/ext/autodoc/__init__.py:1557:56: error: Argument "parameters" to "replace" of "Signature" has incompatible type "Parameter"; expected "Optional[Sequence[Parameter]]"
+ sphinx/ext/autodoc/__init__.py: note: In member "format_signature" of class "MethodDocumenter":
+ sphinx/ext/autodoc/__init__.py:2137:60: error: Argument "parameters" to "replace" of "Signature" has incompatible type "Parameter"; expected "Optional[Sequence[Parameter]]"
+ sphinx/ext/autodoc/__init__.py: note: In member "import_object" of class "RuntimeInstanceAttributeMixin":
+ sphinx/ext/autodoc/__init__.py:2284:55: error: Argument 2 to "import_object" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/autodoc/__init__.py: note: In member "import_object" of class "UninitializedInstanceAttributeMixin":
+ sphinx/ext/autodoc/__init__.py:2333:51: error: Argument 2 to "import_object" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/autodoc/__init__.py: note: In member "isinstanceattribute" of class "AttributeDocumenter":
+ sphinx/ext/autodoc/__init__.py:2405:51: error: Argument 2 to "import_object" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/napoleon/docstring.py: note: In member "_consume_returns_section" of class "GoogleDocstring":
+ sphinx/ext/napoleon/docstring.py:317:39: error: Unsupported operand types for + ("List[str]" and "str")
+ sphinx/ext/napoleon/docstring.py:319:29: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
+ sphinx/ext/napoleon/docstring.py: note: In member "_fix_field_desc" of class "GoogleDocstring":
+ sphinx/ext/napoleon/docstring.py:380:53: error: Argument 1 to "_get_initial_indent" of "GoogleDocstring" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/napoleon/docstring.py:384:53: error: Argument 1 to "_indent" of "GoogleDocstring" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/napoleon/docstring.py: note: In member "_format_field" of class "GoogleDocstring":
+ sphinx/ext/napoleon/docstring.py:453:45: error: Unsupported operand types for + ("List[str]" and "str")
+ sphinx/ext/napoleon/docstring.py: note: In member "_strip_empty" of class "GoogleDocstring":
+ sphinx/ext/napoleon/docstring.py:856:25: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
+ sphinx/domains/changeset.py: note: In member "clear_doc" of class "ChangeSetDomain":
+ sphinx/domains/changeset.py:127:20: error: "object" has no attribute "docname"
+ sphinx/domains/changeset.py:128:36: error: Argument 1 to "remove" of "list" has incompatible type "object"; expected "ChangeSet"
+ sphinx/directives/code.py: note: In member "pyobject_filter" of class "LiteralIncludeReader":
+ sphinx/directives/code.py:263:25: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
+ sphinx/directives/code.py: note: In member "start_filter" of class "LiteralIncludeReader":
+ sphinx/directives/code.py:310:32: error: Incompatible return value type (got "str", expected "List[str]")
+ sphinx/directives/code.py:315:32: error: Incompatible return value type (got "str", expected "List[str]")
+ sphinx/directives/code.py: note: In member "end_filter" of class "LiteralIncludeReader":
+ sphinx/directives/code.py:338:32: error: Incompatible return value type (got "str", expected "List[str]")
+ sphinx/directives/code.py:343:36: error: Incompatible return value type (got "str", expected "List[str]")
+ sphinx/cmd/quickstart.py: note: In function "main":
+ sphinx/cmd/quickstart.py:538:28: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/cmd/quickstart.py: note: At top level:
+ sphinx/cmd/quickstart.py:604:19: error: Argument 1 to "main" has incompatible type "str"; expected "List[str]"
+ sphinx/cmd/build.py: note: In function "make_main":
+ sphinx/cmd/build.py:196:33: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/cmd/build.py:199:36: error: Argument 1 to "run_make_mode" has incompatible type "str"; expected "List[str]"
+ sphinx/cmd/build.py: note: In function "build_main":
+ sphinx/cmd/build.py:202:34: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/cmd/build.py: note: In function "main":
+ sphinx/cmd/build.py:287:28: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/cmd/build.py: note: At top level:
+ sphinx/cmd/build.py:298:19: error: Argument 1 to "main" has incompatible type "str"; expected "List[str]"
+ sphinx/cmd/make_mode.py: note: In function "run_make_mode":
+ sphinx/cmd/make_mode.py:163:35: error: Argument 3 to "Make" has incompatible type "str"; expected "List[str]"
+ sphinx/writers/text.py: note: In member "rewrap" of class "Table":
+ sphinx/writers/text.py:188:17: error: Unsupported target for indexed assignment ("int")
+ sphinx/writers/text.py:188:49: error: Value of type "int" is not indexable
+ sphinx/writers/text.py: note: In function "__str__":
+ sphinx/writers/text.py:208:13: error: Need type annotation for 'width'
+ sphinx/writers/text.py:208:43: error: Argument 1 to "enumerate" has incompatible type "int"; expected "Iterable[<nothing>]"
+ sphinx/writers/text.py: note: In member "__str__" of class "Table":
+ sphinx/writers/text.py:246:51: error: Argument 2 to "cell_width" of "Table" has incompatible type "int"; expected "List[int]"
+ sphinx/__main__.py:15:15: error: Argument 1 to "main" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/apidoc.py: note: In function "main":
+ sphinx/ext/apidoc.py:400:28: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/domains/cpp.py: note: In member "describe_signature" of class "ASTNestedName":
+ sphinx/domains/cpp.py:773:49: error: Argument 1 to "len" has incompatible type "object"; expected "Sized"
+ sphinx/domains/cpp.py:777:32: error: Argument 1 to "len" has incompatible type "object"; expected "Sized"
+ sphinx/domains/cpp.py:778:23: error: Value of type "object" is not indexable
+ sphinx/domains/cpp.py: note: In member "get_id" of class "ASTTemplateArgs":
+ sphinx/domains/cpp.py:1714:22: error: Item "ASTTemplateArgConstant" of "Union[ASTType, ASTTemplateArgConstant]" has no attribute "__iter__" (not iterable)
+ sphinx/domains/cpp.py:1714:22: error: Item "ASTType" of "Union[ASTType, ASTTemplateArgConstant]" has no attribute "__iter__" (not iterable)
+ sphinx/domains/cpp.py: note: In member "_symbol_lookup" of class "Symbol":
+ sphinx/domains/cpp.py:4192:21: error: "ASTNestedNameElement" has no attribute "__iter__" (not iterable)
+ sphinx/domains/c.py: note: In member "describe_signature" of class "ASTNestedName":
+ sphinx/domains/c.py:215:49: error: Argument 1 to "len" has incompatible type "object"; expected "Sized"
+ sphinx/domains/c.py:219:32: error: Argument 1 to "len" has incompatible type "object"; expected "Sized"
+ sphinx/domains/c.py:220:25: error: Value of type "object" is not indexable
+ sphinx/domains/c.py: note: In member "_symbol_lookup" of class "Symbol":
+ sphinx/domains/c.py:1739:22: error: "ASTIdentifier" has no attribute "__iter__"; maybe "__str__"? (not iterable)
+ sphinx/ext/intersphinx.py:431:23: error: Argument "argv" to "inspect_main" has incompatible type "str"; expected "List[str]"
+ sphinx/ext/autosummary/__init__.py: note: In function "extract_summary":
+ sphinx/ext/autosummary/__init__.py:516:19: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
+ sphinx/ext/autosummary/generate.py: note: In function "generate_autosummary_docs":
+ sphinx/ext/autosummary/generate.py:365:26: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
+ sphinx/ext/autosummary/generate.py:365:48: error: Unsupported operand types for + ("str" and "List[str]")
+ sphinx/ext/autosummary/generate.py: note: In function "main":
+ sphinx/ext/autosummary/generate.py:619:28: error: Incompatible default for argument "argv" (default has type "str", argument has type "List[str]")
+ sphinx/writers/latex.py: note: In member "__init__" of class "LaTeXTranslator":
+ sphinx/writers/latex.py:322:17: error: "str" has no attribute "remove"
+ sphinx/writers/latex.py:335:21: error: "str" has no attribute "remove"
+ sphinx/writers/latex.py: note: In member "depart_row" of class "LaTeXTranslator":
+ sphinx/writers/latex.py:965:25: error: "bool" has no attribute "index"


aiohttp (https://github.com/aio-libs/aiohttp.git)
+ aiohttp/connector.py:127: error: "Callable[[], None]" has no attribute "__iter__" (not iterable)  [attr-defined]
+ aiohttp/web.py:586: error: Argument 1 to "main" has incompatible type "str"; expected "List[str]"  [arg-type]

[hatal175]

Not entirely sure where most of these come from (first diff excluded). Can anyone help?

[JelleZijlstra]

Seems like mypy picks the wrong list.__getitem__ overload when you do list[:], but I can't reproduce that writing similar code locally, or even running mypy master directly on Sphinx with your code.

[hauntsaninja]

So this is because mypy_primer isn't really updated for the modular typeshed changes. For some projects, we point mypy at a Python executable in a project venv (using --python-executable), so it can look at the project's dependencies. Unfortunately, as of modularised typeshed, when this happens there's no longer a types-typing-extensions to look at (that's only present in mypy's venv, since types-typing-extensions is now a mypy dependency)

I think there's a pretty strong case to be made that we should just pretend typing_extensions is part of the stdlib (mentioned previously in #5109 (comment)). At least from a mypy user's perspective, it's super unintuitive that passing --python-executable can cause complete mayhem for normal usage of stdlib, even if your project doesn't use typing_extensions at all.
Thoughts? cc @JukkaL @srittau

Here's a minimal repro (where ~/dev/typeshed is checked out to this PR):

~/tmp/repro λ python3 -m venv mypyenv
~/tmp/repro λ mypyenv/bin/pip install git+https://github.com/python/mypy.git
...
~/tmp/repro λ python3 -m venv projectenv
~/tmp/repro λ vim x.py
~/tmp/repro λ cat x.py
from typing import List
x: List[str]
x = x[:]
~/tmp/repro λ mypyenv/bin/mypy --python-executable projectenv/bin/python x.py
Success: no issues found in 1 source file
~/tmp/repro λ mypyenv/bin/mypy --python-executable projectenv/bin/python --custom-typeshed-dir ~/dev/typeshed x.py
x.py:4: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
Found 1 error in 1 file (checked 1 source file)

Here's how you can repro mypy_primer locally:

mypy_primer --new 2ac722ac7 --old 2ac722ac7 --custom-typeshed-repo ~/dev/typeshed --new-typeshed 479fa0e3 --old-typeshed f44f38fd --debug --output concise -k sphinx

Here's a quick and dirty patch for mypy_primer, if people feel badly about giving typing_extensions honourary stdlib status:

diff --git a/mypy_primer.py b/mypy_primer.py
index 6b3d5ca..819377d 100644
--- a/mypy_primer.py
+++ b/mypy_primer.py
@@ -345,6 +345,7 @@ class Project:
         if self.pip_cmd:
             assert "{pip}" in self.pip_cmd
             venv.create(self.venv_dir, with_pip=True, clear=True)
+            await run("{pip} install 'types-typing-extensions>=3.7.0'".format(pip=str(self.venv_dir / "bin" / "pip")), shell=True, cwd=repo_dir)
             await run(
                 self.pip_cmd.format(pip=str(self.venv_dir / "bin" / "pip")),
                 shell=True,

[JelleZijlstra]

@hauntsaninja thanks for figuring it out! I agree that putting typing-extensions in the stdlib makes sense. It would be very confusing if users ran into this issue whenever we use typing-extensions in stdlib stubs.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore.git)
- tests/wasm/json2mc.py:103: note:     def __getitem__(self, _SupportsIndex) -> Any
+ tests/wasm/json2mc.py:103: note:     def __getitem__(self, SupportsIndex) -> Any
- tests/wasm/json2mc.py:159: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "_SupportsIndex"
+ tests/wasm/json2mc.py:159: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "SupportsIndex"
- tests/wasm/json2mc.py:161: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "_SupportsIndex"
+ tests/wasm/json2mc.py:161: error: Invalid index type "Optional[int]" for "List[Module]"; expected type "SupportsIndex"