type annotation for docutils is incomplete

[tk0miya]

It seems third_party/3/docutils contains only very few classes and modules of docutils.
https://github.com/python/typeshed/tree/master/third_party/3/docutils

For example, the only one class reference is defined at nodes.pyi. But actual docutils.nodes module provides many classes and functions:

>>> import docutils.nodes
>>> dir(docutils.nodes)
['Admonition', 'BackLinkable', 'Bibliographic', 'Body', 'Decorative', 'Element', 'FixedTextElement', 'General', 'GenericNodeVisitor', 'Inline', 'Invisible', 'Labeled', 'Node', 'NodeFound', 'NodeVisitor', 'Part', 'PreBibliographic', 'Referential', 'Resolvable', 'Root', 'Sequential', 'SkipChildren', 'SkipDeparture', 'SkipNode', 'SkipSiblings', 'SparseNodeVisitor', 'Special', 'StopTraversal', 'Structural', 'Targetable', 'Text', 'TextElement', 'Titular', 'TreeCopyVisitor', 'TreePruningException', '__builtins__', '__cached__', '__doc__', '__docformat__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_add_node_class_names', '_call_default_departure', '_call_default_visit', '_non_id_at_ends', '_non_id_chars', '_non_id_translate', '_non_id_translate_digraphs', '_nop', 'abbreviation', 'acronym', 'address', 'admonition', 'attention', 'attribution', 'author', 'authors', 'block_quote', 'bullet_list', 'caption', 'caution', 'citation', 'citation_reference', 'classifier', 'colspec', 'comment', 'compound', 'contact', 'container', 'copyright', 'danger', 'date', 'decoration', 'definition', 'definition_list', 'definition_list_item', 'description', 'docinfo', 'doctest_block', 'document', 'dupname', 'emphasis', 'ensure_str', 'entry', 'enumerated_list', 'error', 'field', 'field_body', 'field_list', 'field_name', 'figure', 'footer', 'footnote', 'footnote_reference', 'fully_normalize_name', 'generated', 'header', 'hint', 'image', 'important', 'inline', 'label', 'legend', 'line', 'line_block', 'list_item', 'literal', 'literal_block', 'make_id', 'math', 'math_block', 'node_class_names', 'note', 'option', 'option_argument', 'option_group', 'option_list', 'option_list_item', 'option_string', 'organization', 'os', 'paragraph', 'pending', 'problematic', 'pseudo_quoteattr', 'raw', 're', 'reference', 'reprunicode', 'revision', 'row', 'rubric', 'section', 'serial_escape', 'sidebar', 'status', 'strong', 'subscript', 'substitution_definition', 'substitution_reference', 'subtitle', 'superscript', 'sys', 'system_message', 'table', 'target', 'tbody', 'term', 'tgroup', 'thead', 'tip', 'title', 'title_reference', 'topic', 'transition', 'types', 'unicodedata', 'version', 'warning', 'warnings', 'whitespace_normalize_name']

As a result, mypy fails for hinting.

from docutils import nodes

def hello(node):
    # type: (nodes.Node) -> None
    pass

$ mypy foo.py
foo.py:3: error: Name 'nodes.Node' is not defined

[tk0miya]

Is there anything I can do? If needed, I'll try to create stubs.

[JelleZijlstra]

Thanks for reporting this! Yes, sounds like we just need to write more stubs. PRs would be appreciated.

[tk0miya]

I'm reading CONTRIBUTING.md now. Did you already get permission from docutils project?
If done, I'll try to do it.

[JelleZijlstra]

I'm not sure actually—it looks like the docutils stubs were added back in 2015 (337abed), when typeshed's norms weren't as clearly established. Perhaps @matthiaskramm remembers; I couldn't find anything on a quick search of the docutils mailing lists.

[matthiaskramm]

337abed merged pytype's and mypy's stubs. The docutils stubs go back to 2014, when mypy still had its own pyi files.
(See e.g. python/mypy@81d515c)

[jdufresne]

Until more complete stubs can be generated, would it be reasonable to delete the nodes.pyi file? It is extremely incomplete (just one file so not that useful) and results in the Name 'nodes.Node' is not defined described above. What do you think?

[danieleades]

not sure if this is the right place for this, but i'm having trouble getting mypy to perform typechecking of the reference class.

i'm working through the sphinx extension 'todo' tutorial here - https://www.sphinx-doc.org/en/master/development/tutorials/todo.html

but mypy is none too happy about the docutils types-

...
# Create a reference
newnode = nodes.reference("", "")
innernode = nodes.emphasis(_("here"), _("here"))
newnode["refdocname"] = todo_info["docname"]  # error: Unsupported target for indexed assignment ("reference")
...

now, looking at the docutils source code, this should be ok.

reference is derived from the Element class, which implements __getitem__ and __setitem__, so it should be fine. Indeed at runtime it is fine.

why is mypy failing to recognise the base class?

i see only this stub - https://github.com/python/typeshed/blob/4d734e38dde295e93c27efd6ead3bb5c05d03fe5/stubs/docutils/docutils/nodes.pyi, but it doesn't appear to include the base classes.

(I don't know anything about these stub files by the way)

[Akuli]

The docutils stubs were added in #5192, and lots of things are missing from them, including the whole Element class. Feel free to create pull requests that improve them. Let us know if you need help with that.

[danieleades]

The docutils stubs were added in #5192, and lots of things are missing from them, including the whole Element class. Feel free to create pull requests that improve them. Let us know if you need help with that.

i need help with that

can you point me to any references on how to write stub files? best practices?

[danieleades]

seems like it should be possible (in principle) to generate stub files from the source code. is there a tool for that?

[Akuli]

stubgen is a tool that generates stubs from source code, but before we rewrite the stubs with it, maybe @srittau should tell us what exactly he did in #5192. Afaik stubgen shouldn't result in so many classes missing.

Stub file syntax is a subset of Python syntax, so there isn't much to learn. Best practices etc are explained in CONTRIBUTING.md.

[danieleades]

Will try and find some time over the next couple of days to investigate