WARNING: Cannot treat a function defined as a local function

[newAM]

With python 3.8.1 documenting a dataclass leads to this warning:

WARNING: Cannot treat a function defined as a local function: "mod.Foo"  (use @functools.wraps)

This only occurs in python 3.8.1, python 3.8.0 does not show this warning.

This is minimally reproducible with 3 files:

conf.py

project = "unit_test"
extensions = ["sphinx.ext.autodoc", "sphinx_autodoc_typehints"]
source_suffix = ".rst"
master_doc = "doc"

doc.rst

Doc
###

.. automodule:: mod
    :members:

mod.py

from dataclasses import dataclass


@dataclass
class Foo:

    pass

Command used:

python3.8 -m sphinx . build

Relevant versions:

Sphinx==2.3.1
sphinx-autodoc-typehints==1.10.3

[newAM]

I cloned the repo to debug further, but the unit tests seem to be failing out-of-the-gate for me:

test_sphinx_output[False]

_________________________________________________________________________________________________________________ test_sphinx_output[False] __________________________________________________________________________________________________________________
tests/test_sphinx_autodoc_typehints.py:505: in test_sphinx_output
    assert text_contents == expected_contents
E   assert 'Dummy Module...-- function\n' == 'Dummy Module...-- function\n'
E     Skipping 3852 identical leading characters in diff, use -v to show
E     - __init__() -> None
E     + __init__()
E       
E             Return type:
E                "None"
E       ...
E     
E     ...Full output truncated (14 lines hidden), use '-vv' to show
------------------------------------------------------------------------------------------------------------------ Captured stdout teardown ------------------------------------------------------------------------------------------------------------------
# testroot: root
# builder: text
# srcdir: /tmp/pytest-of-alex/pytest-13/dummy
# outdir: /tmp/pytest-of-alex/pytest-13/dummy/_build/text
# status: 
Running Sphinx v2.3.1
building [mo]: targets for 0 po files that are out of date
building [text]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                 
build succeeded, 4 warnings.

The text files are in ../../../../../tmp/pytest-of-alex/pytest-13/dummy/_build/text.

# warning: 
WARNING: Cannot treat a function defined as a local function: "dummy_module.Class.locally_defined_callable_field"  (use @functools.wraps)
WARNING: Cannot resolve forward reference in type annotations of "dummy_module.function_with_unresolvable_annotation": name 'a' is not defined
WARNING: Cannot treat a function defined as a local function: "dummy_module.DataClass"  (use @functools.wraps)
WARNING: Cannot treat a function defined as a local function: "dummy_module.DataClass.__init__"  (use @functools.wraps)

test_sphinx_output[True]

__________________________________________________________________________________________________________________ test_sphinx_output[True] __________________________________________________________________________________________________________________
tests/test_sphinx_autodoc_typehints.py:505: in test_sphinx_output
    assert text_contents == expected_contents
E   assert 'Dummy Module...-- function\n' == 'Dummy Module...-- function\n'
E     Skipping 3891 identical leading characters in diff, use -v to show
E     - __init__() -> None
E     + __init__()
E       
E             Return type:
E                "None"
E       ...
E     
E     ...Full output truncated (14 lines hidden), use '-vv' to show
------------------------------------------------------------------------------------------------------------------ Captured stdout teardown ------------------------------------------------------------------------------------------------------------------
# testroot: root
# builder: text
# srcdir: /tmp/pytest-of-alex/pytest-13/dummy
# outdir: /tmp/pytest-of-alex/pytest-13/dummy/_build/text
# status: 
Running Sphinx v2.3.1
building [mo]: targets for 0 po files that are out of date
building [text]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                 
build succeeded, 4 warnings.

The text files are in ../../../../../tmp/pytest-of-alex/pytest-13/dummy/_build/text.

# warning: 
WARNING: Cannot treat a function defined as a local function: "dummy_module.Class.locally_defined_callable_field"  (use @functools.wraps)
WARNING: Cannot resolve forward reference in type annotations of "dummy_module.function_with_unresolvable_annotation": name 'a' is not defined
WARNING: Cannot treat a function defined as a local function: "dummy_module.DataClass"  (use @functools.wraps)
WARNING: Cannot treat a function defined as a local function: "dummy_module.DataClass.__init__"  (use @functools.wraps)

This occurs for both 3.7.6 and 3.8.1 with tox.

[newAM]

Looks like this is a breaking change introduced by bpo-34776. The issue tracker mentions it was merged into 3.7 and 3.8, but I do not see mention of it in the 3.7 change-log.

[torfsen]

I just ran into the issue on Python 3.7.6.

[agronholm]

I can repro this but the warnings don't give me much clue about where they originate from or what causes them. Anyone?

[agronholm]

Ok, so this is caused by the __init__() method actually being a local function. This means that data classes require special treatment.

[agronholm]

It's unfortunate that dataclasses.InitVar[] does not store its parameter so it is inaccessible at run time.

[newAM]

I am trying to better understand the changes made in bpo-34776, but reading the dataclasses source is leading to a lot of questions; does anyone know why a string of python is being constructed then exec is being called on it? This style is unfamiliar to me.

def _create_fn(name, args, body, *, globals=None, locals=None,
               return_type=MISSING):
    # Note that we mutate locals when exec() is called.  Caller
    # beware!  The only callers are internal to this module, so no
    # worries about external callers.
    if locals is None:
        locals = {}
    if 'BUILTINS' not in locals:
        locals['BUILTINS'] = builtins
    return_annotation = ''
    if return_type is not MISSING:
        locals['_return_type'] = return_type
        return_annotation = '->_return_type'
    args = ','.join(args)
    body = '\n'.join(f'  {b}' for b in body)

    # Compute the text of the entire function.
    txt = f' def {name}({args}){return_annotation}:\n{body}'

    local_vars = ', '.join(locals.keys())
    txt = f"def __create_fn__({local_vars}):\n{txt}\n return {name}"

    ns = {}
    exec(txt, globals, ns)
    return ns['__create_fn__'](**locals)

[adamtheturtle]

If I ignore this one warning, what will the difference be in my rendered docs between those docs when I am using a Python version which hits this bug and a Python version which does not?

I guess from reading that the __init__ function of a @dataclass class will not be properly rendered.

[newAM]

What will the difference be in my rendered docs between those docs when I am using a Python version which hits this bug and a Python version which does not?

I have seen zero difference, but I do not render __init__ in my docs.

[ravwojdyla]

For those looking for a temp workaround until #153 gets merged, in your sphinx conf.py file:

import logging as pylogging
from sphinx.util import logging

# Workaround for https://github.com/agronholm/sphinx-autodoc-typehints/issues/123
# When this https://github.com/agronholm/sphinx-autodoc-typehints/pull/153
# gets merged, we can remove this
class FilterForIssue123(pylogging.Filter):
    def filter(self, record: pylogging.LogRecord) -> bool:
        # You probably should make this check more specific by checking
        # that dataclass name is in the message, so that you don't filter out
        # other meaningful warnings
        return not record.getMessage().startswith("Cannot treat a function")

logging.getLogger("sphinx_autodoc_typehints").logger.addFilter(FilterForIssue123())
# End of a workaround

I couldn't find a better way to do this with sphinx's suppress_warnings, but there is probably a more idiomatic way to do this.

[cthoyt]

I have a hacky solution for this originally from https://github.com/pykeen/pykeen/blob/5d838af8c88d143420db06ddf994582c27a6d568/src/pykeen/utils.py#L343-L352

def fix_dataclass_init_docs(cls: Type) -> Type:
    """Fix the ``__init__`` documentation for a :class:`dataclasses.dataclass`.

    :param cls: The class whose docstring needs fixing
    :returns: The class that was passed so this function can be used as a decorator

    .. seealso:: https://github.com/agronholm/sphinx-autodoc-typehints/issues/123
    """
    cls.__init__.__qualname__ = f'{cls.__name__}.__init__'
    return cls

Then you apply it as a decorator to your dataclass (you can always just do this to a class directly, but I like the decorator route)

[gaborbernat]

A PR for this would be welcome.

[cthoyt]

A PR for this would be welcome.

If you point me towards the right place to put it, I will give it a shot

[gaborbernat]

First try to create a test to reproduce the issue here https://github.com/tox-dev/sphinx-autodoc-typehints/blob/main/tests/test_sphinx_autodoc_typehints.py#L757, then look at what fails in https://github.com/tox-dev/sphinx-autodoc-typehints/blob/main/src/sphinx_autodoc_typehints/__init__.py and fix it there 👌