Adapt the Python 3.7 AST changes

[Perlence]

Some bytecode tests are expected to fail in Python 3.7.0a1 and 3.7.0a2, because there was a bug that prevented some environ vars from having effect. It was fixed in python/cpython@d7ac061.

Resolves #2818.

[coveralls]

Coverage decreased (-0.007%) to 92.244% when pulling fd7bfa3 on Perlence:rewrite-python-37-docstring into 0b540f9 on pytest-dev:master.

[RonnyPfannschmidt]

i think we need to add python 3.7 to travis to get the needed results

the failures shown are known and will be fixed

[nicoddemus]

I'm trying to understand this, why remove the docstring from the beginning of the AST in the test? If it should not be there, then perhaps we should assert what we expect to be there in >=3.7?

[Perlence]

Module docstring is now (in 3.7) a part of Module node, it's not in the body. So I check if there's docstring expression in Python prior to 3.7 and then remove it so the following body items have the same indexes on both Pythons. This way, for example, m.body[0:2] is a slice of imports.

I didn't like this approach. So I split this test in 3: one without docstrings that works on both versions, one with docstrings that's skipped prior to 3.7, and one with docstrings for versions after 3.7. This looked quite ugly as well to me, so I reverted to this variant.

[nicoddemus]

Module docstring is now (in 3.7) a part of Module node, it's not in the body. So I check if there's docstring expression in Python prior to 3.7 and then remove it so the following body items have the same indexes on both Pythons. This way, for example, m.body[0:2] is a slice of imports.

@Perlence thanks for the detailed explanation, everything is clear to me now.

[nicoddemus]

i think we need to add python 3.7 to travis to get the needed results

We already run Python 3.7 on Travis, that's the "nightly" build.

the failures shown are known and will be fixed

Indeed. I would like to understand better the change to the tests first though.

I propose we make a serious effort of promoting the "nightly" build from Allowed Failures to normal mode only when Python 3.7 reaches beta status.

[methane]

We're thinking change again in 3.7b3. In this time, docstring will be moved from attribute to special AST node named DocString. See this example:

source = """\
def foo():
    "docstring"
"""

import ast
t = ast.parse(source)
print(ast.dump(t))

3.6: ... FunctionDef(name='foo', args=arguments(...), body=[Expr(value=Str(s='docstring'))], decorator_list=[], returns=None) ...

3.7b2: ... FunctionDef(name='foo', args=arguments(...), body=[], decorator_list=[], returns=None, docstring='docstring') ...

3.7b3?: ... FunctionDef(name='foo', args=arguments(args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[DocString(s='docstring')], decorator_list=[], returns=None) ...

Of course, ast.getdocstring() works in all versions.
With this change, you can get lineno of docstring from the DocString node (while there is bug for now).

We want feedback from AST users. Please post your comment on this issue or Python-Dev mailing list.

[nicoddemus]

@methane thanks for the ping!