Treat ASCII-only literals as str when using unicode_literals

[ilevkivskyi]

Fixes #3619

This PR only affects code that uses from __future__ import unicode_literals.

It looks like there is a solution that will not require modifying typed_ast. Note that this will treat ASCII-only literals with an explicit prefix u'foo' as str, not unicode. This can be changed, but only by updating typed_ast (the point is that Str has an attribute has_b indicating the presence of b prefix, we will need also has_u).

I know unicode_literals are bad, but I have seen several people complaining about this recently.
@JukkaL @JelleZijlstra what do you think?

[JelleZijlstra]

Will this work if I do from __future__ import absolute_import, unicode_literals? That works at runtime.

Also, I don't think the asname check is necessary. This program:

from __future__ import unicode_literals as why_would_you_do_this
print(type(''))

prints "unicode".

[ilevkivskyi]

OK, good points. I will relax the check now.

[JelleZijlstra]

I left some comments on the implementation. If it's not too hard to do in typed_ast, I think we should also make sure that strings with an explicit u prefix remain unicode.

I'm also not entirely sure this is a good idea, since it does introduce a difference with how these things actually work at runtime. For example, perhaps somebody is passing ['a', 'list', 'of', 'strings'] to a function that accepts List[unicode], and with this change the list will become a List[str] instead of List[unicode]. We should see if that is an issue in real-world codebases that use unicode_literals; @rowillia could you help check that?

[JelleZijlstra]

Could you add tests for some of the other syntactic variations, like from __future__ import absolute_import, unicode_literals and from __future__ import unicode_literals as something_else?

[ilevkivskyi]

OK, will do.

[ilevkivskyi]

@JelleZijlstra

If it's not too hard to do in typed_ast, I think we should also make sure that strings with an explicit u prefix remain unicode.

This is basically one-two lines (plus some boilerplate, but it is auto-generated). The main problem is that this will require a typed_ast release and updating its stub. I can do this if we agree we need this change.

[ilevkivskyi]

Are we going to push in this direction? If it is not that important, maybe better just close this?

[JukkaL]

This requires more analysis/experimentation to proceed. Also, I think that we'd need to handle the explicit u prefix.

[ilevkivskyi]

Also, I think that we'd need to handle the explicit u prefix.

OK, I will then prepare a typed_ast PR (this PR can be updated however only after the release of typed_ast, otherwise tests will fail).

[ilevkivskyi]

The corresponding typed_ast PR python/typed_ast#49

[ilevkivskyi]

I am closing this since it seems many people don't like unicode_literals. Instead I propose to have a separate discussion about how to use the typed_ast feature we are not using: typed_ast preserves original prefix for string/bytes literals in both Python 2 and 3.

[gvanrossum]

Is that separate discussion somewhere? Because in the end we did release the typed_ast feature (with a lot of delays and some fumblings). Was it all for naught? Or is there actually some use for the kind attribute on Str?

[ilevkivskyi]

I think there was an attempt to use Str.kind for better bytes vs unicode story (in the typing repo IIRC). In mypy it is only currently used for literal types (this is why this feature was released).

[gvanrossum]

Oh, that's a good enough use case for me!