Treat ascii-only literals as str, not unicode when using unicode_literals

[JukkaL]

Many typeshed stubs only accept str when ascii-only unicode values also work at runtime. This causes a lot of problems with code using unicode_literals, since generally most literals will have unicode as their type. A quick fix would be to infer str as the type for ascii-only string literals when using unicode_literals, but only then. This would cause some unsafety, but it may be worth it for users using unicode_literals. The type safe solution is to manually check every str argument type in the entire typeshed and potentially replace it with unicode or similar, but this would be extremely painful and unlikely to happen any time soon. This would have no effect on code that doesn't use unicode_literals.

Example of how this would work:

from __future__ import unicode_literals

reveal_type('foo')   # str
reveal_type('ß')   # unicode

Some open questions:

• Should u'foo' always have type unicode?
• What's the inferred type of x in x = 'foo'?

[ilevkivskyi]

I am not sure about the two questions, but it looks like supporting this in typed_ast will be very simple. I can just add a flag (e.g. can_be_bytes) to the Str AST node that will be only set if unicode_literals future import is used and all chars are ASCII (very similar to how I made the has_b flag). This flag can then be used in fastparse2 to decide whether to convert this node to StrExpr or UnicodeExpr.

[manu-chroma]

Hey our project cwltool has been mypy since some time and it has been really helpful.

We are trying to adpot use of unicode_literals in our codebase (common-workflow-language/cwltool#442) but it's breaking mypy badly. I'm able to handle issues related stdlibs such as urllib and json API by slightly modifying stubs and adding into our typeshed folder.

But I'm not able to figure out how to deal with these issues.

cwltool/process.py:436: error: No overload variant of "get" of "Mapping" matches argument types [builtins.unicode, builtins.list[<nothing>]]
cwltool/workflow.py:277: error: No overload variant of "get" of "Mapping" matches argument types [builtins.unicode, builtins.unicode]
cwltool/workflow.py:280: error: No overload variant of "get" of "Mapping" matches argument types [builtins.unicode]
cwltool/main.py:232: error: Invalid index type "unicode" for Dict[str, Any]; expected type "str"
cwltool/main.py:489: error: Argument 1 to "__delitem__" of "dict" has incompatible type "unicode"; expected "str"

Can you suggest any workaround until this issue is resolved and new release is made?
Thanks.

[gvanrossum]

TBH I would strongly advise against using `from __future__ import unicode_literals`. It can be very confusing, and the issues you are running into here are caused by invariance of Dict, see http://mypy.readthedocs.io/en/latest/common_issues.html#invariance-vs-covariance (tl;dr: Dict[unicode, X] is not compatible with Dict[str, X]).

[manu-chroma]

Thanks @gvanrossum. Though from __future__ import unicode_literals helped so far, we will drop it and proceed with explicitly use of u"..". It's doable and should fix mypy issues.

[eric-wieser]

Many typeshed stubs only accept str when ascii-only unicode values also work at runtime.

On both python 2 and 3, or just 2?

[gvanrossum]

This is entirely a Py2 thing.

[JukkaL]

This doesn't worth fixing since unicode_literals will hopefully soon be a thing of the past.