fix redis str types

[jtschoonhoven]

Currently the redis client library is typed to assume that methods that return a string-type response are encoded as bytes. However the client provides a decode_responses option that configures the client to automatically decode them to unicode. Manually decoding strings is tedious so this is a popular option. But enabling it currently invalidates the types defined in Typeshed.

This is an attempt to make the default string type configurable using generics and overloads.

The gist of this PR is to make the Redis class a generic that accepts either a Text or bytes type variable. It attempts to select between these automatically by detecting whether encode_responses is True, in which case it self-defines as a Redis[Text] rather than Redis[bytes] (the default).

This is my first Typeshed PR and I'm fairly new to generics and overloads, so comments and suggestions are very welcome.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

- dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, int, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[Union[bytes, float, int, str], Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[bytes, bytes]]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[float, Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[Any, Any]]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:194:9: error: "Redis" has no attribute "pipelineassert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:194:9: error: "Redis[Any]" has no attribute "pipelineassert_called_once_with"
- dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[bytes, bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[Any, Any]]" has no attribute "assert_called_once_with"

+ zerver/lib/redis_utils.py:22: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:26: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:45: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]

[JelleZijlstra]

This seems like a good idea but you'll also need overloads with bool, in case the user passes a variable of type bool that can't be inferred as True or False.

Looking at the primer failures, most of them are just minor changes in the signature, but the zerver ones (which are here: https://github.com/zulip/zulip/blob/master/zerver/lib/redis_utils.py#L22) are real; these mean that users upgrading to this PR and using strict mypy options will need to make some changes. On balance that seems fine though.

[JelleZijlstra]

(pushed the wrong button)

[hauntsaninja]

I'd really like something like https://mail.python.org/archives/list/[email protected]/thread/SZRXAEF2JO5RKOKS7V4JFT7LZOAOOMUU/ / python/typing#307 for these kinds of situations

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

- dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, int, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[Union[bytes, float, int, str], Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[bytes, bytes]]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[float, Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[Any, Any]]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:194:9: error: "Redis" has no attribute "pipelineassert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:194:9: error: "Redis[Any]" has no attribute "pipelineassert_called_once_with"
- dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[bytes, bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[Any, Any]]" has no attribute "assert_called_once_with"

+ zerver/lib/redis_utils.py:22: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:26: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:45: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

- dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:32:30: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis")
+ dragonchain/lib/database/redis_utest.py:38:34: error: Incompatible types in assignment (expression has type "None", variable has type "Redis[Any]")
- dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:110:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, int, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:142:9: error: "Callable[[Union[str, bytes], Union[str, bytes, None], Union[bytes, float, str, None], Optional[Mapping[Union[bytes, float, int, str], Union[bytes, float, int, str]]]], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[Union[bytes, float, int, str], Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[bytes, bytes]]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:146:9: error: "Callable[[Union[float, Iterable[Union[bytes, float, int, str]]], int], Optional[Tuple[Any, Any]]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:154:9: error: "Callable[[Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:170:9: error: "Callable[[Union[str, bytes], Union[str, bytes]], Optional[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[Any]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:194:9: error: "Redis" has no attribute "pipelineassert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:194:9: error: "Redis[Any]" has no attribute "pipelineassert_called_once_with"
- dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[bytes, bytes]]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:210:9: error: "Callable[[Union[str, bytes]], Dict[Any, Any]]" has no attribute "assert_called_once_with"

+ zerver/lib/redis_utils.py:22: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:26: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]
+ zerver/lib/redis_utils.py:45: error: Missing type parameters for generic type "redis.StrictRedis"  [type-arg]

[jtschoonhoven]

@JelleZijlstra updated per your comment.
@hauntsaninja fully agree.

Thanks for your suggestions! LMK if there's any other changes or if you think it's ready to ship.