On mypy libraries

[o11c]

I notice that a lot of current and future code in the mypy repository is not strictly related to mypy's core goal of being a static type-checker. You can detect these by the fact that they have few, if any, imports from the rest of mypy.

It would be cleaner if these were segregated to a separate package, especially for unit-testing purposes.

Current examples of nodes that are, or are leaning, this way:

• codec: Already a subpackage, which is documented to be used in a certain way, but it should be simple to keep it minimally compatible even if we figure out how to autoregister encodings.
• errors: I am planning a full rewrite (in a new mypy.syntax package) to add column and caret support, and possibly also other output formats for machine consumption. Some of this logic is obviously mypy-specific; other parts would be useful to others.
• messages: It's not clear to me why this exists at all right now, but if it continues to exist, it ought to be a simple wrapper around gettext. Clearer vision is required to see whether or not it will have any value on its own. See also Remove obsolete messages from mypy.messages #242
• myunit: I am currently doing related cleanup as part of Fix/Improve Test Driver #721 (I need a separate module to run it as __main__ since it is also imported-from), but it would be nice if mypy tests could be run entirely by a standard test framework (I find pytest more convenient in that you don't have to arbitrarily add classes and can do proper fixtures instead of plain setup/teardown). This would make it easier to, say, collect coverage, run subsets of tests, and run tests in parallel (the current parallelization in Fix/Improve Test Driver #721 is a hack and very coarse)
• stubgen: Ought to be a subpackage at least, and separate out the __main__ logic from the look-at-python logic.
• util: Although not mypy-specific, does not stand on its own. Leave it here, every project has a garbage module.

And stuff I have created or am working on:

• waiter: a module I just added as a hack to make testing less intolerable. Only truly effective on Linux with Python 3.3 or later. Might be acting funny (but correct) on Travis, but I'm not sure. The details might change significantly if we stop running on the legacy infrastructure.
• myargs: an idea I have to make implement argument parsing in a typesafe and readable manner, by introspecting the function/constructor annotations (with typing.get_type_hints - should it defer to __init__ automatically if passed a class and __call__ if passed any other Callable?).

Possible approaches for what to do with general-purpose modules:

1. Do nothing and keep a big jumbled mess.
2. Move them to a single mypylibs package, e.g. mypylibs.myunit. (Name subject to bikeshedding; still will be the same pip package though).
3. Move them to under a mypy.libs subpackage, e.g. mypy.libs.myunit. (Name subject to bikeshedding).
4. Make them all subpackages of mypy, even if they would only have an __init__.py.
5. Move them to top level, e.g. myunit.
6. Move big ones to top level and keep the small ones under mypy.
7. Move big ones to top level and keep the small ones under mypylibs.
8. Move big ones to top level and keep the small ones under mypy.libs.

My preference is 2, but keep open the possibility of 7. Any such top-level packages should keep the my* pattern.

[o11c]

Related: should mypy.myunit be importable in Python2? It would certainly simplify. But currently the only justification is if I port the codecs testsuite to use myunit instead of unittest (to avoid having to add a special case to travis.py), and it just feels weird to require the codecs to work in order to test the codecs.

For now I'll go with the exception, but it is something to think about.

[JukkaL]

I've decided to do 9: don't do anything for now, but we may return to this later after mypy is (closer to) production quality.

Mypy is a small enough project that splitting it further probably won't improve things much (if at all), but moving things around will likely confuse people and slow down progress.

To make my point clear: I want to focus on things like improving Python, library and PEP 484 compatibility, fixing bugs, improving usability, improving documentation, improving performance and adding tests -- not on restructuring the mypy implementation. I'm much happier with the quality of the mypy code base than most projects I've worked on and it has the benefit that a bunch of people can already find their way around it.

Smaller refactorings such as splitting individual modules or cleaning up some functions is 100% okay but should be discussed first to confirm that it will be a net benefit. Each refactoring also has a cost associated (somebody has to do it, somebody has to review it, it may break stuff, people have to learn how the new thing works, open branches won't merge cleanly, etc.).