Integrate typeshed into mypy

[matthiaskramm]

I started working on a pull request for this, so I'm opening this issue to discuss details and track progress.

The idea is to

1. Add typeshed (http://github.com/python/typeshed) as a submodule to mypy, e.g. as "./typeshed/" or "./stubs/", and then to tweak default_lib_path in mypy/build.py to work with the typeshed directory structure. (Which is a bit more fine-grained than the one in mypy. It's documented at
   https://github.com/python/typeshed/blob/master/README.md)
2. Merge the stubs of typeshed and mypy. Both have things the other doesn't have - typeshed has a lot of stubs from C extensions, including obscure things like _sre, strop, syslog and cmath, whereas mypy already also has stubs for the .py part of the standard library. (But a first step might be to make a branch in typeshed that's essentially a copy of mypy/stubs/ just to make sure the submodule+loading logic works.)

[refi64]

AAAAAH!!!! Submodules are "theoretically" good but bite where it hurts when it comes to larger repos.

[matthiaskramm]

I've never had any issues with submodules. They're exactly the right solution for the problem at hand (wanting to extract out and share a piece of functionality and maintain it independently).

[o11c]

With recent versions of git, it's a lot harder to screw up submodules. If you're afraid of them because of experience with old versions of git, your knowledge is obsolete.

[o11c]

Likely add some sort of directory helper method here: o11c@bb2daf5

[JukkaL]

This could be really nice. At least merging the stubs is a no-brainer and I haven't done it yet only because of lack of time.

However, I've had some bad experiences with submodules myself (using older versions of git, I admit). I'll play around with them a little and see if I have any objections any more.

An alternative would be to import the contents of typeshed periodically to the mypy repo as commits. This would have its own set of issues.

[matthiaskramm]

The three alternatives typically are:

1. manual copying
2. submodules
3. subtrees (https://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html)

In the past, I've tried all three and found submodules the most convenient and the least error-prone (mostly because they prevent you from changing things in the wrong repository), but your mileage might vary. I'll be perfectly happy to use whatever setup you decide to go with.

[jhance]

Submodules are not bad per-se. The problem that comes with them is when people want to use them as a replacement for SVN partial checkouts.

I think in this scenario a submodule fits the problem nicely.

[JukkaL]

I added some comments to #884. Here is another workflow which is more work to support but may give a better experience in the most common workflows for new contributors. I don't really care that much about the workflows for core contributors such as me as we can figure it out or script it, but for new contributors I'd like to give an experience that is as close as possible to a vanilla git(hub) experience.

The idea goes like this:

• Mypy repo will have copies of all stubs, exactly like now.
• There will be scripts for pushing changes to stubs from mypy to typeshed, and vice versa. These will be mostly run by core developers instead of casual individual contributors. Dealing with local stub changes that aren't supposed to be pushed to typeshed will be a little tricky.

Now all the existing mypy workflows will work exactly like they currently do, plus there are two additional workflows, one for pushing stub changes from mypy to typeshed and the other for pulling changes from typeshed to mypy.

Benefits over submodule:

• Simpler workflow for (non-core) mypy contributors that is exactly like a vanilla github workflow.
• Full commit history for the entire mypy repo, including stubs.
• Can easily have commits that target both stubs and non-stub code. Merging and rebasing works without issues.
• No submodules!

Costs over submodule:

• More work to set up and maintain for core developers. We need to write scripts and run them periodically.
• It'll be more likely that mypy and typeshed will be periodically out of sync. We'll probably want multiple people who can update stubs so that there is no bottleneck. I can give somebody else push access to mypy so that they can review and merge PRs targeting stubs when I'm not available.
• The commit histories of typeshed and mypy will be similar but not the same.

I'm not really sure whether this will be worth it. I'll ask around and try to get more opinions on git submodules.

[o11c]

I very very strongly recommend against a git subtree-style workflow. Instead, we should just add a call to git submodule update --init in runtests.py so that people who don't have the recursive clone get something working - or possibly use a separate script and just warn (nicer for cases our script hasn't thought of).

We want stubs to be committed to typeshed early, even if their PR branch is not ready. PR branches should not contain submodule updates, those should be periodically done in core (and we can have our scripts encourage people to use typeshed master rather than the commit the submodule points to - or perhaps offer tested and latest variants).

I created #903 to make life easier for me, not for anybody else.

[matthiaskramm]

o11c: I wouldn't run any git commands in runtests.py, just out of principle. If anything, it should detect a missing submodule and output a nice message with instructions what to do.
Also, git by default initializes all submodules upon checkout.

JukkaL: If you allow PRs that cause mypy and typeshed to become out of sync, you (and other core contributors) will take on the extra workload to bring them back into sync. Sure, scripts can help, but it still means extra work for potentially every PR - work that normally would/should be done by the contributor.
It seems the key missing feature for the submodule approach is to have local stub changes in mypy. A possible solution might be a high-precedence "overload/" directory in mypy that contains stubs that are (temporarily) mypy specific.

[matthiaskramm]

Correction: git will not automatically initialize submodules, at least not on all Linux distributions. (E.g. on Gentoo git does automatic submodule cloning, but Ubuntu it doesn't)
But you can use "git clone --recurse-submodules", or add "[clone] recurse_submodules = yes" to your .gitconfig. That feature was added in git 1.6.5.

[jhance]

E.g. on Gentoo git does automatic submodule cloning

No, it does not.