Add stubtest with lots of errors (currently) (#1560)

* Add stubtest with lots of errors

* Update test.yml

* Add -ignore-positional-only for now

* Fix CI

* Fix CI

* [pre-commit.ci] auto fixes from pre-commit.com hooks

* Fix CI

* Address review

* Update `allowlist_todo.txt`

* Update CONTRIBUTING.md

Co-authored-by: Petter Friberg <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Petter Friberg <[email protected]>

Author: 3 people

.github/workflows/test.yml

@@ -64,6 +64,29 @@ jobs:
       - name: Run tests
         run: PYTHONPATH='.' pytest
 
+  stubtest:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.11']
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install binutils libproj-dev gdal-bin
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          pip install -U pip setuptools wheel
+          pip install -r ./requirements.txt
+
+      - name: Run stubtest
+        run: bash ./scripts/stubtest.sh
+
   matrix-test:
     runs-on: ubuntu-latest
     strategy:

CONTRIBUTING.md

@@ -7,6 +7,7 @@ This project is open source and community driven. As such we encourage code cont
 3. Improve plugin code and extend its capabilities
 4. Write tests
 5. Update dependencies
+6. Fix and remove things from our `stubtest/allowlist_todo.txt`
 
 ## Tutorials
 
@@ -76,23 +77,19 @@ If you get some unexpected results or want to be sure that tests run is not affe
 rm -r .mypy_cache
 ```
 
+### Testing stubs with `stubtest`
 
-### Generating Stubs using Stubgen
+Run `bash ./scripts/stubtest.sh` to test that stubs and sources are in-line.
 
-The stubs are based on auto-generated code created by Mypy's stubgen tool (see: [the stubgen docs](https://mypy.readthedocs.io/en/stable/stubgen.html)).
-To make life easier we have a helper script that auto generates these stubs. To use it you can run:
+We have two special files to allow errors:
+1. `scripts/stubtest/allowlist.txt` where we store things that we really don't care about: hacks, django internal utility modules, things that are handled by our plugin, things that are not representable by type system, etc
+2. `scripts/stubtest/allowlist_generate.txt` where we store all errors there are right now. Basically, this is a TODO list: we need to work through this list and fix things (or move entries to real `allowlist.txt`). In the end, ideally we can remove this file
 
-```bash
-python ./scripts/stubgen-django.py --django_version 3.2
-```
-
-You can also pass an optional commit hash as a second kwarg to checkout a specific commit, e.g.
+You might also want to disable `incremental` mode while working on `stubtest` changes.
+This mode leads to several known problems (stubs do not show up or have strange errors).
 
-```bash
-python ./scripts/stubgen-django.py --django_version 3.2 --commit_sha <commit_sha>
-```
+**Important**: right now we only run `stubtest` on Python 3.11 (because it is the latest released version at the moment), any other versions might generate different outputs. Any work to create per-version allowlists is welcome.
 
-The output for this is a gitignored folder called "stubgen" in the repo's root.
 
 ## Submission Guidelines
 

scripts/stubtest.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+# Run this script as `bash ./scripts/stubtest.sh`
+
+set -e
+
+export MYPYPATH='.'
+
+# TODO: remove `--ignore-positional-only` when ready
+stubtest django \
+    --mypy-config-file mypy.ini \
+    --ignore-positional-only \
+    --allowlist scripts/stubtest/allowlist.txt \
+    --allowlist scripts/stubtest/allowlist_todo.txt \
+    --generate-allowlist

scripts/stubtest/allowlist.txt

@@ -0,0 +1,15 @@
+# This is a true allow list with things that we really don't care about.
+# `stubtest.generated.txt` is autogenerated by `stubtest --generate-allowlist`
+# and might cointain actual problems and things that we *do want* to fix.
+#
+# Please, move things here when you are sure that they really should be ignored.
+# Comments about why things are ignored are mandatory.
+
+# We don't need locales:
+django.conf.locale.*
+
+# We don't need migrations:
+django.contrib.admin.migrations.*
+django.contrib.auth.migrations.*
+django.contrib.flatpages.migrations.*
+django.contrib.contenttypes.migrations.*