Add ability to report on linecounts of specific flags

[ddfisher]

For example: mypy --config-file mypy_self_check.ini --linecount-report linecounts --linecount-breakdown strict_optional mypy tells me that mypy has 24,054 lines of checked strict Optional code.

[ilinum]

Wow, this is really awesome!

Do you think you can add tests for it in reports.test?

[ilinum]

It's probably better to use tree.fullname() instead of tree._fullname

[gvanrossum]

@ddfisher are you still planning to work on this? This branch is stale and its 2 month anniversary is coming up soon.

[ilinum]

I can take this over if needed. This PR is mostly ready in my opinion -- just needs a couple of tests.

[gvanrossum]

Go for it! (Though fair warning: this isn't going to end up in the 0.530 release.)

[ilinum]

Okay, so after looking into this some more I realized that the way linecount-breakdowns report is presented does not make sense.

So, let's say I run mypy --config-file mypy_self_check.ini --linecount-report linecounts --linecount-breakdown strict_optional mypy, it produces two files: linecounts/linecount.txt and strict_optional-breakdown.txt. These files are identical except for two lines:

linecounts$ diff linecount.txt strict_optional-linecount.txt 
1,2c1
<   42064   42216   2996   3006 total
<    4940    4940    264    264 mypy.semanal
---
>   37124   37276   2732   2742 total

Essentially, strict_optional-linecount.txt is missing the modules that have the provided flag disabled and has a different total.
The only useful thing that linecount_breakdown provides is the total number of lines that are checked with strict-optional and total number that's not.

Here is what I think it should be like: whenever you run mypy with --linecount-report, in addition to the usual linecount.txt it should generate linecount-breakdowns.txt which lists total for all flags that are explicitly enabled or disabled in the config file.
Here is what I think linecount-breakdowns.txt should look like:

400 2400 strict_optional
324 3124 disallow_any=generics

I haven't started implementing it yet. Does this seem like a good idea?

[gvanrossum]

400 2400 strict_optional
324 3124 disallow_any=generics

What do the numbers mean? Are 400 and 324 the number of lines where the flag made a difference and 2400/3124 the number of lines where the flag was set that way? (I.e. the total line count for the files covered by that flag setting.)

I wonder if there's a way to combine this with my idea from #4018.

[ilinum]

I was thinking 400 would be the number of lines checked with strict-optional and 2400 is the number of lines that are checked without strict-optional.

I like your idea but I think it would be too complicated to implement for all flags.
In #4018 you need to implement it separately for every flag. It's easy to implement it if the code is:

if options.flag_name:
    self.fail(...)

Essentially, you would just add a statement before the if statement to mark current module as needing the flag disabled.
But it gets more complicated with flags like strict-optional.

Implementing the breakdowns the way I suggest would be fairly simple:
For every file, check find all flags that are explicitly enabled or disabled by comparing the per-module options to global options. Then, for all explicitly enabled or disabled flags, add the number of lines in the current file to some global count.

[gvanrossum]

Yeah, #4018 doesn't work for --strict-optional because that affects many type checks.

A flag that just counts per flag which files have it on and which files have it off doesn't feel very useful, and reporting it in terms of line counts seems even less appropriate (since it's quite possible that a flag only affects a small fraction of the lines in a file).

Maybe it would make some sense to have a report that printed some numbers for each flag that is on in some files and off in others, e.g.

strict_optional: on in 7 files (400 lines), off in 3 files (2400 lines).

I like having this as a sentence rather than a table, because this way it is self-explanatory, and I don't think that we'll have a lot of other tools that would need to parse such a report.

[gvanrossum]

Is this PR dead now? (I noticed the stale branch just now.)

[ddfisher]

I think my original intention behind this PR was to make it available for our linecount report. Is there a form of this PR that we're still interested in?

[gvanrossum]

I don't have time to look into it myself -- I am offering two choices: close it, or keep it open, and in the latter case you and Stas can finish it at your leisure. Mypy development at Dropbox has taken a different turn so I don't think we're holding our breath for this.

[ddfisher]

As far as I know, no one particularly cared about this except for (previously) mypy at Dropbox, so I think closing this is the right call. Thanks!