Precompile common regular expressions

[brettlangdon]

For very large tox.ini configurations there is a lot of time spent calling re._compile for static regular expressions. By precompiling these common/static regular expressions I was able to get our tox -l run from 7.79s to 6.63s

Example of our tox.ini

$ python ./src/tox/__main__.py -c ~/example/tox.ini -l | wc -l
1199

$ time python ./src/tox/__main__.py -c ~/example/tox.ini -l > /dev/null
python ./src/tox/__main__.py -c ~/example/tox.ini -l > /dev/null  7.79s user 0.97s system 107% cpu 8.122 total

$ time python ./src/tox/__main__.py -c ~/example/tox.ini -l > /dev/null
python ./src/tox/__main__.py -c ~/example/tox.ini -l > /dev/null  6.63s user 0.95s system 109% cpu 6.924 total

Thanks for contributing a pull request!

If you are contributing for the first time or provide a trivial fix don't worry too
much about the checklist - we will help you get started.

Contribution checklist:

(also see CONTRIBUTING.rst for details)

• wrote descriptive pull request text
• added/updated test(s)
• updated/extended the documentation
• added relevant issue keyword
  in message body
• added news fragment in changelog folder
  • fragment name: <issue number>.<type>.rst for example (588.bugfix.rst)
  • <type> is must be one of bugfix, feature, deprecation,breaking, doc, misc
  • if PR has no issue: consider creating one first or change it to the PR number after creating the PR
  • "sign" fragment with "by :user:<your username>"
  • please use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files - by :user:superuser."
  • also see examples
• added yourself to CONTRIBUTORS (preserving alphabetical order)

[asottile]

I highly doubt your performance numbers, for instance re.search is defined as _compile(pattern).search(...) which goes through a cache

[brettlangdon]

The results I see locally with this change are pretty consistent, dropping about 1s off execution time.

_compile is using a cache, but also adds the indirection of an additional function call plus a lookup in the cache which isn't needed. It adds up, especially on a large file with a ton of envs. For example for our config it is called 1.6M times.

Before

Tue Jun 23 20:14:30 2020    ./tox.profile

         25212245 function calls (24405995 primitive calls) in 13.372 seconds

   Ordered by: internal time
   List reduced from 1888 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1631094    0.679    0.000    1.065    0.000 re.py:289(_compile)
   421805    0.581    0.000    8.473    0.000 __init__.py:1610(factor_line)
   389147    0.536    0.000    4.538    0.000 __init__.py:1440(_expand_envstr)
   391925    0.510    0.000    2.319    0.000 __init__.py:1445(expand)

After

Tue Jun 23 20:16:40 2020    ./tox.profile

         20402481 function calls (19596231 primitive calls) in 11.232 seconds

   Ordered by: internal time
   List reduced from 1886 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   421805    0.511    0.000    6.337    0.000 __init__.py:1615(factor_line)
   389147    0.497    0.000    3.036    0.000 __init__.py:1445(_expand_envstr)
    23057    0.458    0.000    0.530    0.000 shlex.py:133(read_token)
   391925    0.455    0.000    1.308    0.000 __init__.py:1450(expand)
   781084    0.418    0.000    0.418    0.000 {method 'split' of 're.Pattern' objects}

[asottile]

[gaborbernat]

macOs isn't happy but seems unrelated to this PR.