Skip to content

Enable --[no-]compile which will compile or not compile pyc files #1335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 22, 2013

Conversation

dstufft
Copy link
Member

@dstufft dstufft commented Nov 22, 2013

  • Enables compilation for Wheel files
  • Adds an easy command line for distutils/setuptools compilation options

This fixes #1324

"""
del script.environ["PYTHONDONTWRITEBYTECODE"]
script.pip("install", "--compile", "--no-use-wheel", "INITools==0.2")
assert os.path.exists(script.site_packages_path/"INITools/__init__.pyc")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use imp.cache_from_source

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm doing this instead..

# There are many locations for the __init__.pyc file so attempt to find
    #   any of them
    exists = [
        os.path.exists(script.site_packages_path/"INITools/__init__.pyc"),
    ]

    exists += glob.glob(
        script.site_packages_path/"INITools/__pycache__/__init__*.pyc"
    )

    assert any(exists)

Since imp.cache_from_source isn't available on every Python and some OSs package differently.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, the uninstall code checks if backwardcompat.uses_pycache == True, then uses imp.cache_from_source

* Enables compilation for Wheel files
* Adds an easy command line for distutils/setuptools compilation
  options
@dstufft
Copy link
Member Author

dstufft commented Nov 22, 2013

All tests passed except for PyPy which is known to be broken with 2.2.

dstufft added a commit that referenced this pull request Nov 22, 2013
Enable --[no-]compile which will compile or not compile pyc files
@dstufft dstufft merged commit 9319efc into pypa:develop Nov 22, 2013
@dstufft dstufft deleted the compile-wheel-pyc branch November 22, 2013 04:35
@pfmoore
Copy link
Member

pfmoore commented Nov 22, 2013

This does not seem to put the pyc files into RECORD, so they will not be removed on uninstall. This is bad as it will leave clutter around and could potentially mean that uninstalled modules are still importable (from the pyc files).

I've not tested the details - I only saw this after it was merged. Did anyone check this? I see no test that does so.

At a minimum, if the RECORD handling cannot be fixed easily (does the compile step even say what pyc files are created?) then --no-compile should be the default until a resolution can be found.

@dstufft
Copy link
Member Author

dstufft commented Nov 22, 2013

This is the RECORD for twine installed from Wheel

twine/__about__.py,sha256=A_vrkPkebq02SLPYcwdKxk7R6HZnuuAtwRCeHB-BPuk,1110
twine/__init__.py,sha256=AyNIeZSGwLc5bK0j2DE9aCHOGSlZ7ddv4piOqwrugKA,912
twine/__main__.py,sha256=kcJL2D7oY-8mSEHyZUsYnG712wgjw5JdSC11bAaXRJc,824
twine/application.py,sha256=wfHsPQF8z4skkm8MW-hb3Eli91rHE_sgwZPz7Daqu-w,3063
twine/exceptions.py,sha256=3nX1OB4x_UKyHmQn6Sby6tkxQITZN8KbEYn7KNhZ4sg,683
twine/utils.py,sha256=xY8z3g39yFaPqAndqgTpMaIWNjBHaJcgBwVD7Mp7a_A,1399
twine/wheel.py,sha256=-V0kAWKJknpW-gWoWJRPpSf614CRgFiIkEmBHXSa7qE,2677
twine/commands/__init__.py,sha256=jwqAHlhAJWVIKZCsG0mS-wuXSQY26LfE1WGR5xk0NP0,726
twine/commands/upload.py,sha256=UqEbOlTGgW2kWQdSdpPI-fhrd2O8iHtHuHNx4H_U5s0,7405
twine-1.2.2.data/scripts/twine,sha256=elKIESRsSGMC7ljphQQIFyUSnGTZwLe9JPBlV4zPphw,282
twine-1.2.2.dist-info/DESCRIPTION.rst,sha256=XuDgyW5TFHnm2KgxWvtUoGmyEaYwIoihwnR2QtYJqOA,3305
twine-1.2.2.dist-info/entry_points.txt,sha256=KUfmP4skuIXX7OYYEQUT1EqA-ZtazAri07vWOt9rloU,47
twine-1.2.2.dist-info/METADATA,sha256=H6oIO3G7kz4N05D41t_roExxfmKgGFxK0RypcTeQUMY,4549
twine-1.2.2.dist-info/pydist.json,sha256=cQ_EFbalhIGsmq-Gxjl28a8Ow0xRol75vZIF5TzEc4U,1433
twine-1.2.2.dist-info/RECORD,,
twine-1.2.2.dist-info/top_level.txt,sha256=TQJA2ao69Pkj8Kk3OblQwf6k16du-g35xR5WeZR_Now,6
twine-1.2.2.dist-info/WHEEL,sha256=SXYYsi-y-rEGIva8sB8iKF6bAFD6YDhmqHX5hI3fc0o,110
/Users/dstufft/t/myVE/bin/twine,sha256=PKLESU6t2ElXvm4dQYN7VkOE6RyQ_wh1OjsNpt6Uguc,235
twine/__main__.pyc,,
twine/commands/upload.pyc,,
twine/commands/__init__.pyc,,
twine/__about__.pyc,,
twine/wheel.pyc,,
twine/application.pyc,,
twine/exceptions.pyc,,
twine/utils.pyc,,
twine/__init__.pyc,,

and the pip uninstall output

pip uninstall twine
Uninstalling twine:
  /Users/dstufft/t/myVE/bin/twine
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/DESCRIPTION.rst
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/METADATA
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/RECORD
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/WHEEL
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/entry_points.txt
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/pydist.json
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine-1.2.2.dist-info/top_level.txt
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__about__.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__about__.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__init__.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__init__.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__main__.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/__main__.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/application.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/application.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/commands/__init__.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/commands/__init__.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/commands/upload.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/commands/upload.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/exceptions.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/exceptions.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/utils.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/utils.pyc
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/wheel.py
  /Users/dstufft/t/myVE/lib/python2.7/site-packages/twine/wheel.pyc
Proceed (y/n)?

The compilation step happens before the RECORD step so it should pick it up as part of that.

@pfmoore
Copy link
Member

pfmoore commented Nov 22, 2013

Apologies, you are correct. I misunderstood the code - didn't read it in enough detail. I have confirmed that this works on Python 3.3.

Blame it on checking emails before my first caffeine hit, plus having had trouble with this stuff in the past when using wininst installers :-)

@qwcode
Copy link
Contributor

qwcode commented Nov 22, 2013

It only works in py3 because of PR #646 I added a year ago.
If you take that change out, pip leaves __pycache__ (and the parent pkg dirs) laying around.
the --record feature of distutils by itself was not pycache aware, when I looked at it.

@dstufft
Copy link
Member Author

dstufft commented Nov 22, 2013

It should work in Py3 regardless because the compilation happens prior to the RECORD being generated.

twine/__about__.py,sha256=A_vrkPkebq02SLPYcwdKxk7R6HZnuuAtwRCeHB-BPuk,1110
twine/__init__.py,sha256=AyNIeZSGwLc5bK0j2DE9aCHOGSlZ7ddv4piOqwrugKA,912
twine/__main__.py,sha256=kcJL2D7oY-8mSEHyZUsYnG712wgjw5JdSC11bAaXRJc,824
twine/application.py,sha256=wfHsPQF8z4skkm8MW-hb3Eli91rHE_sgwZPz7Daqu-w,3063
twine/exceptions.py,sha256=3nX1OB4x_UKyHmQn6Sby6tkxQITZN8KbEYn7KNhZ4sg,683
twine/utils.py,sha256=xY8z3g39yFaPqAndqgTpMaIWNjBHaJcgBwVD7Mp7a_A,1399
twine/wheel.py,sha256=-V0kAWKJknpW-gWoWJRPpSf614CRgFiIkEmBHXSa7qE,2677
twine/commands/__init__.py,sha256=jwqAHlhAJWVIKZCsG0mS-wuXSQY26LfE1WGR5xk0NP0,726
twine/commands/upload.py,sha256=UqEbOlTGgW2kWQdSdpPI-fhrd2O8iHtHuHNx4H_U5s0,7405
twine-1.2.2.data/scripts/twine,sha256=elKIESRsSGMC7ljphQQIFyUSnGTZwLe9JPBlV4zPphw,282
twine-1.2.2.dist-info/DESCRIPTION.rst,sha256=XuDgyW5TFHnm2KgxWvtUoGmyEaYwIoihwnR2QtYJqOA,3305
twine-1.2.2.dist-info/entry_points.txt,sha256=KUfmP4skuIXX7OYYEQUT1EqA-ZtazAri07vWOt9rloU,47
twine-1.2.2.dist-info/METADATA,sha256=H6oIO3G7kz4N05D41t_roExxfmKgGFxK0RypcTeQUMY,4549
twine-1.2.2.dist-info/pydist.json,sha256=cQ_EFbalhIGsmq-Gxjl28a8Ow0xRol75vZIF5TzEc4U,1433
twine-1.2.2.dist-info/RECORD,,
twine-1.2.2.dist-info/top_level.txt,sha256=TQJA2ao69Pkj8Kk3OblQwf6k16du-g35xR5WeZR_Now,6
twine-1.2.2.dist-info/WHEEL,sha256=SXYYsi-y-rEGIva8sB8iKF6bAFD6YDhmqHX5hI3fc0o,110
/Users/dstufft/t/myVE/bin/twine,sha256=CnaNAvIBQ_uswnwsNTX41ppEIvXsNy3U21ShnueQs84,235
twine/__pycache__/utils.cpython-33.pyc,,
twine/__pycache__/__about__.cpython-33.pyc,,
twine/__pycache__/exceptions.cpython-33.pyc,,
twine/commands/__pycache__/__init__.cpython-33.pyc,,
twine/__pycache__/__main__.cpython-33.pyc,,
twine/__pycache__/application.cpython-33.pyc,,
twine/__pycache__/wheel.cpython-33.pyc,,
twine/commands/__pycache__/upload.cpython-33.pyc,,
twine/__pycache__/__init__.cpython-33.pyc,,
Uninstalling twine:
  /Users/dstufft/t/myVE/bin/twine
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/DESCRIPTION.rst
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/METADATA
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/RECORD
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/WHEEL
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/entry_points.txt
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/pydist.json
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine-1.2.2.dist-info/top_level.txt
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__about__.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__init__.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__main__.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/__about__.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/__init__.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/__main__.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/application.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/exceptions.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/utils.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/__pycache__/wheel.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/application.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/commands/__init__.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/commands/__pycache__/__init__.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/commands/__pycache__/upload.cpython-33.pyc
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/commands/upload.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/exceptions.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/utils.py
  /Users/dstufft/t/myVE/lib/python3.3/site-packages/twine/wheel.py
Proceed (y/n)?

@qwcode
Copy link
Contributor

qwcode commented Nov 22, 2013

ok, you're just talking about wheels (and I see now how the cache files are being added indirectly)
my pt was just that --record alone was not capturing the pycache files for sdist uninstalls. pip has to do it after the fact, because distutils is unaware of pycache

@qwcode
Copy link
Contributor

qwcode commented Nov 22, 2013

most of pip.wheel.uninstallation_paths can be removed now, because it was handling wheel pyc removal before.

def uninstallation_paths(dist):
    """
    Yield all the uninstallation paths for dist based on RECORD-without-.pyc

    Yield paths to all the files in RECORD. For each .py file in RECORD, add
    the .pyc in the same directory.

    UninstallPathSet.add() takes care of the __pycache__ .pyc.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 5, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants