Sandboxed test execution fails when any plugin is installed

[hoefling]

I'm building a package using sandboxing (setuptools.sandbox module). I found out that I can't execute tests with pytest once I have installed any plugin. Here are the steps to reproduce from scratch:

1. Create a fresh virtual environment and preinstall pytest.

$ virtualenv /tmp/tstenv
$ source /tmp/tstenv/bin/activate
$ pip install pytest pytest-runner
$ pip list
Package       Version
------------- -------
appdirs       1.4.3
packaging     16.8
pip           9.0.1
py            1.4.32
pyparsing     2.2.0
pytest        3.0.6
pytest-runner 2.11.1
setuptools    34.3.2
six           1.10.0
wheel         0.29.0

2. Add project stub to play with.

$ mkdir -p /tmp/pytest-issue
$ cd $_
$ cat <<EOM >setup.py
from setuptools import setup

setup(
    name='project-stub',
    version='0.1',
    packages=[]
)
EOM
$ cat <<EOM >sandboxed_setup.py
import setuptools.sandbox

setuptools.sandbox.run_setup('setup.py', ['pytest'])
EOM
$ cat <<EOM >test_foo.py
def test_bar():
    assert True
EOM

3. Now run the tests in sandboxed mode. This works as expected, one test is found and executed successfully.

$ python sandboxed_setup.py
running pytest
running egg_info
writing top-level names to project_stub.egg-info/top_level.txt
writing dependency_links to project_stub.egg-info/dependency_links.txt
writing project_stub.egg-info/PKG-INFO
reading manifest file 'project_stub.egg-info/SOURCES.txt'
writing manifest file 'project_stub.egg-info/SOURCES.txt'
running build_ext
============================== test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /tmp/pytest-issue, inifile: 
collected 1 items 

test_foo.py .

============================ 1 passed in 0.02 seconds ===========================

4. Install any pytest plugin and rerun the tests:

$ pip install pytest-variables
$ python sandboxed_setup.py
running pytest
running egg_info
writing dependency_links to project_stub.egg-info/dependency_links.txt
writing project_stub.egg-info/PKG-INFO
writing top-level names to project_stub.egg-info/top_level.txt
reading manifest file 'project_stub.egg-info/SOURCES.txt'
writing manifest file 'project_stub.egg-info/SOURCES.txt'
running build_ext
error: SandboxViolation: mkdir('/tmp/tstenv/lib/python3.5/site-packages/pytest_variables/__pycache__',) {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

Sys info

```bash $ emerge --info Portage 2.3.4 (python 3.5.3-final-0, default/linux/amd64/13.0/desktop/plasma, gcc-5.4.0, glibc-2.23-r3, 4.10.1-gentoo x86_64) ================================================================= System uname: Linux-4.10.1-gentoo-x86_64-Intel-R-_Core-TM-_i3-4100M_CPU_@_2.50GHz-with-gentoo-2.3 KiB Mem: 8055596 total, 1400008 free KiB Swap: 8388604 total, 8388604 free Timestamp of repository gentoo: Mon, 13 Mar 2017 20:00:01 +0000 sh bash 4.3_p48-r1 ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1 distcc 3.2rc1 x86_64-pc-linux-gnu [disabled] app-shells/bash: 4.3_p48-r1::gentoo dev-java/java-config: 2.2.0-r3::gentoo dev-lang/perl: 5.22.3_rc4::gentoo dev-lang/python: 2.7.12::gentoo, 3.5.3::gentoo, 3.6.0::gentoo dev-util/cmake: 3.7.2::gentoo dev-util/pkgconfig: 0.28-r2::gentoo sys-apps/baselayout: 2.3::gentoo sys-apps/openrc: 0.23.2::gentoo sys-apps/sandbox: 2.10-r3::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo sys-devel/automake: 1.14.1::gentoo, 1.15::gentoo sys-devel/binutils: 2.25.1-r1::gentoo sys-devel/gcc: 5.4.0-r3::gentoo sys-devel/gcc-config: 1.7.3::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1::gentoo sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.23-r3::gentoo Repositories:

gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: -1000

crossdev
location: /usr/local/portage/overlays/crossdev
masters: gentoo
priority: 10

brother-overlay
location: /usr/local/portage/overlays/brother-overlay
sync-type: git
sync-uri: https://github.com/stefan-langenmaier/brother-overlay.git
masters: gentoo
priority: 50

hering-overlay
location: /usr/local/portage/overlays/hering-overlay
sync-type: git
sync-uri: https://github.com/internethering/hering-overlay.git
masters: gentoo
priority: 50

hoefling-private
location: /usr/local/portage/overlays/hoefling-private
sync-type: git
sync-uri: https://github.com/hoefling/hoefling-private-overlay.git
masters: gentoo
priority: 50

pypi
location: /var/lib/layman/pypi
masters: gentoo
priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@eula"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.3/conf /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release
/etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d
/etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=native"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news
parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch
userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ rsync://mirror.eu.oneandone.net/gentoo/
http://mirror.netcologne.de/gentoo/ rsync://mirror.netcologne.de/gentoo/
http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp.halifax.rwth-aachen.de/gentoo/
rsync://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo
rsync://ftp-stud.hs-esslingen.de/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete
--stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus
declarative dri dts dvd dvdr emboss encode exif fam ffmpeg firefox flac fortran gdbm gif glamor gpm iconv icu ipv6 jpeg kipi
kwallet lcms ldap libnotify mad mmx mng modules mp3 mp4 mpeg multilib ncurses networkmanager nls nptl ogg opengl openmp pam
pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt5 readline sdl seccomp semantic-desktop session spell sqlite
sse sse2 ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis wayland widgets wxwidgets x264
xattr xcb xcomposite xinerama xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem
bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio
via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias
authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user
autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include
info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack
vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df
interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm
earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2
rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux"
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console
presenter-minimizer" NETBEANS_MODULES="apisupport java javafx profiler websvccommon webcommon enterprise"
OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python3_5 python3_6
pypy3" QEMU_SOFTMMU_TARGETS="i386 x86_64" RUBY_TARGETS="ruby21" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2
psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark
dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS,
PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

  </p>
</details>

[nicoddemus]

It seems the problem is that assertion rewriting is writing the cached .pyc files to disk, and sandboxing doesn't like that. You can set PYTHONDONTWRITEBYTECODE to disable this behavior.

[RonnyPfannschmidt]

its also a wont-fix, the setuptools sandbox is broken by design and will eventually disappear

[hoefling]

@RonnyPfannschmidt: interesting, didn't know that. Can you point me on any official info about its removal? As far as I can see, there's no deprecation warnings in the module itself, its code is being maintained (last modified on Feb 24th).