Investigate travis test failure

[pnasrat]

http://travis-ci.org/#!/pnasrat/pip/jobs/1323864

All other builds were fine, and this was just whitespace changes from:

http://travis-ci.org/#!/pnasrat/pip/builds/1322785

May be flakiness.

0======================================================================
81FAIL: Test installing an editable git package from a repository, upgrading the repository,
82----------------------------------------------------------------------
83Traceback (most recent call last):
84  File "/home/vagrant/virtualenv/python3.1/lib/python3.1/site-packages/nose/case.py", line 198, in runTest
85    self.test(*self.arg)
86  File "/home/vagrant/builds/pnasrat/pip/tests/test_upgrade.py", line 167, in test_editable_git_upgrade
87    assert 'some different version' in version2.stdout
88AssertionError
89
90----------------------------------------------------------------------
91Ran 151 tests in 358.095s
92
93FAILED (SKIP=2, failures=1)

[pnasrat]

Tested tests_upgrade on 3.1 locally - fine.

[pnasrat]

Full suite runs with 3.1 locally, closing as transient.

Ran 151 tests in 859.345s

OK (SKIP=2)

[pnasrat]

Got same failure on 2.5, definitely something odd so reopening, although it's not likely to be the code changes.

[pnasrat]

AssertionError: Output: 0.1

[pnasrat]

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[19e1eab533c2eb4dbb28eae083cfa3636e9c22b1] Fixed #487 - don't check for best-version-already-installed with explicit-URL requirements when upgrading.
pnasrat@grq01oob:((19e1eab...)|BISECTING) pip$ git bisect log
git bisect start
# bad: [f68b1d2a8fbdff16bb2ec51b876384ba431f6356] Add assertion message for improved debugging of flaky test
git bisect bad f68b1d2a8fbdff16bb2ec51b876384ba431f6356
# good: [87c719c0e8f7061d1fb1ba82c151a0eaeb1d4916] Merge pull request #488 from pnasrat/482-no-ordering
git bisect good 87c719c0e8f7061d1fb1ba82c151a0eaeb1d4916
# bad: [58ad2238717700d2ade36dac28570c12a3aa3ccd] Fix issue #510 - prevent sdist filename matches on partial project names. Thanks casevh for the report.
git bisect bad 58ad2238717700d2ade36dac28570c12a3aa3ccd
# bad: [c33e700551d7c1bea63e9dffb0159d7763388f23] Merge branch 'issue-58-origin-less-git' into develop
git bisect bad c33e700551d7c1bea63e9dffb0159d7763388f23
# good: [e5221a9987df759a271cd534ec2885af80bf50fe] expect stderr
git bisect good e5221a9987df759a271cd534ec2885af80bf50fe
# bad: [b883aa66777bfccec5190c73c5287b3d29750dd6] Add a __main__.py to enable 'python -m pip' on Python 2.7+. Thanks Alexey Luchko.
git bisect bad b883aa66777bfccec5190c73c5287b3d29750dd6
# bad: [db3eadd28e5f3121d0880bcadfa3cafd1e95498b] Fixed a test that was failing intermittently due to pyc file race conditions.
git bisect bad db3eadd28e5f3121d0880bcadfa3cafd1e95498b

[pnasrat]

@carljm do you have more context on the issue and the fix, this may have introduced the current flakiness, or the race may still exist.

SHA: db3eadd

[pnasrat]

Note to self may be seeing different failure on OS X, however in travis we see:

http://travis-ci.org/#!/pnasrat/pip/builds/1332316 pass on my develop branch
http://travis-ci.org/#!/pypa/pip/builds/1332714 fail on the merge

[pnasrat]

This looks like git rev parsing

======================================================================
FAIL: It can upgrade a VCS requirement that has distributions on the index.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pnasrat/Development/pip/ve/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/pnasrat/Development/pip/tests/test_upgrade.py", line 210, in test_upgrade_vcs_req_with_dist_found
    run_pip("install", req)
  File "/Users/pnasrat/Development/pip/tests/test_pip.py", line 470, in run_pip
    result = env.run('pip', *args, **kw)
  File "/Users/pnasrat/Development/pip/tests/test_pip.py", line 358, in run
    return TestPipResult(super(TestPipEnvironment, self).run(cwd=cwd, *args, **kw), verbose=self.verbose)
  File "/Users/pnasrat/Development/pip/ve/lib/python2.7/site-packages/scripttest/__init__.py", line 255, in run
    result.assert_no_error(quiet)
  File "/Users/pnasrat/Development/pip/ve/lib/python2.7/site-packages/scripttest/__init__.py", line 402, in assert_no_error
    "Script returned code: %s" % self.returncode)
AssertionError: Script returned code: 1
-------------------- >> begin captured stdout << ---------------------
Script result: pip install git+file:///Users/pnasrat/Development/pip/tests/tests_cache/virtualenv@c21fef2c2d53cf19f49bcc37f9c058a33fb50499#egg=virtualenv
  return code: 1
-- stderr: --------------------
fatal: '/Users/pnasrat/Development/pip/tests/tests_cache/virtualenv' does not appear to be a git repository
fatal: The remote end hung up unexpectedly

ls tests/tests_cache/virtualenv@c21fef2c2d53cf19f49bcc37f9c058a33fb50499/
AUTHORS.txt     MANIFEST.in     scripts         virtualenv.py
HACKING         bin         setup.py        virtualenv_embedded
LICENSE.txt     docs            tests           virtualenv_support

[pnasrat]

>>> from tests.local_repos import *
>>> local_checkout("git+git://github.com/pypa/virtualenv@c21fef2c2d53cf19f49bcc37f9c058a33fb50499")

This creates a dir with the @ revision in the cache. Then when we try and install from it we assume the revision is just that from the dir name. I can't see how this is working on other platforms - the following patch fixes the OS X failure for me.

diff --git a/tests/local_repos.py b/tests/local_repos.py
index 4a1a171..008a1a5 100644
--- a/tests/local_repos.py
+++ b/tests/local_repos.py
@@ -57,7 +57,8 @@ def _get_vcs_and_checkout_url(remote_repository):
         branch = os.path.basename(remote_repository)
         repository_name = os.path.basename(remote_repository[:-len(branch)-1]) # remove the slash
     else:
-        repository_name = os.path.basename(remote_repository)
+        repository_name, _ = vcs_class(remote_repository).get_url_rev()
+        repository_name = repository_name.rsplit('/', 1)[1]

     destination_path = os.path.join(tests_cache, repository_name)
     if not os.path.exists(destination_path):

[pnasrat]

Documenting more as this gets odder the more I investigate. That change caused a failure in test_vcs_backends.py as I started version testing (on 2.4). I then backed out so I'm at SHA: f68b1d2

Setup is 3 virtualenvs on Mac OS X 10.7.3

for p in 24 27 31; do ./${p}/bin/nosetests test_vcs_backends test_upgrade; done
..........................
----------------------------------------------------------------------
Ran 26 tests in 78.193s

OK
..........................
----------------------------------------------------------------------
Ran 26 tests in 77.863s

OK
..........................
----------------------------------------------------------------------
Ran 26 tests in 103.824s

I'm a little confused, as test_upgrade was reliably failing for myself and @dstufft on OS X. I definitely feel local_repos is trying to be too clever and our test code shouldn't call into our self to do setup. No closer on the travis failure though, that may be the race mentioned in @carljm's bug. The unpredictability in failures hints at that or a test encapsulation issue.

[pnasrat]

I've temporarily annotated the test as @nottest and intend to fix the underlying testing infrastructure then re-enable.

[msabramo]

Still an issue or can this be closed?

[xavfernandez]

Test is still there: https://github.com/pypa/pip/blob/develop/tests/functional/test_install_upgrade.py#L211-L233 but skipped... Would be nice to unskip it.