Strange Travis failures

[angerman]

Travis seems to fail with:

cabal/dist-newstyle/src/pretty-show-1.6.16/dist: renamePath:rename: unsatisfied constraints (Directory not empty)Configuring library for pretty-show-1.6.16..

lately a lot.

So I've basically run git bisect with git clean -xfd && cabal new-build cabal-install on the tree.

That ended up with:

84ab693b0162c372c5127e0a27a52bf902beea55 is the first bad commit
commit 84ab693b0162c372c5127e0a27a52bf902beea55
Author: Mikhail Glushenkov <mikhail.glushenkov@gmail.com>
Date:   Thu Jan 4 10:35:30 2018 +0000

    Unbreak Haddock doc generation.
    
    + a whitespace fix.

:040000 040000 375f64d216a741dba303b827e5da2513bd47f490 93da2a12bae5c288aecccff300d636296751937f M	cabal-install
bisect run success

log is

git bisect start
# good: [abc18aa7a2677a549f2ff57729a0dee026a7f69c] Capitalisation.
git bisect good abc18aa7a2677a549f2ff57729a0dee026a7f69c
# bad: [18317ce96afbdc08e317b41d10802132266b2ec3] Add buildDir includes as needed.
git bisect bad 18317ce96afbdc08e317b41d10802132266b2ec3
# good: [ab585e864929a8d19ed844d82ff2e9d002ae6313] Merge remote-tracking branch 'upstream/dcoutts/project-config-explicit-includes'
git bisect good ab585e864929a8d19ed844d82ff2e9d002ae6313
# bad: [67276f870a55e6a913b98ef7b2ca22a8def21a6e] Merge pull request #4861 from fgaz/new-run/allow-running-tests-benchs
git bisect bad 67276f870a55e6a913b98ef7b2ca22a8def21a6e
# bad: [ce775dbbfe7d04fb9376e2562002a3bbcb0337ed] Add duplicate field negative tests.
git bisect bad ce775dbbfe7d04fb9376e2562002a3bbcb0337ed
# bad: [ec9ce1c82dcff66608a7b1875e11c220828ec2ee] cabal-testsuite: test cabal new-test with coverage
git bisect bad ec9ce1c82dcff66608a7b1875e11c220828ec2ee
# bad: [552ae22bdc7a265bd57b131f3749e239476bd7ca] Rename rewriteFileV to rewriteFileEx.
git bisect bad 552ae22bdc7a265bd57b131f3749e239476bd7ca
# good: [6e42aa2502ff8e318607d8614693c712f5aac709] Merge pull request #4991 from ezyang/pr/T4986
git bisect good 6e42aa2502ff8e318607d8614693c712f5aac709
# good: [a3e1833890afe970da5eeae3b9d5cac8473734f1] Restore BC compatibility for rewriteFile, fixes #4981
git bisect good a3e1833890afe970da5eeae3b9d5cac8473734f1
# bad: [587bc30577759a93615f391ec42afeaecef2f964] Merge pull request #4996 from ezyang/pr/rewrite-file-bc
git bisect bad 587bc30577759a93615f391ec42afeaecef2f964
# bad: [84ab693b0162c372c5127e0a27a52bf902beea55] Unbreak Haddock doc generation.
git bisect bad 84ab693b0162c372c5127e0a27a52bf902beea55
# first bad commit: [84ab693b0162c372c5127e0a27a52bf902beea55] Unbreak Haddock doc generation.

hope that's of any use...

[phadej]

ogre@bigtest:~/cabal$ /opt/cabal/2.0/bin/cabal new-build cabal-install -j10
In order, the following will be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

2.0 does something weird?

---

The quick'n'dirty solution is to not parellise builds, but that would hurt in time.

[phadej]

oh, we had bad commits there otherwise too:

Distribution/Simple/Register.hs:423:62: error:
    • Couldn't match type ‘Distribution.ModuleName.ModuleName’
                     with ‘IPI.ExposedModule’

EDIT: not really proposing, but maybe we should squash more often.

[angerman]

A second run, which basically ran:

#!/bin/bash
(git clean -xfd && cabal new-build cabal-install -j4) && (git clean -xfd && cabal new-build cabal-install -j4) && (git clean -xfd && cabal new-build cabal-install -j4)

yielded

912f2236875a2c364a703dd3600aacdd1674abb6 is the first bad commit
commit 912f2236875a2c364a703dd3600aacdd1674abb6
Author: Mikhail Glushenkov <mikhail.glushenkov@gmail.com>
Date:   Tue Dec 19 08:27:48 2017 +0000

    Remove the constraint on pretty-show.
    
    The issue is fixed now.

:100644 100644 8947e15203e54a549198ca7618d98f2201436be0 65aa59ba590b11c4d90ddf3703c590f1e9edab6b M	cabal.project
bisect run success

log is

git bisect start
# good: [abc18aa7a2677a549f2ff57729a0dee026a7f69c] Capitalisation.
git bisect good abc18aa7a2677a549f2ff57729a0dee026a7f69c
# bad: [18317ce96afbdc08e317b41d10802132266b2ec3] Add buildDir includes as needed.
git bisect bad 18317ce96afbdc08e317b41d10802132266b2ec3
# bad: [ab585e864929a8d19ed844d82ff2e9d002ae6313] Merge remote-tracking branch 'upstream/dcoutts/project-config-explicit-includes'
git bisect bad ab585e864929a8d19ed844d82ff2e9d002ae6313
# good: [f0d688a9814d3316c5a1267733c4c8a211500de8] Formatting.
git bisect good f0d688a9814d3316c5a1267733c4c8a211500de8
# bad: [a3163408c570e3edd876c5d3280f08a5e8e64235] Add INLINE pragmas for funAppMon methods of ParsecParser
git bisect bad a3163408c570e3edd876c5d3280f08a5e8e64235
# bad: [7a6559af48d4668087387e03d11ef038ac4f9e6e] Unbreak the build with directory < 1.2.3.0.
git bisect bad 7a6559af48d4668087387e03d11ef038ac4f9e6e
# good: [b06aac9553887cb88645a049379afa4de52ae4f8] Merge pull request #4952 from gwils/issue-4443
git bisect good b06aac9553887cb88645a049379afa4de52ae4f8
# bad: [86af280bcd7e3c16f929d92a17a1239be6d9d96a] Merge pull request #4929 from grayjay/duplicate-flag-message
git bisect bad 86af280bcd7e3c16f929d92a17a1239be6d9d96a
# good: [5270950fdf41f387450b04f48c3d01a600bb2bd8] Call simplifyVersionRange in solver convenience functions (.||.) and (.&&.).
git bisect good 5270950fdf41f387450b04f48c3d01a600bb2bd8
# good: [4f7ac10dfd93afed1c6810a19df146d8adbe1fb3] Solver: Combine dependencies on the same package in the same 'build-depends'.
git bisect good 4f7ac10dfd93afed1c6810a19df146d8adbe1fb3
# bad: [912f2236875a2c364a703dd3600aacdd1674abb6] Remove the constraint on pretty-show.
git bisect bad 912f2236875a2c364a703dd3600aacdd1674abb6
# first bad commit: [912f2236875a2c364a703dd3600aacdd1674abb6] Remove the constraint on pretty-show.

[angerman]

Something's fishy with that commit:

cabal @ 912f22368 $ git clean -xfd && cabal new-build cabal-install -j4 --dry-run                                                                                                                                                             Removing dist-newstyle/
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

cabal @ b06aac955 $ git clean -xfd && cabal new-build cabal-install -j4 --dry-run                                                                                                                                                             Removing dist-newstyle/
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.13 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

[angerman]

912f223 is

From 912f2236875a2c364a703dd3600aacdd1674abb6 Mon Sep 17 00:00:00 2001
From: Mikhail Glushenkov <mikhail.glushenkov@gmail.com>
Date: Tue, 19 Dec 2017 08:27:48 +0000
Subject: [PATCH] Remove the constraint on pretty-show.

The issue is fixed now.
---
 cabal.project | 1 -
 1 file changed, 1 deletion(-)

diff --git a/cabal.project b/cabal.project
index 8947e1520..65aa59ba5 100644
--- a/cabal.project
+++ b/cabal.project
@@ -1,6 +1,5 @@
 packages: Cabal/ cabal-testsuite/ cabal-install/ solver-benchmarks/
 constraints: unix >= 2.7.1.0,
-             pretty-show == 1.6.13,
              cabal-install +lib +monolithic
 
 -- Uncomment to allow picking up extra local unpacked deps:
-- 
2.16.0

I don't yet see why that change would start producing multiple entries...

[angerman]

So, with the constraint, we pick pretty-show once, without the constraint we pick it twice? Anyone got any idea where that might come from?

[angerman]

Found it!
In pretty-show > 1.6.13, the cabal file gained the following line

  build-tool-depends: happy:happy

I don't know why, but that line causes cabal

$ cabal --version
cabal-install version 2.0.0.1
compiled using version 2.0.1.0 of the Cabal library 

to start adding pretty-show twice, and cabal-install twice to the build plan.
As a work around I would suggest, we remove that line from pretty-show for the time being?

@phadej, @23Skidoo any better idea?

[gbaz]

this rabbithole seems relevant?

#4009
haskell/happy#113

(and tangentially: #130)

Note the release of a new happy 3 days ago?

[angerman]

The really confusing part is, why the solver starts to think it needs to build cabal-install and pretty-show twice once show-prettys cabal file lists the build-tool-depends. As @phadej noted on IRC, this doesn't happen with cabal HEAD apparently (I haven't verified this).

The bootstrapping cabal on travis however does seem to exhibit this bug.

[phadej]

FWIW, My run of git bisect found the same commit

912f2236875a2c364a703dd3600aacdd1674abb6 is the first bad commit
commit 912f2236875a2c364a703dd3600aacdd1674abb6
Author: Mikhail Glushenkov <mikhail.glushenkov@gmail.com>
Date:   Tue Dec 19 08:27:48 2017 +0000

    Remove the constraint on pretty-show.
    
    The issue is fixed now.

:100644 100644 8947e15203e54a549198ca7618d98f2201436be0 65aa59ba590b11c4d90ddf3703c590f1e9edab6b M      cabal.project
bisect run success

[phadej]

@23Skidoo

% git show e80e1a11
commit e80e1a11fc6863de91a689ef87b24c1401659e41
Author: Mikhail Glushenkov <mikhail.glushenkov@gmail.com>
Date:   Tue Dec 12 10:57:00 2017 +0000

    Temporarily pin pretty-show to 1.6.13.
    
    Should unbork the test suite.

diff --git a/cabal.project b/cabal.project
index 65aa59b..8947e15 100644
--- a/cabal.project
+++ b/cabal.project
@@ -1,5 +1,6 @@
 packages: Cabal/ cabal-testsuite/ cabal-install/ solver-benchmarks/
 constraints: unix >= 2.7.1.0,
+             pretty-show == 1.6.13,
              cabal-install +lib +monolithic
 
 -- Uncomment to allow picking up extra local unpacked deps: