Migrate integration tests to shell based format, fixed travis #2868

phadej · 2015-10-13T21:11:05Z

Improves on #2864

phadej · 2015-10-13T22:23:41Z

ping @BardurArantsson @23Skidoo

23Skidoo · 2015-10-13T23:50:04Z

@phadej Does it also work on Windows (with msys2)?

/cc @grayjay

BardurArantsson · 2015-10-14T06:59:12Z

I'm 99% percent sure it won't run on Windows, I believe a previous iteration did this exact thing.

@phadej Thanks for the fixes. I'll rework the "/bin/bash" one (see comment there) and see if I can improve on the "extra-source-files" bit sometime during the next week or so (again, see comment there).

phadej · 2015-10-14T08:12:35Z

The only test failing on windows.

phadej · 2015-10-14T08:22:19Z

Hmm, it was easy after all:

phadej · 2015-10-14T08:36:37Z

For extra-source-files, IIRC recent enough Cabal supports globs? can we use them in Cabal itself?

phadej · 2015-10-14T13:24:54Z

And if someone wants to test on windows (now or later): https://dev.modern.ie/tools/vms is handy (minghc works nice too).

phadej · 2015-10-14T13:57:55Z

And last comments:

for OSX you might need to cabal updatewith just builded cabal once to get 00-index.tar into the right place (not the place Haskell Platform puts it) - alternatively tell where HP placed it to configure script. This is very minor, won't do anything to fix this.
stack test doesn't work still (package-tests do not work with stack #2711):

 cabal: Use of GHC's environment variable GHC_PACKAGE_PATH is incompatible with
Cabal.

this LGTM otherwise. The extra-source-files cleanup could be done later.

I need this PR in to get going with parsec parser (there is something weird happening in my branch, and this version of integration tests is much much more nicer to debug 👍).

BardurArantsson · 2015-10-14T15:17:44Z

Yeah, I agree that the glob vs. extra-sources thing isn't a show-stopper, it can be fixed in a diferent PR. (I'll create the issue after this PR lands. If I forget, please somebody remind me.)

I'm not sure what to make of #2711, but given that it's a previously existing issue and I don't think this necessarily changes anything much about a possible resolution, I think we should just punt on fixing #2711 with this PR -- and get this merged ASAP so everybody else who wants to use EasyTests(TM) becomes unblocked.

I'll go squash into a series of meaningful commits which build individually a little later tonight (or tomorrow). At least, I think it would be best to have that, I don't know if there's a project policy for this in Cabal/cabal-install land?

@phadej : I'm not sure if I can preserve authorship of all your patches if we want each individual commit to build -- do you mind if I just pull the "source" -> "." and "extra-sources" ones into the main commit here? (I think the Windows-compat bits can be separate, but if you don't mind I'd also be happy to just pull them in.)

23Skidoo · 2015-10-14T15:31:38Z

Globs of form foo/*.bar are supported, but recursive globs are not (yet).

BardurArantsson · 2015-10-14T15:36:28Z

Re: the glob thing. I think the only downside here that it's yet-another-surprise for when building on travis suddenly fails rather inexplicably, but perhaps we should just have a FAQ-type thing? (Or maybe a "checklist-before-creating-a-PR"?). Obviously, it would be ideal to just have an "extra-source-dir" or full recursive glob support, but until then this really only affects the PR phase of development. During normal running-of-tests, it's completely irrelevant.

23Skidoo · 2015-10-14T15:42:21Z

cabal-install/tests/IntegrationTests.hs

@@ -0,0 +1,276 @@
+-- | Groups black-box tests of cabal-install and configures them to test


I wonder if the tasty-golden package can be used to simplify code in this file.

I did check out tasty-golden, but it doesn't really seem to get us much since all the traversal stuff it pretty specialized anyway. Did you have something particular in mind?

Nope, it just looked like tasty-golden does something similar. I haven't read the code closely.

phadej · 2015-10-14T17:05:57Z

@BardurArantsson, I'm not particularly attached to those commits, thet are quite mechanical anyway. Actually I'll just squash everything myself.

This is required for the reworking of integration tests to use shell scripts.

Fixes haskell#2797 Squashed commits: - Use ., not source

- findExecutable sh - Use single quotes for cabal exec

BardurArantsson · 2015-10-14T20:14:32Z

@phadej Great, thanks. AFAICT this is good to go from my perspective.

23Skidoo · 2015-10-15T12:59:41Z

cabal-install/tests/IntegrationTests/exec/should_run/exit_with_failure_without_args.sh

@@ -0,0 +1,6 @@
+. ../common.sh
+
+# We should probably be using a .err file and should_fail,


One possible future solution for these kinds of problems is to add support for platform-specific .err files.

Or we could add support for matching against a set of regexes.

Also see how LLVM solves this: http://llvm.org/docs/CommandGuide/FileCheck.html

Yeah, I considered line-by-line regex comparison instead, but it's a bit noisy if you have to escape all the lines containing special characters (and where you don't want special treatment). FileCheck looks somewhat similar to that, but I think it's probably overkill for what we want.

Maybe we could add something simple here like line-based comparison, but with

RE:this is a regex this is not

i.e. just using the a RE: prefix to designate regexes. (It's ulikely enough to appear in actual output that it wouldn't need to be "escapable".)

The RE: prefix idea sounds good.

I'll create an issue to add it and assign it to myself. (No guarantees on when I'll get 'round to it, though.)

23Skidoo · 2015-10-15T13:17:29Z

OK, I think this is good to go.

@phadej @BardurArantsson Thanks for your contributions!

Migrate integration tests to shell based format, fixed travis

BardurArantsson · 2015-10-15T14:04:34Z

... and thanks to @ezyang and @grayjay for early review, testing and feedback!

23Skidoo reviewed Oct 14, 2015
View reviewed changes

BardurArantsson added 2 commits October 14, 2015 20:06

Move Compat.TempFile to Internal.* module and expose from Cabal

f5a94f3

This is required for the reworking of integration tests to use shell scripts.

Migrate integration tests to use shell scripts

87d55fb

Fixes haskell#2797 Squashed commits: - Use ., not source

phadej force-pushed the gh-2797-2 branch 2 times, most recently from 122c330 to 1919061 Compare October 14, 2015 17:15

Fix windows build

25b18bf

- findExecutable sh - Use single quotes for cabal exec

phadej force-pushed the gh-2797-2 branch from 1919061 to 25b18bf Compare October 14, 2015 17:16

23Skidoo mentioned this pull request Oct 14, 2015

Migrate integration tests to shell based format #2864

Closed

23Skidoo reviewed Oct 15, 2015
View reviewed changes

23Skidoo added a commit that referenced this pull request Oct 15, 2015

Merge pull request #2868 from phadej/gh-2797-2

41d8906

Migrate integration tests to shell based format, fixed travis

23Skidoo merged commit 41d8906 into haskell:master Oct 15, 2015

BardurArantsson mentioned this pull request Oct 15, 2015

Add flexibility for .err and .out comparison in cabal-install integration tests #2872

Closed

phadej deleted the gh-2797-2 branch October 15, 2015 15:29

BardurArantsson mentioned this pull request Oct 21, 2015

WIP/Search upwards for package directory #2811

Closed

BardurArantsson mentioned this pull request Jan 17, 2016

[RFC] Move dependency solver to library #2768

Closed

		@@ -0,0 +1,276 @@
		-- \| Groups black-box tests of cabal-install and configures them to test

		@@ -0,0 +1,6 @@
		. ../common.sh

		# We should probably be using a .err file and should_fail,

Migrate integration tests to shell based format, fixed travis #2868

Migrate integration tests to shell based format, fixed travis #2868

Uh oh!

Conversation

phadej commented Oct 13, 2015

Uh oh!

phadej commented Oct 13, 2015

Uh oh!

23Skidoo commented Oct 13, 2015

Uh oh!

BardurArantsson commented Oct 14, 2015

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

BardurArantsson commented Oct 14, 2015

Uh oh!

23Skidoo commented Oct 14, 2015

Uh oh!

BardurArantsson commented Oct 14, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phadej commented Oct 14, 2015

Uh oh!

BardurArantsson commented Oct 14, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

23Skidoo commented Oct 15, 2015

Uh oh!

BardurArantsson commented Oct 15, 2015

Uh oh!

Uh oh!