CP-1129 Add flag to automatically start `pub serve` when running tests/coverage #92

greglittlefield-wf · 2015-11-12T16:17:23Z

Ultimate Problem

Tests and coverage currently cannot be run with files served from a Pub server.

See #79.

Solution

Add PubServeTask
- Allows creation and management of pub serve instances.
Add pub-serve CLI option and pubServe config option to test/coverage tasks.
- Automatically spins up a Pub server that is used to run the tests.
Add integration tests for dart_dev coverage/dart_dev test in projects that require a Pub server.

Other changes

Tweak Reporter.logGroup code to be more flexible, and indent for multiline Stream events.

Testing

Verify that integration tests for test/coverage tasks pass.
Verify that a real consumer package can run dart_dev test/dart_dev coverage using a pub server.
- E.g., web_skin_dart with transformer code gen: https://ci.webfilings.com/build/417851
Verify that pub serve output is well-formatted.

FYA: @evanweible-wf @trentgrover-wf
FYI: @jacehensley-wf @aaronlademann-wf

greglittlefield-wf · 2015-11-12T16:20:16Z

lib/src/tasks/test/api.dart

+  if (pubServe) {
+    // Start `pub serve` on the `test` directory
+    pubServeTask = serve.pubServe(additionalArgs: ['test']);
+    var serveInfo = await pubServeTask.serveInfos.first;


I'd like to log some output here... At the very least, an indication that the pub server is starting, but ideally the pub serve output up until it starts.

Should pubServeTask be a field on TestTask, so that it can be accessed and logged accordingly in test/cli.dart?

Yeah at this point, that's probably what I would do. You could break it up into two different parts - the logs up until the server is ready, and the remainder. That way you could print the first part immediately and the rest of it when the server is stopped.

evanweible-wf · 2015-11-16T19:15:50Z

@greglittlefield-wf I think your approach here is fine. As you've seen, the current dart_dev architecture makes this feel a bit hacky, and it's definitely something I want to improve in 2.0.0. But for now, this seems to me like it should work

greglittlefield-wf · 2015-11-16T19:26:17Z

Cool, I'll proceed with this approach and work toward finalizing it. Thanks for the feedback!

codecov-io · 2015-11-18T00:42:15Z

Current coverage is `45.12%`

Merging #92 into master will decrease coverage by -2.93% as of 16761fa

Powered by Codecov. Updated on successful CI builds.

greglittlefield-wf · 2015-11-19T20:00:04Z

This is ready for final review. Any clue as to why tests are timing out on Travis? They seem to run fine locally.

greglittlefield-wf · 2015-11-19T21:26:17Z

lib/src/tasks/coverage/api.dart

+          }).timeout(const Duration(seconds: 10), onTimeout: () {
+            throw new TimeoutException(
+                'failed to detect `pub serve` directory containing ${htmlFile.path}');
+          });


Hmm, now that I think about it, if we're passing in additionalArgs: ['test'] to startPubServe, this stream will almost assuredly contain only one event.

This should probably be changed to:

PubServeInfo serveInfo = await pubServeTask.serveInfos.first; if (!path.isWithin(serveInfo.directory, htmlFile.path)) { throw '`pub serve` directory `${serveInfo.directory}` does not contain test file `${htmlFile.path}`.'; }

evanweible-wf · 2015-11-19T22:25:10Z

Travis CI failure is an issue with Dart 1.13, I'm looking into it

greglittlefield-wf · 2015-11-19T23:39:07Z

lib/src/tasks/coverage/api.dart

-        if (line.contains(_testsPassedPattern)) {
-          break;
+
+        // Run the test in content-shell.


The diff for this section is easier to look at if you ignore whitespace changes.

evanweible-wf · 2015-11-20T15:23:24Z

This should fix the CI failure once merged: #98

trentgrover-wf · 2015-11-20T15:51:44Z

+1

rmconsole3-wf · 2015-11-20T15:51:55Z

@greglittlefield-wf This pull request has merge conflicts, please resolve.

greglittlefield-wf · 2015-11-20T16:07:08Z

Merge conflicts have been resolved, and it looks like the build is passing now. Thanks, @evanweible-wf!

evanweible-wf · 2015-11-20T17:10:32Z

@greglittlefield-wf This looks great! Could you add this config option to the readme?

greglittlefield-wf · 2015-11-20T17:46:19Z

@evanweible-wf Yup! Added in 851ae16.

evanweible-wf · 2015-11-20T18:48:54Z

+1

@trentgrover-wf @maxwellpeterson-wf @dustinlessard-wf any of you care to take a look as well?

After that, last step would be a rebase and a squash into a single commit @greglittlefield-wf

greglittlefield-wf · 2015-11-20T19:45:04Z

@evanweible-wf Rebased and squashed.

maxwellpeterson-wf · 2015-11-20T19:50:19Z

+1

evanweible-wf · 2015-11-20T19:51:13Z

+1

dustinlessard-wf · 2015-11-20T19:52:00Z

+1 cool

trentgrover-wf · 2015-11-20T20:38:49Z

+1
@jayudey-wf ready for merge

jayudey-wf · 2015-11-23T13:40:05Z

do you think we need to add an update to the readme about ensuring that you have this in your pubspec.yaml if you want to serve your tests?

- test/pub_serve:
    $include: test/**_test{.*,}.dart

greglittlefield-wf · 2015-11-23T15:30:49Z

@jayudey-wf That's a good idea, I'll add that.

greglittlefield-wf · 2015-11-23T15:43:26Z

K, added info to README about setting up test to work with transformers in greglittlefield-wf@1855b61. Needs a +1.

@evanweible-wf Should I squash that commit in as well?

evanweible-wf · 2015-11-23T18:16:37Z

@greglittlefield-wf sure, might as well. Readme changes look good

greglittlefield-wf · 2015-11-23T18:22:48Z

Squashed and pushed.

maxwellpeterson-wf · 2015-11-23T18:38:16Z

+1

evanweible-wf · 2015-11-23T18:57:03Z

+1 @jayudey-wf ready for merge.

jayudey-wf · 2015-11-23T19:22:58Z

QA Resource Approval: +10

Testing instruction
Dev +1's
Dev/QA +10 with detail of what was tested
- verified that tests were passing, installed into two separate projects and made sure that test and coverage were still working as expected w/ and w/o the pub server going, pub serve logging looked ok
Unit test created/updated
All unit tests pass

Merging into master.

CP-1129 Add flag to automatically start `pub serve` when running tests/coverage

greglittlefield-wf reviewed Nov 12, 2015
View reviewed changes

greglittlefield-wf changed the title ~~Add flag to automatically start pub serve when running tests~~ Add flag to automatically start pub serve when running tests/coverage Nov 19, 2015

greglittlefield-wf mentioned this pull request Nov 19, 2015

Allow coverage to be run on localhost #95

Closed

greglittlefield-wf reviewed Nov 19, 2015
View reviewed changes

rmconsole3-wf added the Merge Conflicts label Nov 20, 2015

rmconsole2-wf removed the Merge Conflicts label Nov 20, 2015

greglittlefield-wf force-pushed the pub_serve branch from 851ae16 to cd82d2b Compare November 20, 2015 19:44

jayudey-wf changed the title ~~Add flag to automatically start pub serve when running tests/coverage~~ CP-1129 Add flag to automatically start pub serve when running tests/coverage Nov 23, 2015

greglittlefield-wf force-pushed the pub_serve branch from b19d342 to 223123f Compare November 23, 2015 15:38

greglittlefield-wf force-pushed the pub_serve branch 2 times, most recently from 0cbcb2e to 1855b61 Compare November 23, 2015 15:41

Add option to run browser tests using pub serve in coverage/test tasks

8b44e52

greglittlefield-wf force-pushed the pub_serve branch from 1855b61 to 8b44e52 Compare November 23, 2015 18:22

jayudey-wf added a commit that referenced this pull request Nov 23, 2015

Merge pull request #92 from greglittlefield-wf/pub_serve

5d2b6cc

CP-1129 Add flag to automatically start `pub serve` when running tests/coverage

jayudey-wf merged commit 5d2b6cc into Workiva:master Nov 23, 2015

jayudey-wf mentioned this pull request Nov 25, 2015

CP-1127 Release dart_dev 1.0.5 #106

Merged

greglittlefield-wf mentioned this pull request Dec 11, 2015

Ability to run tests that require '--pub-serve' #79

Closed

evanweible-wf mentioned this pull request Dec 11, 2015

Coverage: serve browser tests with pub so XHR requests have a valid Origin #89

Closed

CP-1129 Add flag to automatically start pub serve when running tests/coverage #92

CP-1129 Add flag to automatically start pub serve when running tests/coverage #92

Uh oh!

Conversation

greglittlefield-wf commented Nov 12, 2015

Ultimate Problem

Solution

Other changes

Testing

Uh oh!

greglittlefield-wf Nov 12, 2015

Choose a reason for hiding this comment

Uh oh!

evanweible-wf Nov 16, 2015

Choose a reason for hiding this comment

Uh oh!

evanweible-wf commented Nov 16, 2015

Uh oh!

greglittlefield-wf commented Nov 16, 2015

Uh oh!

codecov-io commented Nov 18, 2015

Current coverage is 45.12%

Uh oh!

greglittlefield-wf commented Nov 19, 2015

Uh oh!

greglittlefield-wf Nov 19, 2015

Choose a reason for hiding this comment

Uh oh!

evanweible-wf commented Nov 19, 2015

Uh oh!

greglittlefield-wf Nov 19, 2015

Choose a reason for hiding this comment

Uh oh!

evanweible-wf commented Nov 20, 2015

Uh oh!

trentgrover-wf commented Nov 20, 2015

Uh oh!

rmconsole3-wf commented Nov 20, 2015

Uh oh!

greglittlefield-wf commented Nov 20, 2015

Uh oh!

evanweible-wf commented Nov 20, 2015

Uh oh!

greglittlefield-wf commented Nov 20, 2015

Uh oh!

evanweible-wf commented Nov 20, 2015

Uh oh!

greglittlefield-wf commented Nov 20, 2015

Uh oh!

maxwellpeterson-wf commented Nov 20, 2015

Uh oh!

evanweible-wf commented Nov 20, 2015

Uh oh!

dustinlessard-wf commented Nov 20, 2015

Uh oh!

trentgrover-wf commented Nov 20, 2015

Uh oh!

jayudey-wf commented Nov 23, 2015

Uh oh!

greglittlefield-wf commented Nov 23, 2015

Uh oh!

greglittlefield-wf commented Nov 23, 2015

Uh oh!

evanweible-wf commented Nov 23, 2015

Uh oh!

greglittlefield-wf commented Nov 23, 2015

Uh oh!

maxwellpeterson-wf commented Nov 23, 2015

Uh oh!

evanweible-wf commented Nov 23, 2015

Uh oh!

jayudey-wf commented Nov 23, 2015

Uh oh!

Uh oh!

CP-1129 Add flag to automatically start `pub serve` when running tests/coverage #92

CP-1129 Add flag to automatically start `pub serve` when running tests/coverage #92

Current coverage is `45.12%`