Comprehensive workflow end-to-end testing

[screamerbg]

The PR adds tests to Circle CI for the following workflows and backward compatibility

mbed OS 5.0 - new program (latest mbed OS)

• new program with mbed OS 5.0
  • create mbed program inside existing repository
• list program libraries
• install python modules from requirements.txt
• compile a program (use one of the tests)
• compile a test using mbed test

mbed OS 5.0 - mbed Client example (SiP workshop)

• import mbed OS 5.0 based example (git)
  • import mbed program inside existing repository
• update example to previous version which replaces or modifies the whole tree layout, garbage collects obsolete repos or replaces them from different URLs
• update example to the latest revision which must restore the original layout
• compile the example

mbed OS pre-5.0 - mbed Client example (Morpheus workshop)

• import mbed OS 5.0 based example (hg)
• update example to previous version which replaces or modifies the whole tree layout, garbage collects obsolete repos or replaces them from different URLs
• update example to the latest revision which must restore the original layout
• compile the example

mbed OS 2.0 - USB example

• import mbed OS 2.0 based example (hg) from 2013
• update mbed SDK to release 85
• update example to the latest revision which reverts mbed SDK to release 75-ish
• compile the example

Commands used:

• mbed new which also clones mbed-os
• mbed import (indirectly mbed deploy) for git, hg, bld (mbed 2.0) based example programs
• mbed update <rev> to a specific revision, which tests garbage collecting of obsolete repos and also repos with different URLs
• mbed update to latest repo in a branch (tests GC and repos with different URLs as well
• mbed compile for mbed OS 5.0, mbed Morpheus and mbed 2.0 SDK programs
• mbed toolchain to set default toolchain
• mbed target to set default target
• mbed config to set default protocol
• mbed ls

[sg-]

Awesome +💯

[bridadan]

Wooo tests!

+1

[jupe]

This is very good starting point, but there is one problem, I cannot run this in my own machine because it based on circle config file.. Or can I?

[screamerbg]

@jupe Add circle_tests.py script to execute Circle CI tests locally 08d8d8a

[jupe]

I propose to use python unittest instead of this little hacky solution..

[ghost]

What is the motivation behind not using the standard unitest framework?

[screamerbg]

@jupe Hope you're happy to contribute it :)

[screamerbg]

@mjs-arm The original requirement was to provide comprehensive workflow test coverage for mbed CLI, where every pull request is tested against the workflow tests and also against the low-level py.test (http://pytest.org/latest/) tests that @geky contributed.

The workflow tests in this PR are using CircleCI's YAML script file and it was requested for these tests to be locally reproducible, thus I provided an additional script that uses exactly the CircleCI YAML script.

A unittest implementation was not part of the requirements, but I'd like to welcome you to contribute unittest based tests which can complement the tests provided in this PR.

[pan-]

@screamerbg CircleCi is not a test framework or a test runner but in this PR it is used like one (in a hacky way like pointed out by @jupe ) and you have to provide another script to reproduce the behavior locally.

Why is this not written with py.test if it is already used by other tests ?

Then py.test test is enough to launch all the tests.

[pan-]

Why other tests are disabled now ?

[screamerbg]

Thanks for spotting this

[geky]

Question, what benefit is there for using unittest here? These aren't unit tests.

Maybe it would be better if circleci wasn't in the middle, maybe a list of commands in a py file or just a shell script, but this seems like a perfectly fine test to have.

[jupe]

Unittest framework give u a lot of benefits: e.g. proper results view (e.g. junit file), assert functions, coverage, test discover and runners, ... All what u need anyway.. Why to reinvent the wheel? Most of python based apps and tools we have uses this.

[jupe]

unittest framework doesn't mean that u couldnt create e2e tests with it.

[geky]

I mean, just because you can doesn't mean you should. This sounds like you're changing a tire with pliers.

It'd be nice to have a single entry point, but it's also nice to not have a ~10min test in the middle of a unit tests. I'm sure there's a command to filter it out, but at that point I'd rather just have two different commands for different tests.

Just my 2 cents. It's all going through circleCI anyways, and I like CI over no CI.

[sg-]

Just my 2 cents. It's all going through circleCI anyways, and I like CI over no CI.

I suggest we merge since tests are better than no tests and @jupe @pan- if you'd like to refactor based on a unit test framework please submit a PR.