Add Tests from json-schema/JSON-Schema-Test-Suite GitHub Project

[mchiocca]

I added json-schema/JSON-Schema-Test-Suite as a repository in composer.json with an explicit reference to a newly created tag. When composer install is executed, the repository will be cloned and its contents placed in the vendor directory. I added a new test that executes all of the draft-03 tests from that repository except for a few that are marked as skipped. So, full draft-03 testing is done using the tests from the JSON-Schema-Test-Suite GitHub project. I also added a new test that executes a majority subset of the draft-04 test suite. Some draft-04 functionality is not yet implemented so those tests are currently being skipped.

[mchiocca]

@justinrainbow. Scrutinizer failed to run the PHP analyzer successfully, but I don't know why. Travis passed and both PHP builds were successful so all PHPUnit tests passed. I can't seem to schedule a new Scrutinizer run. Can you please investigate? Let me know if there is something I need to do to fix the problem and I will add a new commit to this request. It would be great to merge this request since it adds the full draft-03 test suite and makes available the draft-04 suite as well.

[mchiocca]

@justinrainbow. Hello. I thought perhaps that the first Scrutinizer failure might be a fluke so I added draft-04 unit tests, fixed draft-04 format issues with ipv4 and hostname and pushed those changes hoping that the build would succeed. The tests pass, but alas, Scrutinizer once again failed. Here is what it says from the log.

PHP Analyzer Run failed.
All tools exited non-successfully. Please check the log for details, and try running the inspection again.

I don't see anything in the log that indicates why PHP Analyzer failed. I've contacted Scrutinizer-ci support and have asked them to investigate. I can't schedule a new Scrutinizer run, but you probably can. So, can you please take a look at the Scrutinizer problem and let me know if there is something I need to do to fix it?

I noticed that .travis.yml has a before_script that runs composer install and that the Scrutinizer configuration has a "before_commands" so perhaps composer install could be added to "before_commands"? I don't know if that will fix the problem or not.

Other successful Scrutinizer runs don't download all of the dependencies. It just says that they are all up to date. But for some reason, Scrutinizer is downloading the dependencies after it clones JSON-Schema-Test-Suite. It doesn't appear that the PHP Analyzer is running correctly based on what I see in the log.

Any help with this would be greatly appreciated. It would be great to have the draft-03 and draft-04 test suites included in with this project. Thanks!

[mchiocca]

@justinrainbow. Yes. It looks like adding composer install to "before commands" may solve the problem. Look at the Scrutinier documentation for Custom Commands. You'll see this.

before_commands:
    - composer install

So if you change the Scrutinizer config file for the json-schema project accordingly, the Scrutinizer issue may likely go away. Please consider trying this. If it works you'll be able to merge #43.

[mchiocca]

@justinrainbow. Hello. As you can see from #46, demand is growing for draft-04 support, which we would like to begin working on. In order to do that, it is very important that this pull request be merged since it adds the full test suites for both draft-03 and draft-04. Unfortunately, Scrutinizer CI is failing. Can you try adding composer install to before_commands in the Scrutinizer CI configuration file? I'm hoping that that change will fix the CI failure issue. Thanks. Very much appreciated!

[justinrainbow]

I ran the tests and Scrutinzer CI stuff locally and everything checked out. The only change was moving the new dependency into the require-dev group so it doesn't need to be installed if you have no need to run the tests.

[mchiocca]

Hello and thanks for incorporating the test suites! I'm having a Scrutinizer issue though. I created an account on scrutinizer-ci.com and added my repository. I fetched/pulled your changes (new .scrutinizer.yml file, etc.) and then pushed to sync my repository with yours. But Scrutinizer is timing out. It looks like the composer install --dev in before_commands of .scrutinizer.yml isn't being executed. It's instead being done by activation of PHP Analyzer. Scrutinizer is timing out waiting for builds to finish. Have you had similar issues with running Scrutinizer on your repository on scrutinizer-ci.com? Here is my log. Does this look like your log?

Scheduling package installation
Installing Package
> $ git clone 'https://github.com/mchiocca/json-schema.git' '/home/scrutinizer/package'
> Cloning into '/home/scrutinizer/package'...
> $ cd /home/scrutinizer/package
> $ git checkout -qf 61ee8fe93aac9561a365edf24351cc8918e6e049
Package was installed. Found Config Files: .scrutinizer.yml
Installation succeeded; scheduling parsing of configuration data
Parsing configuration files in repository
Parsing ".scrutinizer.yml"
Configuration does not inherit, finalizing
Scheduling PHP Analyzer Run
Dependencies are not yet available, scheduling analysis
Installing PHP dependencies through composer
> $ composer config -g notify-on-install false
> $ composer install --dev --no-progress --no-scripts --no-custom-installers
> Loading composer repositories with package information
> Installing dependencies (including require-dev) from lock file
>   - Installing json-schema/json-schema-test-suite (1.1.0)
>     Cloning 1.1.0
> 
>   - Installing phpunit/php-token-stream (1.1.5)
>     Downloading
> 
>   - Installing symfony/yaml (v2.2.1)
>     Downloading
> 
>   - Installing phpunit/php-text-template (1.1.4)
>     Downloading
> 
>   - Installing phpunit/phpunit-mock-objects (1.2.3)
>     Downloading
> 
>   - Installing phpunit/php-timer (1.0.4)
>     Downloading
> 
>   - Installing phpunit/php-file-iterator (1.3.3)
>     Downloading
> 
>   - Installing phpunit/php-code-coverage (1.2.11)
>     Downloading
> 
>   - Installing phpunit/phpunit (3.7.21)
>     Downloading
> 
> phpunit/phpunit suggests installing phpunit/php-invoker (>=1.1.0,<1.2.0)
> Generating autoload files
Dependencies were installed, calculating dependency builder order
Scheduling analysis for phpunit/phpunit
Moving phpunit/phpunit to persistent storage
Scheduling analysis for mchiocca/json-schema
Moving justinrainbow/json-schema to persistent storage
Dependency analysis completed, waiting for their builds to finish.

Also, you might want to update the README.md where it says php composer.phar install by adding the --dev option, which is needed in order to run the tests with phpunit. Thanks again!

[mchiocca]

Hello @justinrainbow. I contacted Scrutinizer-ci support about the issues with downloading, installing, and building PHP dependencies. They can't provide an immediate analysis and resolution to the problem. All they did was suggest to disable PHP Analyzer for the time being since it's causing Scrutinizer to constantly fail. Not a very satisfying response, but that's all they could recommend for now. Thanks.