Roadmap

[nickl-]

@Julian have you given the road ahead much thought?

Some food for thought:

• sanity checking intact - as per Automate sanity checking the suite #14
• implementation test script - the actual tests using the specific supported library complete with installation of latest release and implementation unit tests to ensure satisfaction.
• CI automation - hook up to travis to automate sanity checks and implementation scripts

Lets elaborate...

[Julian]

I want to get #14 with the sanity check + Travis implemented by the end of the day or tomorrow, so yeah that's on my mind :).

Can you elaborate on what you mean by your second point? There are already a number of implementations using the test suite (including my own).

[Julian]

Oh and, not sure if this is included in the type of roadmap you had in mind, but I'd like to get specs added for whatever nooks and crannies of the draft 3 schema that I've overlooked (getting some more eyes on that would help :) and then to start adding / moving specs to test JSON Pointer / references, and then after that start on draft 4.

[nickl-]

Anything that needs doing are acceptable as signposts on the roadmap, in no particular order either.

Feel free to edit the first post to add new items and move completed items to a "completed" list perhaps.

following up on No2 shortly

[nickl-]

Number 2 might blur the boundaries of scope perhaps. What I am suggesting is that we collect a list of supported libraries/json-schema validators and that we do the compatibility test implementations for each.

This would help with collecting data for the sites implementations page allowing us to make indisputable recommendations based on first hand experience and the numbers don't lie.

If out of scope, perhaps a better approach is to create sub project for each to be maintainable separately and allowing their CI tests to be executed and reported on separately.

Perhaps something that would be in scope is an executor of sorts. I command which will run through the tests and in turn passes as command line argument the json uri to validate. considering the exit status and compiling a report of the results. This would simplify adoption by minimizing the effort required to execute tests and producing normative reports. We could go one step further perhaps and capture these results on a central database which we can query again for previously mentioned stats requirement. Thoughts?

Has any libraries, that you know of, adopted these test assets themselves yet, as of date?

[Julian]

Aha, OK think I'm getting it a bit more.

I like the idea of being able to quantify how well implementations that are listed in the site perform on the suite a lot. That definitely sounds good to me,

The executor also sounds good. So you mean something like

jsonschema_suite foo

where jsonschema_suite consecutively invoked the command foo with each schema and instance and records exit statuses for each one? That sounds like a good thing to have as an alternative to the way things are using this now.

If you want to see something using this, my validator loads and runs these tests here and essentially this comprises 90% of my suite.

The full list is in the README, the other validators basically do similar things with jsunit and whatever Haskell thing.

[Julian]

Some of this has been done, so I'm gonna move the remaining idea here to its own ticket.

[nickl-]

@Julian You've been busy busy busy, well done! =)

Catch you on #20,