Description
The most contentious topic in JSON Schema, for many years, has been what sort of re-use functionality should be supported. In particular, how to both use "additionalProperties": false to prevent misspelled property names, and allow adding more property names when re-using such a schema.
This is not actually the intended purpose of "additionalProperties", and as of draft-06 "propertyNames" is preferred, but can be cumbersome. Also, many years of telling people not to use "additionalProperties" in this manner have had no effect whatsoever.
The proposal that attracts both the strongest support and strongest objections is #15 "$merge"/"$patch". It is by far the most powerful, but also breaks the theoretical model of JSON Schema validation as a constraint system in which you can add but never remove constraint. These two keywords allow arbitrary transformations.
We need to balance several things:
- Feature support for people who consider this a barrier to adoption
- Theoretical purity for those who do not need and do not want these features
- Interoperability for both groups, either uniform support or a clear fail-fast outcome
The simplest way to implement this is to allow declaring that your schema requires an optional feature. The implementations can either process your schema, or error out with a message that the feature is not supported.
This is only needed for extensions to core, such as "$merge" and "$patch". Unlike "format", which is still of use to applications even when not used for validation, and does not change the behavior of anything else when it is ignored, there is no possible way to use a schema involving "$merge" or "$patch" without supporting those keywords.
So I am proposing adding "$requires" to core as an extensibility mechanism. The value of this keyword MUST be an array. Elements in the array MUST be URIs identifying the extensions required to properly make use of the schema.
Implementations MUST error out if they do not support, or do not recognize, a required extension.
Implementations MUST NOT automatically attempt to dereference the URI, as such URIs need not be dereferencable at all, and the canonical server should not be overloaded by unnecessary requests.
The initial declared extensions would be something like http://json-schema.org/draft-07/extensions.$merge or tag:json-schema.org,2017-10-15:extensions.$patch depending on whether we want to stick with HTTP URIs or use an obviously not-dereferenceable scheme such as tag.
"$merge" and "$patch" would be separately declared extensions, as different users prefer different approaches, needing one or the other or both.
Also paging @erayd
Metadata
Metadata
Type
Projects
Relationships
Development
Participants
Activity
erayd commentedon Sep 1, 2017
A few thoughts - posting this from my phone, so briefly:
Can we call it $features rather than $require? That feels more semantically accurate, and prevents confusion with languages / frameworks that use 'require' as an inclusion keyword.
If we're using a URI to refer to a feature, that URI could also point to a meta-schema that validates use of the feature - essentially the same role that $schema serves for the core stuff.
If this proposal goes ahead, it gives custom keywords a proper home. Noting this, can we then go ahead and ban undefined keywords completely? I.e. a keyword must be defined by the core or an enabled feature in order to be valid, and unknown keywords result in a validation fail.
erayd commentedon Sep 1, 2017
To further clarify my point (2) above, I'm not suggesting that implementations dereference this, merely that defining the URI as the canonical home of a meta-schema that validates the referenced feature makes sense, and allows used to easily locate the definition for any given feature.
Anthropic commentedon Sep 1, 2017
@erayd re: 3, I think failing on undefined keywords would be a huge pain without a prefix based comment exclusion. I always test schema by slashing keywords "//keyword" to reduce what I am testing, it would be an unwelcomed outcome to not be able to easily exclude large parts of a schema when testing due to validation errors I don't want.
erayd commentedon Sep 1, 2017
@Anthropic
That's a really good point. What if the default was to ban undefined keywords, but there was also a mechanism to turn that constraint off?
Anthropic commentedon Sep 1, 2017
I'd prefer a definable prefix exclusion, but that's a whole other discussion already going on in the
$commentthread I've been meaning to get back to :)As for this I agree the name doesn't feel right to me, I'd prefer something like
$keywords,$usesor$requiredSpecs. Not set on anything clearly, but$requiresis too close torequiredfor my comfort.18 remaining items
handrews commentedon Sep 2, 2017
@erayd that's a good point. If it's not a general extension mechanism, why have one at all?
The only thing I can think of is that it makes the nature of the support very clear to have to declare that you require it. I don't think I'm quite articulating what I mean here, I'll have to think on it. But I could see just documenting that the keywords MUST fail-fast with these specific keywords, which is part of giving them a trial run.
While the extensibility feature would be rushed for draft-07, adding $merge/$patch support wouldn't really, seeing as they were first proposed in March of 2014 (at the old repo).
To say that there is "problematic history" is an understatement.
erayd commentedon Sep 2, 2017
@handrews
Indeed. I'd love a general extension mechanism, simply because it feels neater - i.e. it gives a logical home to keywords that aren't part of the official spec - but I don't want to push for it strongly if the concensus is that one isn't desired; IMO reaching an agreement is more important.
What about the following?
$mergeand$patchmake it into the spec for draft-07, but are explicitly noted as experimental / may vanish in a later version / may bring about the early heat-death of the universe etc.$merge/$patch) MAY be supported, but MUST fail validation if unsupported and present in a schema."$experimental": true.Does that sound like an acceptable compromise?
handrews commentedon Sep 2, 2017
@erayd I like that, let's see what @Relequestual thinks.
There are still several other major project members that have not recently weighed in on seriously moving forward with $merge/$patch (whether it's experimental or not) and I want to wait for them to reach a consensus.
erayd commentedon Sep 2, 2017
I agree, waiting for more input before proceeding is a good thing.
Relequestual commentedon Sep 6, 2017
I dissagree with this sentiment, but I can understand why you might feel this way. However...
... I totaly agree with this, and feel that...
... sounds like the right solution.
To paraphrase to make sure we're on the same page and I'm not missunderstanding here...
Having experimental key words which MAY be supported but MUST cause validation failure for a schema that includes them, if not supported by the validator, sounds like an ideal solution moving forward. Testing the water, and evaluating uptake from implementors and json schema authors feels like a great way to validate the use case, without forcing compliance and without causing unnesessery unrest!
I'm always happier to validate the requirement to make key changes if it's going to really upset thigs. This is a solution I'm happy with.
I think forcing people to use "experimental: true" would make them realise that it may not be supported, and to look into the spec arounf these key words closer. Experimental sounds better than Requires, personally, but that's just a name for the key word.
If you're dead set against it, are you still happy to trial them as experimental keywords? If they get uptake, and it seems people want this and it doesn't cause problems, would you then consider alternative technology and leave the project?
Maybe it's because I haven't used JSON Schema enough or written written a validator, but I don't understand what the conceptual model is, why it's important, or why we should care if that changes to make the specification more useable. I'm not suggesting we discuss that specifics here (maybe slack sometime), just pointing out my opinions may be less valid given my lack of understanding. Also, I can't be the only one to think this way.
dlax commentedon Sep 6, 2017
To me,
$requiresclearly indicates that there's extension required. But since having a "generic" extension mechanism is not considered for draft-07, this keyword would be misleading.By the way, I think that @erayd's proposal #388 (comment) would be a nice and sane step forwards. And I guess that if the feature gets popular enough, it could then be moved from the experimental side to an extension, once an extension mechanism gets defined.
Relequestual commentedon Sep 6, 2017
I'm really against having an "extensions mechanism" unless it's only to work out which key words are useful while the spec is still in draft. Mainly because I want to totally avoid the chance of fragmentation (although you could argue the draft versions already create this situation).
As for
$requiresvs something else... I'm happy with$requiresif others feel it fits.For now, I'll say let's +1 this (the issues proposal) forward. Would the URI's be expected to resolve, and if so, what to, and for what purpose? @handrews ?
handrews commentedon Sep 6, 2017
I suppose that came across as more snarky than intended :-P
I just mean that the point of the draft process is to try things and see whether they are worth proceeding to RFC or not. Putting something in a draft doesn't mean it will make RFC, nor should it. We've ripped out several things since draft-04, particularly in hyper-schema.
JSON Schema is not my personal project.
That's all there is to it.
No, I'm not happy to trial them. I don't need a "solution" to this whole thing that I don't consider a "problem" in the first place. But.... JSON Schema is not my personal project. I think we have an obligation to listen to the community. Maybe draft-07 is not the right time. Maybe we should wrap up all of the format, hyper-schema, and application/schema-instance+json work as draft-07, and have a full six months of debate just on how to move this forward.
But either way, my personal distaste for this is irrelevant. I reserve the right to be a grump about it (up to a point, anyway).
Probably not.
It's remotely possible, but if the keywords really don't cause problems, then I won't have much real cause for objection and would have to somehow sell my colleagues on switching technologies. That would be a high bar. FWIW, I have previously left projects that considered me essential in some way and they kept going just fine. No one person is really that important; someone else would step up.
It's the sort of thing that I care a lot about and other people often don't. I like to work in abstractions and applying them. I dislike synthesizing from examples. Most people work the other way around. The way I approach things is neither superior or inferior, it's just a different perspective. It helps with some things and gets in the way of others. You can decide which is going on here :-)
It sounds like @erayd's
$experimentaloption is the most broadly supported. I like it as well.Let's think a bit more on whether this is the right draft to go ahead with here, or if, since we have avoided really resolving this for years, perhaps a few months of rigorous debate focused on producing a solution (I argue we have just debated aimlessly to date) would make us all happier with the path forward.
handrews commentedon Sep 6, 2017
Note that if we decide to go a few more months on this, we could let it be the sole driver of draft-08 which could go out as little as 2 months after draft-07. We don't have to wait six months.
Relequestual commentedon Sep 7, 2017
ha fair enough.
When I say "you" in that context, I meant as an editor of the specification, not as a consumer or implementor of it in your daily work. But I think you clarified that if the community wants it (which I think it is), then we should TRY work it into a draft spec and see if it causes issues or not.
If we do add it (and I think we're in agreement that we should try in maybe draft-8), then you're on under no obligation to use it in your schemas. And it's not like (from what I understand) you're day job involves consuming schemas created by other people.
Yeah, I think that's what I've tried to clarify with the above.
Duly noted =D
I'm totally open to hearing about it! I was going to ask if you'd write a (short, brefi) wiki page on the conceptual model what it is, why it's important, how it should / can be applied to thinking about making changes to the spec. I'm happy to write my questions as headers as a wiki page to get you started, if you're up for doing it. (I bet it would help me understand and maybe others too). Of course if you want to wait till after draft-7 (or just would rather not), that's fine also.
Yeah, I totally agree on this one! =]
erayd commentedon Sep 7, 2017
Noting how contentious the whole issue around
$merge/$patchet al has been, this feels like a really good idea. Making it the sole focus for draft-08 removes distractions and hopefully means we'll get a better concensus on what the best path to proceed might be. It also means that there is more opportunity for discussion rather than trying to rush something into draft-07.handrews commentedon Sep 10, 2017
As explained in the email to the mailing list, I'm going to close this as the generic extension mechanism isn't really motivated by use cases, and we're deferring the merge/patch decision to the next draft. If we still want this general sort of approach then, we will open a new issue focusing on the
$experimentalform of this proposal which attracted more support. But we don't have any other use cases for it so I'm not going to open it immediately. Only if we need it.