v6 annotation: custom error messages while validating

[handrews]

Originally written by @epoberezkin at https://github.com/json-schema/json-schema/wiki/Custom-error-messages-(v5-proposal)
with additional requests by @the-t-in-rtf at json-schema/json-schema#222

Add keyword errors that would contain error messages, potentially templated, that would be added to errors reported by validators when some keyword fails the validation. Example:

{
  "properties": {
    "age": {
      "minimum": 13,
      "errors": {
        "minimum": "Should be at least ${schema} years, ${data} years is too young."
      }
    },
    "gender": {
      "enum": ["male", "female"],
      "errors": {
        "enum": {
          "text": "Gender should be ${schema/0} or ${schema/1}",
          "action": "replace"
        }
      }
    }
  }
}

They can be merged using absolute or relative JSON pointers:

{
  "properties": {
    "age": { "minimum": 13 },
    "gender": { "enum": ["male", "female"] }
  },
  "errors": {
    "#/properties/age/minimum": "Should be at least ${schema} years, ${data} years is too young.",
    "#/properties/gender/enum": {
      "text": "Gender should be ${schema/0} or ${schema/1}",
      "action": "replace"
    }
  }
}

[handrews]

I designated this an "annotation" proposal, but it really seems to be about enabling a tool/library to present customized instance-dependent error messages to the tool's callers. This feels to me very much like a UI generation feature, and it is a matter of some debate as to whether comprehensive UI generation falls within this specification or not.

My personal view is that this is not within the scope of JSON Schema, although that is a discussion that needs resolution in issue #55 . This seems like it should go in some sort of extension to JSON Schema that specifically adds a layer of functionality for building and interacting with UIs.

This proposal goes well beyond the normal documentation goals of JSON Schema, as the rules expressed in these errors could easily be included in a description string. That would be sufficient to document the requirements in a more human-friendly way than whatever the validator spits out.

We need to get clear on the scope of JSON Schema, and then I think it will be obvious whether this fits or not.

[handrews]

For an opposing view, here is a comment by @the-t-in-rtf at json-schema/json-schema#222 :

@erosb and @saibotsivad, the v5 draft already has custom error messages in it. I am arguing for clarifying how one type of rule is handled, but I'll take a minute to explain why (I feel) the new errors construct is definitely needed.

@saibotsivad, until I saw the v5 draft, I used the description field to provide more human-readable errors. However, requiring schema authors to use that field for errors seems limiting. They cannot use the description field to (for example) describe to implementers the purpose of the field if they have to use the field for errors. There is also no support for displaying schema or user-supplied data in descriptions, which is also part of the proposal.

Setting both of those aside for a minute, let's talk about internationalization and reusing schema material for a minute. The draft v5 proposal gives an example of using document-level errors to define deep custom error messages at the document level. This means that you can refer to material within another schema and then add or provide alternate error messages. @YurySk, this could provide one approach to internationalization, i.e. you could have a schema-jp.json that wraps the material for schema.json and provides alternate error messages in Japanese. It also provides a nice way to transition from v4 schemas to v5 compatibility, where you can add the new errors material without rewriting an existing remote schema that might not be under your control.

To answer @erosb, the reason this needs to be handled within an errors construct rather than by the validator is that it gives schema authors the ability to clarify their intent from within the schema. Handling it by convention and requiring validators to take care of this work means you need presentation logic that interprets and generates a custom message based on the deep structure of the error. With the new errors block, you can reasonably expect to simply present the custom error message itself. You could do something similar with description fields, but because these cannot be overridden or added at the document level, you would have less options in reusing schemas or internationalizing them as outlined above.

[handrews]

Note that we now have issue #67 for features related to UI generation, if this ends up being more appropriate there.

[awwright]

I don't view internationalization as being a JSON Schema concern, that's something that can be handled by other software or specifications. There's specific things that implementations can do, however, to ensure that software is easier to localize, like providing specific error codes and arguments, that can be translated into English or Japanese or Sindarin or whatever.

For fields like "description", I see those as being for developers only, not for end-users. However to the extent it still would need to be translated, we might also look at offering links to "translated" JSON Schemas, the same way HTML can negotiate and link to a localized version of a document when you request it.

In general, let's look at how other hypermedia systems deal with localization, I can't imagine JSON Schema will need to be too different.

[Anthropic]

@awwright @handrews JSON Schema Form support the error list that Geraint added to tv4 which has numbers associated with the errors, these could act as a preliminary list to build on in a ui-schema.

I totally agree with you that description should not be used for UI, unfortunately JSF was made to use description beneath any form field, which I want to change, but I need an optional ui-schema 'hint' field to help migrate users over to.

Regarding error messages, the Schema Form implementation is currently:

  {
    "key": "address.street",
    "validationMessage": {
      "302": "This field is like, uh, required?"
    }
  }

With the option to add a global list rather than per field of course.

[the-t-in-rtf]

I don't view internationalization as being a JSON Schema concern, that's something that can be handled by other software or specifications. There's specific things that implementations can do, however, to ensure that software is easier to localize, like providing specific error codes and arguments, that can be translated into English or Japanese or Sindarin or whatever.

As long as it's possible to override error messages from a higher level (the root of the schema, an enclosing field), that's enough for me. With that I can reuse all or part of a schema and change the user-facing errors to be context-appropriate. I can also override the error messages with i18n keys if I like and, as you say, handle the rest elsewhere.

[handrews]

The responses to this are some combination of "this can be handled elsewhere" (JSON UI Schema, a higher-level I18N layer which is addressed by several other open issues already) and "this isn't needed", and there's been no further comment for three weeks. I don't have any need for this particular proposal myself.

I'm closing this. If anyone wants to champion it, feel free to re-open it.

[the-t-in-rtf]

I'd reopen it if I could. My last comment was in fact still arguing for it, I would have appreciated a chance to respond. The "errors" keyword was such a huge improvement, it gave schema authors a way to easily present context-appropriate error messages, and across validators. It increased the reusability of schemas across validators and environments, and reduced the work needed to support basic localization and internationalization.

Regardless, I will follow the json-ui-schema discussion and try to keep an open mind.

[handrews]

@the-t-in-rtf I was not trying to shut this down, I just got a different impression from your last comment. I can re-open this, or if you'd rather own the issue you can re-file it.

I've just been getting pushback on opening issues that were left open at the old repo, so I'm trying to close anything I can.

I do think that this and other I18N/L10N concerns belong somewhere else, but that's my personal opinion and not authoritative for the JSON Schema project.

Anyway, if you don't want to own a new issue just comment here and I'll re-open this.

[the-t-in-rtf]

@handrews, I was honestly disappointed that a partially supported draft feature seems to be getting pruned away, but I understand that "draft" is not any kind of guarantee, and I do see the point that you and others have make about separating the concerns.

The real turning point for me was in realizing that the standard outlines what is and isn't valid, but doesn't outline how it should be reported back. Even with the "errors" property, you'd have to deal with how the validator chooses to report the failure, and that would be specific to each validator. Hopefully the JSON Schema UI extension will eventually provide a standard means to customize error output.

For now, it's pretty clear we need to either commit to using the error customization mechanism provided by one of the common validators, or to write something ourselves.

[awwright]

I'm of the opinion that error messages are localized strings, and should be defined along with all of the other localized strings in an application. JSON Schema validators should report back what the failed keyword was, and you can use this information to pick a suitable error message in your application.

[handrews]

@the-t-in-rtf note that this general topic has been revived as #396