Expansion does not take property-scoped contexts for nested properties into account

[niklasl]

There is currently a limitation in the JSON-LD 1.1 expansion algorithm which ignores property-scoped contexts for nested properties.

My specific case is:

{
  "@context": {
    "@version": 1.1,
    "@vocab": "http://purl.org/dc/terms/",
    "bibo": "http://purl.org/ontology/bibo/",
    "Print": "bibo:Book",
    "name": "http://www.w3.org/2000/01/rdf-schema#label",
    "instanceOf": "@nest",
    "contributionByRole": {
      "@id": "@nest",
      "@context": {
        "agent": "@nest",
        "aut": "creator"
      }
    },
    "provisionActivityByType": {
      "@id": "@nest",
      "@context": {
        "Publication": {
          "@id": "@nest",
          "@context": {"date": "published", "agent": "publisher"}
        }
      }
    },
    "identifiedByType": {
      "@id": "@nest",
      "@context": {
        "Isbn": {"@id": "@nest"},
        "value": "bibo:isbn"
      }
    }
  },
  "@id": "book/one",
  "@type": "Print",
  "instanceOf": {
    "contributionByRole": {
      "aut": {
        "agent": {"name": "Some Body"}
      }
    }
  },
  "identifiedByType": {
    "Isbn": {
      "value": "1234567890"
    }
  },
  "provisionActivityByType": {
    "Publication": {
      "date": "1999",
      "agent": {"name": "PubCorp"}
    }
  }
}

From which I expected to get (recompacted for brevity):

{
  "@context": {
    "@vocab": "http://purl.org/dc/terms/",
    "bibo": "http://purl.org/ontology/bibo/",
    "name": "http://www.w3.org/2000/01/rdf-schema#label"
  },
  "@id": "book/one",
  "@type": "bibo:Book",
  "bibo:isbn": "1234567890",
  "published": "1999",
  "publisher": {"name": "PubCorp"},
  "creator": {"name": "Some Body"}
}

(This is actually part of a "trick" (a "context switcheroo" if you will), whereby by compacting BibFrame data into indexed maps, and then swapping out the context with another one mapping to DC+BIBO, as per above, using nests with local contexts, the same structure is semantically differently interpreted (a kind of "poor man's inference", supporting different vocabulary granularities). The input for the above example is thus actual BibFrame data plus an idiomatic context for the shape, using indexes.)

That aside, I do believe that in general, for JSON-in-the-wild, it is quite plausible that structures mapped using nesting properties will vary the meaning of keys within each nest. As it stands, the current algorithm, somewhat arbitrarily IMHO, lacks this possibility. Nothing in the Syntax spec makes it obvious that this is so. In the test suite itself, in the example of test case in06, you could quite plausibly imagine that keys within the defined nests ("links", "attributes" and "relationships") could have different meanings but share the same name (e.g. "links" could use a "title" key with a different meaning from "title" within "attributes").

Furthermore, as far as I can tell, index keys in indexed properties can have property-scoped contexts (assuming that I interpret test case c013 correctly). To support that but not nests is rather unexpected, as they both represent not properties but "sectioned" data.

For reference, see my original question about this on the Public JSON-LD mailing list, and @gkellogg's reply, which included (and I quote):

There was no specific intention to not support this, but looking at the expansion algorithm, it brings up more considerations: for example, we don’t support @context at the top-level of nested properties either.

Nominally, to do what you want would be to duplicate step 8 (including seeing if the key expanding to @nest has an @context defined on it), and see if nesting-key has an embedded context defined, and update active context for the recursive steps; we’d also need to revert after 14.2. Similar steps would be required in the compaction algorithm. We’d also need to consider if we wanted to act on an @context member of nested value after 14.2.2, and exclude the recursive step in that case.

It would be a fairly big change, IMO, and given that we’re about to release an updated CR, probably too late to do it.

Following up on that, I wonder what would happen if step 14.2.2 is just changed from:

    <li>Recursively repeat steps <a href="#alg-expand-each-key-value">13</a>
      and <a href="#alg-expand-resolve-nest">14</a>
      using <var>nested value</var> for <var>element</var>.

to:

    <li>Recursively repeat steps
      <a href="#alg-expand-initialize-property-scoped-context">3</a>,
      <a href="#alg-expand-property-scoped-context">8</a>,
      <a href="#alg-expand-each-key-value">13</a>
      and <a href="#alg-expand-resolve-nest">14</a>
      using <code>@nest</code> as <var>active property</var>, and
      <var>nested value</var> for <var>element</var>.

Would this really require reverting, as opposed to the regular active context state handling done in the algorithm? (Assuming of course that "recursively repeat" here, as is already implicit both here and elsewhere in the algorithm, properly scopes the passed variables to each step without polluting the other steps in the loop.)

(Note that step 3 also has to be assigned id="alg-expand-initialize-property-scoped-context" in the markup if the above is added.)

(Although @gkellogg also noted that the algorithm neither supports local (explicit) contexts at the top-level of nested properties, I consider that a separate (potential) issue for which I see less motivation for. As I see it, the main reason for these features to exist ought to be to handle idiomatic JSON with zero edits, and not "native" JSON-LD. I see no practical reasons for it (I personally see little practical reason for explicit local contexts at all in 1.1, since scoped contexts are possible).)

I do believe that there is a missed opportunity here, especially considering the complexity which has been invested into nesting properties and scoped contexts. The "zero edits" goal, which much of JSON-LD 1.1 has put its force behind, warrants this to be thoroughly considered, even at this late hour, IMHO. I'll do what I can to help, if that's needed.

[gkellogg]

If the notion of "Recursively repeat ..." implies a separate variable scope, then steps 3 and 8, which modify active context could be considered an isolated scope and don't affect the value of active context for the next iteration.

At the least, the Note following 14.2.2 would need to clarify that those steps are done in a scope that does not allow those variables to change on successive repetition. Although, note that they are updating result, which could be considered a pass by reference.

[gkellogg]

Also, I don't see a reasonable way to update the Compaction algorithm, so this would be a one-way transformation. Ideally, compacting a document with a context which uses a scoped-context on @nest would get back to the same place, but steps 12.7 and 12.8 of the Compaction Algorithm, which deal with nesting find terms using the existing active context, The inverse context is based on that, which is where we look for potential terms to return as item active property. It would require that the inverse context also include nested contexts for any in-scope nest properties having a scoped context so that they could introduce terms that make use of that scoped context, but this would lead to overlap and would be crazy complicated.

I'll do a PR that does expansion-only scoped contexts on aliases of @nest with some note that it doesn't work for compaction.

[gkellogg]

@niklasl I addressed this in #388, which includes your specific example as a test case, along with a simpler on.

As I mentioned, not that it doesn't support compaction. (probably should say so).

Please let us know if this addresses your issue.

[pchampin]

As I mentioned, not that it doesn't support compaction. (probably should say so).

I'm worried that we can not keep the compaction algorithm untouched, though.

Consider the following context

{"@context":{
    "foo": "http://example.org/foo",
    "bar": {"@id": "http://example.org/bar", "@nest": "nst"},
    "nst": {"@id": "@nest", "@context": { "bar": "http://different.example.org/bar" }
}}

used to compact

{ "http://example.org/bar": "BAR }

I'm under the impression that it would result to

{ "@context": {...},
  "nst": { "bar": "BAR" }
}

in which bar is now interpreted as http://different.example.org/bar...

[gkellogg]

That is indeed a problem, but I don't see how to solve it in compaction reasonably; maybe we can't support this feature.

[pchampin]

I don't see how to solve it in compaction reasonably

Steps 12.7.2.1 and 12.8.2.1 of the Compaction algorithm currently read:

If nest term is not @nest, or a term in the active context that expands to @nest, an invalid @nest value error has been detected, and processing is aborted.

We could add a condition that prevent compacting to a nesting property with a scoped context:

If nest term is neither @nest nor a term in the active context that expands to @nest, or if nest term is a term whose term definition contains a @context entry, an invalid @nest value error has been detected, and processing is aborted.

This wouldn't break existing test cases (as they don't have scoped context in nesting properties.

maybe we can't support this feature

On the expansion side, it is a natural addition, and @niklasl 's use case is valid. On the compaction side, it does open a can of worms, which we don't have time address more elegantly. I think however that this solution (including my patch above) is a move in the right direction, and does not preclude a future WG to improve the compaction in order to better support this feature.