Breaking Change in Interface?

[BrianMarre]

Thanks in advance for answering a stupid question,

In between version 4.15.0 and 4.16.0 the following code snippet broke

resolver = jsonschema.RefResolver(base_uri=RenderedObject._BASE_URI,
    referrer=uri,
    store=RenderedObject._schema_by_uri)

using the following arguments

base_uri = https://registry.hzdr.de/crp/picongpu/schema/
referrer = https://registry.hzdr.de/crp/picongpu/schema/picongpu.pypicongpu.species.operation.densityprofile.uniform.Uniform
store = (see attached file)

interesting section of store (as far as I can tell)

'https://registry.hzdr.de/crp/picongpu/schema/picongpu.pypicongpu.species.operation.densityprofile.uniform.Uniform': {
    '$id': 'https://registry.hzdr.de/crp/picongpu/schema/picongpu.pypicongpu.species.operation.densityprofile.uniform.Uniform',
    'type': 'object',
    'description': 'Describes a globally constant density.',
    'unevaluatedProperties': False,
    'required': ['density_si'],
    'properties': {
        'density_si': {
            'type': 'number',
            'description': 'density in kg * m^-3',
            'exclusiveMinimum': 0
        }
    }
}

with the following error message

ERROR: test_rendering_with_type (quick.pypicongpu.species.operation.densityprofile.densityprofile.TestDensityProfile)
object with added type information is returned & validated
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/picongpu/test/python/picongpu/quick/pypicongpu/species/operation/densityprofile/densityprofile.py", line 94, in test_rendering_with_type
    uniform.get_generic_profile_rendering_context()
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "~/picongpu/lib/python/picongpu/pypicongpu/species/operation/densityprofile/densityprofile.py", line 93, in get_generic_profile_rendering_context
    serialized_data = self.get_rendering_context()
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "~/picongpu/lib/python/picongpu/pypicongpu/rendering/renderedobject.py", line 189, in get_rendering_context
    RenderedObject.check_context_for_type(self.__class__, serialized)
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "~/picongpu/lib/python/picongpu/pypicongpu/rendering/renderedobject.py", line 207, in check_context_for_type
    resolver = jsonschema.RefResolver(base_uri=RenderedObject._BASE_URI,
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/site-packages/jsonschema/validators.py", line 736, in __init__
    self.store.update(
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/_collections_abc.py", line 999, in update
    for key, value in other:
  File "/opt/spack-modules/linux-ubuntu20.04-skylake_avx512/gcc-10.2.0/python-3.10.4-ez4r7hnfxopga5tpgir2fzxvek45oq7g/lib/python3.10/site-packages/jsonschema/validators.py", line 738, in <genexpr>
    for schema in store.values() if "$id" in schema
TypeError: argument of type 'bool' is not iterable

I am not entirely sure whether this is caused by a bug or a breaking change I missed, but I did not see anything I could identify on a quick scan of the changelog.

Thankful for every pointer, even a RTFM if it specifies a location ;)

test.txt

[Julian]

Thanks! I can try to have a look, but it'd be helpful if the example were something I could copy paste and run rather than assemble.

(Your referrer also looks suspicious, you're passing https://registry.hzdr.de/crp/picongpu/schema/picongpu.pypicongpu.species.operation.densityprofile.uniform.Uniform? That sounds like you're passing a str, but referrers are supposed to be schemas. I doubt that's related to your error though.)

[BrianMarre]

I can give you the actual code, it is all public, but it is rather lengthy and I did not have time yet to separate only the important stuff.

The code is accessible on github at, https://github.com/ComputationalRadiationPhysics/picongpu/tree/dev/lib/python/picongpu/

I will try to give you a smaller example but can not promise anything yet.

Regarding your second question: As far as I can tell it is a schema, specifically this one https://github.com/ComputationalRadiationPhysics/picongpu/blob/dev/share/picongpu/pypicongpu/schema/species/operation/densityprofile/uniform.Uniform.json

but I am not the original author, so take that with a grain of salt ;)

[Julian]

I suspect the commit I just pushed should fix this. I suspect you were referencing schemas which were bools and sticking those in your RefResolver -- I should say that though that of course even though that should work, it's sort of weird (having a boolean schema at some arbitrary URI seems useful only if you're trying to preserve backwards compatibility for something, or at least that's the only case I can think of). I'd double check you really mean to do that. But yeah it should work now.

[BrianMarre]

Well, we are not using them for the standard stuff.
If you are interested, here is a short version of what we do:

PIConGPU is scientific HPC simulation code. The user defines the input to the simulation in c++ files, which we compile directly into the simulation for performance reasons.

To make it easier to use we automatically create these c++-files from a user python script, by:

• taking an standardized object net defined by the user in his script
• converting it to serialized json description
• render them using mustache to create our c++ files from a set of templates

To make sure that the rendering engine will actually produce both semantically and grammatically correct c++-files, we verify the json input against a set of schemas before the actual rendering.

Definitively not a standard use case, but our use case ;)

And yes we are using arbitrary URI as IDs to for our validation schema and yes we have boolean switches for stuff like turning on/off an algorithm. according to the users wishes ;)

[BrianMarre]

Back to the topic, trying it out, will report back

[Julian]

What you're describing sounds "reasonably normal" to me, but is different from what would have triggered this issue, which is where you, in your:

To make sure that the rendering engine will actually produce both semantically and grammatically correct c++-files, we verify the json input against a set of schemas before the actual rendering.

step, you have some schemas which are literally just the JSON values true or false, i.e. they are boolean JSON Schemas (not flags) which either allow all instances (all C++ files in your case) or allow no instances respectively.

It's like doing trivial work (presuming your schema is true) or always blowing up (presuming your schema is false).

[BrianMarre]

Well I can report that the error is indeed fixed, but I will have to take a look if all is working as intended

Nevertheless, thanks for the quick fix.

[BrianMarre]

Found it,
'https://registry.hzdr.de/crp/picongpu/schema/picongpu.pypicongpu.species.operation.densityprofile.densityprofile.DensityProfile': False,

all in order, this checks that nobody passes a unspecialized profile, only specialized one are allowed