(DOCS) Define and document schema for resource manifest #90

michaeltlombardi · 2023-06-26T23:06:27Z

PR Summary

This commit converts the type definitions in the dsc_lib\src\dscresources\resource_manifest.rs file into a minimal JSON schema.

PR Context

For users to be successful in implementing DSC Resources, they'll need both a JSON schema to validate their manifests and to refer to documentation for the resource manifest that goes beyond reading the JSON Schema.

The schema represents the current state of the type definitions, but raises a few questions for discussion. See the comments in the schema file.

The documentation for the schema is currently hand-authored and the structure is experimental, since we have no official support for documenting JSON schemas on the Learn platform.

For readability and referability, I recommend breaking the larger schema up into smaller schemas and referencing across them.

SteveL-MSFT · 2023-06-26T23:19:48Z

docs/schemas/resource-manifest/schema.md

+To be recognized by DSC, every command-based DSC Resource must have a DSC Resource manifest. The manifest must:
+
+1. Be discoverable in the `PATH` environment variable.
+1. Follow the naming convention `<name>.resource.json`.


We should have guidance on <name>, since the <name> is not used as the type name (since it's in the manifest), we should probably recommend namespacing to avoid name collisions (which would only affect if in the same directory).

Updated with specific guidance:

DSC/docs/schemas/resource/manifest.md

Lines 42 to 67 in a7ecc03

### type

The value of this property must be the name of the DSC Resource in its namespace. This value must

use the following syntax:

```text

`<owner>[.<group>][.<area>]/<name>`

```

Each component must be string of alphanumeric characters and underscores. No other characters are

permitted. Every DSC Resource must define an `owner` and a `name`. Use the `group` and `area`

components to organize DSC Resources into related namespaces. For example:

- `Microsoft.SqlServer/Database`

- `Microsoft.SqlServer.Database/Role`

- `Microsoft.SqlServer.Database/User`

- `Microsoft.SqlServer/Endpoint`

- `Microsoft.SqlServer.Endpoint/Permission`

- `Microsoft.SqlServer/Login`

- `Microsoft.SqlServer/MaxDop`

```yaml

Type: string

Required: true

Pattern: ^\w+(\.\w+){0,2}\/\w+$

```

SteveL-MSFT · 2023-06-26T23:20:25Z

docs/schemas/resource-manifest/schema.md

+| Metadata Key |                   Metadata Value                    |
+| :----------: | :-------------------------------------------------- |
+|  `$schema`   | `https://json-schema.org/draft/2020-12/schema`      |
+|    `$id`     | `https://aka.ms/dsc/schemas/resource/manifest.yaml` |


Is there Microsoft precedent already for this type of URL?

I'm not sure - I put it in as a placeholder until we have a canonical URI for the schema.

docs/schemas/resource-manifest/schema.md

SteveL-MSFT · 2023-06-26T23:24:37Z

docs/schemas/resource-manifest/schema.md

+### type
+
+The value of this property must be the name of the DSC Resource in its namespace. The name of the DSC Resource should be separated from its namespace by a forward slash (`/`). For example, a
+resource for managing a SQLServer database might define its `type` as `SQLServer/Database`.


Are there existing Microsoft/Nuget namespacing rules we can point them to? It should be something like <company>.[group.]<area>. So minimally the example here is Microsoft.SqlServer, but it could be Microsoft.SqlServer.Management.

I'll look into that and make sure our guidance aligns, if it exists, or write something to that effect.

There's no specific guidance on the format for these that I found with a quick search, but there is documentation on the process for preserving namespace prefixes on nuget.org, which we may want to adopt or adapt if we host a DSC Resource registry.

docs/schemas/resource-manifest/schema.md

SteveL-MSFT · 2023-06-26T23:28:40Z

docs/schemas/resource-manifest/schema.md

+
+The `executable` property, defining the name of the command to call, is mandatory. The `args`
+`input`, and `returns` properties are optional. For more information, see the
+[Test method's schema documentation][03].


I presume details of when they should implement test directly vs relying on dsc.exe synthetic test will be in the test method details?

SteveL-MSFT · 2023-06-26T23:29:33Z

docs/schemas/resource-manifest/schema.md

+
+### validate
+
+This property defines how to call the DSC Resource to validate an instance.


validate is only intended for group resources, although I guess it could be used by single resources but intent is for single resources to provide JSON schema so dsc.exe does the validation.

Should we have a separate schema for group resources? They can reuse shared definitions, but that might be easier for folks to reason about. Otherwise I can adjust the schema. If we only want a single schema for DSC Resources, we might want a distinguishing key for kind, unless resources can be both a group resource and direct resource.

michaeltlombardi · 2023-06-27T13:24:58Z

docs/schemas/resource-manifest/schema.yaml

+    required:
+      - executable
+    properties:
+      executable:
+        $ref: "#/$defs/executable"
+      args:
+        $ref: "#/$defs/args"
+      input:
+        $ref: "#/$defs/input"


@SteveL-MSFT I also noticed when I was looking at the type definition that input seems to be required for set and test, but not get - is that intentional?

DSC/dsc_lib/src/dscresources/resource_manifest.rs

Lines 91 to 93 in adb2d29

/// How to pass optional input for a Get.

#[serde(skip_serializing_if = "Option::is_none")]

pub input: Option<InputKind>,

DSC/dsc_lib/src/dscresources/resource_manifest.rs

Lines 102 to 103 in adb2d29

/// How to pass required input for a Set.

pub input: InputKind,

DSC/dsc_lib/src/dscresources/resource_manifest.rs

Lines 118 to 119 in adb2d29

/// How to pass required input for a Test.

pub input: InputKind,

Relatedly, neither the ValidateMethod nor ListMethod structs seem to have an input field.

SteveL-MSFT · 2023-06-27T15:29:32Z

docs/schemas/resource-manifest/schema.md

+  JSON schema.
+- `embedded` - When you specify the `embedded` property, DSC uses the defined value as the JSON
+  schema.
+- `url` - When you specify the `url` property, DSC fetches the JSON schema from the defined URL.


Either command or embedded is required during runtime execution while url is intended to be used by tooling for authoring configuration.

I've updated this section to match that information. The schema property is now documented in this file as part of the documentation decomposition:

https://github.com/PowerShell/DSC/pull/90/files#diff-75dd2fb3d50b3244e014d47a21e0b10c785da06be220f36d19fcdab112d268d6

This does raise a question for me: If a command-based DSC Resource needs to publish its schema to a URL for integrating tools, what's the benefit of specifying the command property over embedded/url, both of which can be fetched/used by integrating tools without having the DSC Resource installed locally?

The advantage that springs to mind for me is for development/testing speed, but it otherwise seems like a complication, especially since (without command), the property schema could be condensed to:

schema: title: Instance Schema description: >- Defines how DSC must validate a JSON blob representing an instance of the DSC Resource. Must be a valid JSON schema. To reference a remote schema, use the `$ref` keyword. type: object minProperties: 1 examples: # Functionally schema.embedded - $schema: https://json-schema.org/draft/2020-12/schema $id: https://tailspintoys.com/schemas/dsc/gotstoy title: Golang TSToy Resource type: object required: - scope properties: # snipped for bevity # Functionally schema.url - $ref: https://tailspintoys.com/schemas/dsc/gotstoy

SteveL-MSFT · 2023-06-27T15:31:07Z

docs/schemas/resource-manifest/schema.yaml

@@ -0,0 +1,253 @@
+# yaml-language-server: $schema=https://json-schema.org/draft/2020-12/schema


Did you just convert this to YAML from retrieving the JSON schema from the tool?

No, this initial schema is hand-authored.

michaeltlombardi · 2023-06-27T22:36:47Z

docs/schemas/resource/manifest.md

+```yaml
+Type:     string
+Required: true
+Pattern:  ^\w+(\.\w+){0,2}\/\w+$


I made the validation for the type strict - here it only allows:

<owner>/<name>

<owner>.<group|area>/<name>

<owner>.<group>.<area>/<name>

I can remove or increase the restriction on the group/area component count.

michaeltlombardi · 2023-06-27T22:37:36Z

docs/schemas/resource/manifest.md

+```yaml
+Type:     string
+Required: true
+Valid Values:
+  - '1.0'
+```


I didn't include the Pattern metadata here because the semver regex is very long, but maybe we should?

michaeltlombardi · 2023-06-27T22:39:35Z

docs/schemas/resource/manifest.yaml

+    properties:
+      executable:
+        $ref: "#/$defs/executable"
+      args:
+        $ref: "#/$defs/args"


In the documentation, I noted that DSC Group Resources always get their input for the validate method over stdin because the input field wasn't defined on the struct - not sure if that's true or if the field was missing unintentionally.

michaeltlombardi · 2023-06-27T22:41:43Z

docs/schemas/resource/manifest.yaml

+      config:
+        title: Expected Configuration
+        description: >-
+          Defines whether the provider expects to receive a full and unprocessed
+          configuration as a single JSON blob over stdin or a sequence of JSON
+          Lines for each child resource's configurations.
+        type: string
+        enum:
+          - full
+          - sequence


I'm not sure I correctly understood the difference between full and sequence, or the semantic importance of "full and unprocessed configuration" - when this value is full, does the DSC Resource Provider get the entire configuration, or the configuration for all of its child instances, or something else?

When this value is sequence, does the DSC Resource Provider get one resource at a time as JSON Lines, or is it invoked once per resource, or something else?

michaeltlombardi · 2023-06-27T22:43:55Z

docs/schemas/resource/methods/set.md

+The `preTest` property defines whether the DSC Resource tests the instance internally before enforcing the desired state. Set this property to `true` when the DSC Resource tests the instance. Set this property to `false` to ensure DSC determines tests the instance instead. The default value
+is `false`.


This is my understanding of how DSC uses preTest, but I'm not sure, particularly when preTest is false - does DSC call test first, then set if test indicates it should?

michaeltlombardi · 2023-06-27T22:46:00Z

docs/schemas/resource/methods/set.md

+- `stateAndDiff` - Indicates that the DSC Resource returns the instance's final state and an array
+  of property names that the DSC Resource modified.


Is this required to follow a JSON Schema, or is it two separate blobs?

In other words, this:

{ "foo": 5, "bar": true } ["bar"]

or this (with different key names):

{"state": {"foo": 5, "bar": true}, "properties": ["bar"]}

michaeltlombardi · 2023-06-27T22:46:37Z

docs/schemas/resource/methods/test.md

+- `stateAndDiff` - Indicates that the DSC Resource returns the instance's actual state and an array
+  of property names that are out of the desired state.


Is this required to follow a JSON Schema, or is it two separate blobs?

In other words, this:

{ "foo": 5, "bar": true } ["bar"]

or this (with different key names):

{"state": {"foo": 5, "bar": true}, "properties": ["bar"]}

michaeltlombardi · 2023-06-27T22:47:19Z

docs/schemas/resource/methods/validate.md

+DSC Group Resources must define the `validate` property in their DSC Resource manifest. This
+property defines how DSC can call the DSC Group Resource to test whether instances in the group
+have valid definitions.


I more-or-less guessed at the purpose of this keyword and how it's actually used. Is this wildly inaccurate?

michaeltlombardi · 2023-06-27T22:49:32Z

docs/schemas/resource/manifest.yaml

+      list:
+        title: List Command
+        description: >-
+          Defines how DSC must call the DSC Resource Provider to list its supported
+          DSC Resources.
+        type: object
+        required:
+          - executable
+        properties:
+          executable:
+            $ref: "#/$defs/executable"
+          args:
+            $ref: "#/$defs/args"


As with validate, the struct didn't define the input field.

This commit converts the type definitions in the `dsc_lib\src\dscresources\resource_manifest.rs` file into a minimal JSON schema. The schema represents the current state of the type definitions, but raises a few questions for discussion. The documentation for the schema is currently hand-authored and the structure is experimental, since we have no official support for documenting JSON schemas on the Learn platform. For readability and referability, I recommend breaking the larger schema up into smaller schemas and referencing across them.

michaeltlombardi · 2023-07-21T15:10:10Z

Closing in favor of #94

SteveL-MSFT reviewed Jun 26, 2023

View reviewed changes

docs/schemas/resource-manifest/schema.md Outdated Show resolved Hide resolved

SteveL-MSFT reviewed Jun 26, 2023

View reviewed changes

docs/schemas/resource-manifest/schema.md Outdated Show resolved Hide resolved

SteveL-MSFT reviewed Jun 26, 2023

View reviewed changes

docs/schemas/resource-manifest/schema.md Outdated Show resolved Hide resolved

SteveL-MSFT reviewed Jun 26, 2023

View reviewed changes

docs/schemas/resource-manifest/schema.md Outdated Show resolved Hide resolved

SteveL-MSFT reviewed Jun 26, 2023

View reviewed changes

michaeltlombardi commented Jun 27, 2023

View reviewed changes

SteveL-MSFT reviewed Jun 27, 2023

View reviewed changes

michaeltlombardi commented Jun 27, 2023

View reviewed changes

michaeltlombardi closed this Jul 21, 2023

michaeltlombardi deleted the docs/schemas branch August 3, 2023 18:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(DOCS) Define and document schema for resource manifest #90

(DOCS) Define and document schema for resource manifest #90

michaeltlombardi commented Jun 26, 2023

SteveL-MSFT Jun 26, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 26, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 26, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 26, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 26, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 27, 2023

michaeltlombardi Jun 27, 2023

SteveL-MSFT Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi Jun 27, 2023

michaeltlombardi commented Jul 21, 2023

	### type

	The value of this property must be the name of the DSC Resource in its namespace. This value must
	use the following syntax:

	```text
	`<owner>[.<group>][.<area>]/<name>`
	```

	Each component must be string of alphanumeric characters and underscores. No other characters are
	permitted. Every DSC Resource must define an `owner` and a `name`. Use the `group` and `area`
	components to organize DSC Resources into related namespaces. For example:

	- `Microsoft.SqlServer/Database`
	- `Microsoft.SqlServer.Database/Role`
	- `Microsoft.SqlServer.Database/User`
	- `Microsoft.SqlServer/Endpoint`
	- `Microsoft.SqlServer.Endpoint/Permission`
	- `Microsoft.SqlServer/Login`
	- `Microsoft.SqlServer/MaxDop`

	```yaml
	Type: string
	Required: true
	Pattern: ^\w+(\.\w+){0,2}\/\w+$
	```


		### validate

		This property defines how to call the DSC Resource to validate an instance.

	/// How to pass optional input for a Get.
	#[serde(skip_serializing_if = "Option::is_none")]
	pub input: Option<InputKind>,

	/// How to pass required input for a Set.
	pub input: InputKind,

	/// How to pass required input for a Test.
	pub input: InputKind,

		@@ -0,0 +1,253 @@
		# yaml-language-server: $schema=https://json-schema.org/draft/2020-12/schema

		The `preTest` property defines whether the DSC Resource tests the instance internally before enforcing the desired state. Set this property to `true` when the DSC Resource tests the instance. Set this property to `false` to ensure DSC determines tests the instance instead. The default value
		is `false`.

		- `stateAndDiff` - Indicates that the DSC Resource returns the instance's final state and an array
		of property names that the DSC Resource modified.

		- `stateAndDiff` - Indicates that the DSC Resource returns the instance's actual state and an array
		of property names that are out of the desired state.

(DOCS) Define and document schema for resource manifest #90

(DOCS) Define and document schema for resource manifest #90

Conversation

michaeltlombardi commented Jun 26, 2023

PR Summary

PR Context

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaeltlombardi commented Jul 21, 2023