[native_assets_cli][native_assets_builder] Supporting version skew between SDK and build.dart

[dcharkes]

With #26 we added a version number to to the BuildConfig and BuildOutput.
The way this works is that the SDK passes the version of the protocol that has rolled in to the Dart (and Flutter) SDK. If a user's build.dart cannot deal with that, the build is supposed to fail.
Similarly, the build.dart passes its value of the protocol in the build output. The Dart (and Flutter) SDK check the version and error if the output cannot be used.

@jonasfj suggested that we could use the Dart SDK version instead of the a version number in the protocol and use the SDK lower bound on the package being build to determine what version of the BuildConfig to provide to a package. The main benefit of this approach would be that if we need to do a breaking change to the build configuration, we don't end up in a situation where all dependencies with native assets need to be migrated before developers can update their Dart / Flutter SDK.

If the resolved package:native_assets_cli dependency in the package uniquely determines the protocol version (e.g. all protocol version bump also are a package version bump) we can also support the same behavior by looking at version of that package a package requires.

(With both approaches, the implementation in package:native_asset_cli needs to have a version number in the build config construction so it can keep constructing older versions.)

[jonasfj]

In short, I would suggest that you version the protocol based on the default languageVersion from package_config.json.

So specify input/output like:

• Inputs
  • out_dir (since Dart 3.1): Path where files referenced in assets must be placed.
  • package_root (since Dart 3.1): Path to root package. This is the user package/project the native assets are built for.
  • preferred_link_mode (since Dart 3.2): Either static or dynamic...
• Outputs
  • assets (since Dart 3.1): List of objects on the form:
    • name (since Dart 3.1): Name of the native asset when referenced from @Native attribute, must start with package:<my_package>/path/to/file.dart
    • link_mode (since Dart 3.1): Either static or dynamic...
  • supporting_asset (since Dart 3.3): Path to file that is included as a blob...

If my_package has has native assets and I declare:

name: my_package
environment:
  sdk: ^3.1.0   # This will make the default languageVersion in package_config `3.1`

Then my build script would :

• never be supplied the preferred_link_mode field, and,
• forbidden from outputting the supporting_asset field.

Thus, if my_package wants access to the preferred_link_mode field, I have to bump the SDK constraint to at-least ^3.2.0.
There is no risk that my build script assumes preferred_link_mode is always there, and then when built with an older Dart SDK it crashes.

Similarly, if my_package wants to output the supporting_asset field, then I must bump the SDK constraint to at-least ^3.3.0.
There is no risk my_package outputs the supporting_asset field and relies on the Dart SDK copying this, but when used with an older Dart SDK my package can't find the "supporting asset" and fails in weird manners.

---

It's a bit of work, but a table with inputs and output, field name, description and minimum language version would go far.

[dcharkes]

After more discussions with @jonasfj, relying on the language versions of a package doesn't work.

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

If we would like to support version skew, we should do protocol negotiation. build.dart (and link.dart) should support a mode in which they only return a protocol version. (And the native assets builder should be able to speak every version of the protocol indefinitely.)

If we would like to completely avoid version skew, we could consider not exposing a JSON protocol, but only BuildConfig and BuildOutput in a dart: library. Importing a dart: library means by construction a package works with the version of the Dart / Flutter that is running. (We could break the protocol in arbitrary ways, but we could never break the public API.)

A poor mans version of a dart: library would be to use a helper package, but pin the SDK constraint on a single dart version sdk: '>=3.3.0 <3.4.0' and release a new version of that package with every Dart release. This seems rather brittle.

[dcharkes]

Update on the approach:

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.
2. All new additions to BuildConfig have default values in Dart if they are missing in the JSON. So newer build hooks are not broken on older SDKs.
3. The BuildOutput will be serialized based on the version number from BuildConfig, so older SDKs are not broken by newer build hooks.
4. The BuildOutput can be parsed based on older JSON formats, so older build hooks are not broken with newer SDKs.

This means maintaining an implementation for older versions of BuildConfig and BuildOutput, which we do and cover with tests.

A changelog for both can be found in:

• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_config_CHANGELOG.md
• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_output_CHANGELOG.md

If we ever want to support breaking "1.", we should implement a handshake. E.g. hook/build.dart --version should return only back the version and then the SDK should talk with that version of BuildConfig.

[dcharkes]

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.

We run into a similar issue with versioning resources.json. And it would be nice if we could use the same solution in multiple place.

Ways to handle this more gracefully:

1. Protocol negotiation. Do a process call to the build and link hook to ask them what version they speak. The link hook should then also return a version for resources.json format.
   Downside: More process invocations.
   Upside: True and tried solution.
2. Write multiple .json files and pass them all in with a map from format version to file URI.
   Downside: The parse API of a package starts taking a Map<Version, Uri> instead of Uri. If the file path comes from a command-line invocation (e.g. --config=path/to/config.json) then that needs to be a multi argument.
3. Write a JSON that has a map with { "v1" : ..., "v2" : ... }.
   Downside: huge file.
   Upside: version skew between an older reader reading a newer file is completely hidden in the API of the package. The package itself deals with version skew both ways. (Newer readers reading older formats already works due to the parsers always being able to parse older versions.)

For all three options, we still want to aim to not do major version bumps, so these are all in case we really need to in the future.
For all three options, if we ever have to do a major version bump, this enables the eco system to migrate. We would then at some point do a breaking change announcement to remove an older major version.
For all three options, we will break the current build and link hook implementations.

From an encapsulation POV, I believe option 3 to be the cleanest. The package itself deals with version skew internally, it doesn't leak out into the API, and the callers don't have to know about it.

@mosuem Did I miss any pros/cons for the different options? You were advocating for option 2, some pros I missed there?

[jonasfj]

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

Actually, this could probably be made to work. But it might not be easy.

Suppose we have a world with:

• myapp, an application that uses foo.
• foo, a package with native assets that uses the helper bar package to create a build script.
• bar, a helper package that provides utilities for creating a build script.

When the build system has to build myapp, then the build system will:

• Find that foo has native assets that must be built
• Lookup the default language version of foo in package_config.json.
  Suppose this is 3.4.
• Communicate with build script from foo using native build protocol version 3.4.

For this to work, then bar (the helper package), will (once invoked) need to, either:

• (A) Take 3.4 as a parameter indicating the protocol version.
• (B) Read package_config.json to find the default language version of foo,
  which indicates the protocol version to be used.
  (This is probably the easiest thing to do)

Obviously not all versions of bar can support all versions of the native build protocol. Which notably could cause issues like:

• Author of foo bumps their default language version from 3.4 to 3.5, and if 3.5 protocol is not supported by bar then this will break the build.
  • Though this will probably be discovered in testing before a new version offoo is published.
  • This is probably solved by waiting for a new version of bar to be released.
• If a new version of bar is released that supports newer protocol versions, and foo bumps default language version, but forgets to bump the constraint on bar, then a consumer of foo might get an old version of bar which can't build the latest version of foo.
  • While sad, not bumping lower-bound constraints on a dependency is not a new issue.
    (This happens all the time, indeed pana has plans to do downgrade testing).
  • This is unlikely to happen, if most package authors use a caret-constraint bar: ^1.2.3.
  • Maybe this could be mitigated with lints or downgrade testing in pana.

---

Maybe, I'm missing something, and maybe it's too complicated.

But it kind of feel like writing the native build protocol version in the pubspec.yaml is a natural solution. Even if it means that a helper package like bar will have to support said version of the native build protocol.

And the best way to write the native protocol version in the pubspec.yaml is to use the default language version (which is derived from the SDK constraint, environment.sdk).

Or maybe the native build protocol version should be a completely different field in pubspec.yaml, and use a different version number than the SDK / language version.

[jonasfj]

Talking to Daco, I can certainly see how (3) is just much simpler all around 😄