[native_assets_cli][native_assets_builder] Supporting version skew between SDK and build.dart

[dcharkes]

With #26 we added a version number to to the BuildConfig and BuildOutput.
The way this works is that the SDK passes the version of the protocol that has rolled in to the Dart (and Flutter) SDK. If a user's build.dart cannot deal with that, the build is supposed to fail.
Similarly, the build.dart passes its value of the protocol in the build output. The Dart (and Flutter) SDK check the version and error if the output cannot be used.

@jonasfj suggested that we could use the Dart SDK version instead of the a version number in the protocol and use the SDK lower bound on the package being build to determine what version of the BuildConfig to provide to a package. The main benefit of this approach would be that if we need to do a breaking change to the build configuration, we don't end up in a situation where all dependencies with native assets need to be migrated before developers can update their Dart / Flutter SDK.

If the resolved package:native_assets_cli dependency in the package uniquely determines the protocol version (e.g. all protocol version bump also are a package version bump) we can also support the same behavior by looking at version of that package a package requires.

(With both approaches, the implementation in package:native_asset_cli needs to have a version number in the build config construction so it can keep constructing older versions.)

[jonasfj]

In short, I would suggest that you version the protocol based on the default languageVersion from package_config.json.

So specify input/output like:

• Inputs
  • out_dir (since Dart 3.1): Path where files referenced in assets must be placed.
  • package_root (since Dart 3.1): Path to root package. This is the user package/project the native assets are built for.
  • preferred_link_mode (since Dart 3.2): Either static or dynamic...
• Outputs
  • assets (since Dart 3.1): List of objects on the form:
    • name (since Dart 3.1): Name of the native asset when referenced from @Native attribute, must start with package:<my_package>/path/to/file.dart
    • link_mode (since Dart 3.1): Either static or dynamic...
  • supporting_asset (since Dart 3.3): Path to file that is included as a blob...

If my_package has has native assets and I declare:

name: my_package
environment:
  sdk: ^3.1.0   # This will make the default languageVersion in package_config `3.1`

Then my build script would :

• never be supplied the preferred_link_mode field, and,
• forbidden from outputting the supporting_asset field.

Thus, if my_package wants access to the preferred_link_mode field, I have to bump the SDK constraint to at-least ^3.2.0.
There is no risk that my build script assumes preferred_link_mode is always there, and then when built with an older Dart SDK it crashes.

Similarly, if my_package wants to output the supporting_asset field, then I must bump the SDK constraint to at-least ^3.3.0.
There is no risk my_package outputs the supporting_asset field and relies on the Dart SDK copying this, but when used with an older Dart SDK my package can't find the "supporting asset" and fails in weird manners.

---

It's a bit of work, but a table with inputs and output, field name, description and minimum language version would go far.

[dcharkes]

After more discussions with @jonasfj, relying on the language versions of a package doesn't work.

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

If we would like to support version skew, we should do protocol negotiation. build.dart (and link.dart) should support a mode in which they only return a protocol version. (And the native assets builder should be able to speak every version of the protocol indefinitely.)

If we would like to completely avoid version skew, we could consider not exposing a JSON protocol, but only BuildConfig and BuildOutput in a dart: library. Importing a dart: library means by construction a package works with the version of the Dart / Flutter that is running. (We could break the protocol in arbitrary ways, but we could never break the public API.)

A poor mans version of a dart: library would be to use a helper package, but pin the SDK constraint on a single dart version sdk: '>=3.3.0 <3.4.0' and release a new version of that package with every Dart release. This seems rather brittle.

[dcharkes]

Update on the approach:

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.
2. All new additions to BuildConfig have default values in Dart if they are missing in the JSON. So newer build hooks are not broken on older SDKs.
3. The BuildOutput will be serialized based on the version number from BuildConfig, so older SDKs are not broken by newer build hooks.
4. The BuildOutput can be parsed based on older JSON formats, so older build hooks are not broken with newer SDKs.

This means maintaining an implementation for older versions of BuildConfig and BuildOutput, which we do and cover with tests.

A changelog for both can be found in:

• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_config_CHANGELOG.md
• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_output_CHANGELOG.md

If we ever want to support breaking "1.", we should implement a handshake. E.g. hook/build.dart --version should return only back the version and then the SDK should talk with that version of BuildConfig.

[dcharkes]

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.

We run into a similar issue with versioning resources.json. And it would be nice if we could use the same solution in multiple place.

Ways to handle this more gracefully:

1. Protocol negotiation. Do a process call to the build and link hook to ask them what version they speak. The link hook should then also return a version for resources.json format.
   Downside: More process invocations.
   Upside: True and tried solution.
2. Write multiple .json files and pass them all in with a map from format version to file URI.
   Downside: The parse API of a package starts taking a Map<Version, Uri> instead of Uri. If the file path comes from a command-line invocation (e.g. --config=path/to/config.json) then that needs to be a multi argument.
3. Write a JSON that has a map with { "v1" : ..., "v2" : ... }.
   Downside: huge file.
   Upside: version skew between an older reader reading a newer file is completely hidden in the API of the package. The package itself deals with version skew both ways. (Newer readers reading older formats already works due to the parsers always being able to parse older versions.)

For all three options, we still want to aim to not do major version bumps, so these are all in case we really need to in the future.
For all three options, if we ever have to do a major version bump, this enables the eco system to migrate. We would then at some point do a breaking change announcement to remove an older major version.
For all three options, we will break the current build and link hook implementations.

From an encapsulation POV, I believe option 3 to be the cleanest. The package itself deals with version skew internally, it doesn't leak out into the API, and the callers don't have to know about it.

@mosuem Did I miss any pros/cons for the different options? You were advocating for option 2, some pros I missed there?

[jonasfj]

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

Actually, this could probably be made to work. But it might not be easy.

Suppose we have a world with:

• myapp, an application that uses foo.
• foo, a package with native assets that uses the helper bar package to create a build script.
• bar, a helper package that provides utilities for creating a build script.

When the build system has to build myapp, then the build system will:

• Find that foo has native assets that must be built
• Lookup the default language version of foo in package_config.json.
  Suppose this is 3.4.
• Communicate with build script from foo using native build protocol version 3.4.

For this to work, then bar (the helper package), will (once invoked) need to, either:

• (A) Take 3.4 as a parameter indicating the protocol version.
• (B) Read package_config.json to find the default language version of foo,
  which indicates the protocol version to be used.
  (This is probably the easiest thing to do)

Obviously not all versions of bar can support all versions of the native build protocol. Which notably could cause issues like:

• Author of foo bumps their default language version from 3.4 to 3.5, and if 3.5 protocol is not supported by bar then this will break the build.
  • Though this will probably be discovered in testing before a new version offoo is published.
  • This is probably solved by waiting for a new version of bar to be released.
• If a new version of bar is released that supports newer protocol versions, and foo bumps default language version, but forgets to bump the constraint on bar, then a consumer of foo might get an old version of bar which can't build the latest version of foo.
  • While sad, not bumping lower-bound constraints on a dependency is not a new issue.
    (This happens all the time, indeed pana has plans to do downgrade testing).
  • This is unlikely to happen, if most package authors use a caret-constraint bar: ^1.2.3.
  • Maybe this could be mitigated with lints or downgrade testing in pana.

---

Maybe, I'm missing something, and maybe it's too complicated.

But it kind of feel like writing the native build protocol version in the pubspec.yaml is a natural solution. Even if it means that a helper package like bar will have to support said version of the native build protocol.

And the best way to write the native protocol version in the pubspec.yaml is to use the default language version (which is derived from the SDK constraint, environment.sdk).

Or maybe the native build protocol version should be a completely different field in pubspec.yaml, and use a different version number than the SDK / language version.

[jonasfj]

Talking to Daco, I can certainly see how (3) is just much simpler all around 😄

[dcharkes]

Some notes from discussion with @jonasfj:

Exploring tying protocol version to Dart SDK version

• Tying the protocol version the Dart SDK (dev) version would require adding an entry to a map mapping SDK dev versions to protocol versions when rolling in dart-lang/native into the Dart SDK. One would have to guess the dev version based on what the next roll is going to be and hope that you guess the right one.
• Any reverts to the protocol version would create a non-monotonic mapping.
• One thing that we could do if we go for { "v1" : ... , "v2" : ... } is at some point stop adding v1 if we see all packages in the package graph have a language version of which the SDK always at least speaks v2.
  • Your main app is going to have a language version, but we can't use that for the hooks.
  • The native_assets_cli has a language version but the resolution of that package is likely going to be one of the newest ones (assuming we didn't do any breaking API changes), so we can't use that for the hooks.
  • Then we could try to use the package that contains the hook languages' version. But that doesn't work either because it could be simply only using package:native_assets_cli and package:native_toolchain_c. And if both of those packages support some new feature in the protocol, your build/link hook should automatically update to use that feature after a pub get.
  • But using the language versions of all packages in the whole dependency graph works.
  • But it's probably simpler to just do a breaking change announcement to intend to remove v1 a year or two after we've introduced v2 instead of all the magic.

[dcharkes]

Ways to handle this more gracefully:

4. Postfix existing keys in maps with _v2 when wanting to make a breaking change. This would prevent duplication inside the JSON for all other parts of JSON that are not going v2. Old parsers would just ignore the _v2s, just like they would have ignored a toplevel v2 in option 3. (Thanks @mkustermann for the input!)

For both option 3 and 4, the version: in the format would basically never be bumped to 2.x. Technically speaking, when removing at some point a xxx when a xxx_v2 has been there for a while, would be the actual breaking change. But, we don't want version comparing to fail. Instead the logic should be the same as dart: libs, a breaking change announcement, and then at some point just the removal. Not every breaking change to dart: libs triggers a major version bump to Dart. (Side note major version bumps to Dart are not actually indicative of breaking changes either, one can have two Dart packages that target a different major version e.g. 2.12 and 3.4 still work together!)

Then the question is should these formats have a version at all? Shouldn't they simply be versioned with the SDK? I believe it's always safe to have a version. The question is more what kind of logic is tied to that version. Should users ever have to worry about it? Does it show up in error messages etc?

I think our goal could be for Dart & Flutter stable releases that these versions are never really visible for users.

On dart dev releases and Flutter master channel they might show up while things are rolling. (Version skew between dart and flutter in a Flutter SDK for the build and link config and output.)

(From an implementation point of view, writing the parser and serializer from a certain version is easier than just assuming everything is nullable. It simplifies writing unit tests for checking that version skew works. Of course this is not really a concern for users.)

[dcharkes]

I believe we have explored enough alternatives and will stick with the current approach:

• Version in build/link config and output.
• The hook will output an older version if the config is older.
• The SDK will never break it's input (hooks can have an older version).
• Any breaking changes will be made non-breaking by using _2 keys.

[dcharkes]

The SDK will never break it's input (hooks can have an older version).

We started breaking older versions of config.json input under the following conditions:

1. A stable release of the SDKs (both Dart and Flutter) talk a new version of the protocol.
2. The last stable release of native_assets_cli talks that version of the protocol and has its SDK constraint updated to the last stable release of the SDK. (This guarantees that that code never runs with an older SDK. The SDK constraint means we can safely remove the support for old versions in package:native_assets_cli as used in hooks.)

It's unsafe to remove support for older versions in the config.json in the SDK, but we do so anyway, breaking any packages that have not updated their dependency on package:native_assets_cli to the latest version.

Note that this way of working could potentially break other embedders, because they might not update the protocol in time while still updating Dart.

[dcharkes]

Let's continue the discussion in the existing bug.

At the core of it the CLI build protocol is the ability of a package to tell how app bundling tools (e.g. Flutter SDK, Dart SDK, ...) about various assets it needs at runtime.

Once we have a thriving ecosystem of packages making use of this ability there's a very high bar on keeping the protocol between the app bundling tools and the package's hook stable. Though there may come a time when we have to change the protocol. If we do so, we have to minimize any negative impact this can have on the ecosystem.

The first mechanism is to decouple the protocol into core protocol and protocol extensions (e.g. data assets, code assets). This means a breaking change to one protocol extension limits the possible negative affects to only the packages that use that protocol extension and not others.

How would we approach breaking the core protocol or a protocol extension? We should avoid situations where publishing new app bundling tools (Dart/Flutter/... SDKs) breaks users in a way that's very non-obvious (e.g. updating SDK leads to some dart run / dart build failing and errors are burried deep in the --verbose log). Ideally users would be told: You updated Dart SDK, which comes with breaking change, but package Y wasn't prepared for this breaking change.

Currently the CLI protocol between bundling tool and hook communicates a version number. Though this version number is problematic due to several factors: It will make hook runs fail and errors will be buried deep in logs, the version doesn't cover protocol extensions, ...

I propose us to use the existing mechanism we have for packages for breaking changes: Use package's semantic versioning system. That will make users get nice error messages if packages & SDKs are not compatible. It also allows us to soften the blow of any breaking changes in a way that may be invisible for end users. The idea being roughly as follows:

• Every protocol piece (core protocol, protocol extensions) get their own dart package

• Those packages will get semantic versions for the APIs they surface

• Those packages will get SDK constraints (possibly also Flutter SDK constraints - if flutter specific protocol) as follows

  • We publish them with narrow upper bound constraints: sdk: min < X < max such that

    • min is the minimum SDK where the supported features of the protocol work
    • max is the current SDK versions plus 1
      => We promise that the next SDK will continue to support this version of the protocol package

  • Most common case: no breaking changes, but new SDK release: We'll publish new protocol packages with increased upper bound

  • If we do a breaking change

    • We always have a transition period
    • We release a SDK with the new thing, with protocol packages with a higher minimum SDK constraint
    • At this point packages can start adopting the newer API, but the new Dart SDK still supports the old protocol package-version
    • Support packages (e.g. package:native_toolchain_c, package:data_assets, ...) may publish minor version updates adopting the new protocol (requiring the new protocol version package)
    • We release another SDK at which point the old protocol package-versions published will no longer work (as their upper bound is lower than newer SDK release)

For end users that use

• older SDKs, pub will automatically select older support packages which use older protocol versions
• newer SDKs, pub will automatically select newer support packages with newer protocol versions
  => Overall end users may not even know that breaking changes have appeared, by pub selecting the right packages for the SDK in use.

The same mechanism can also be used to version our dart_api_dl.h: It can be published as package:dart_api_dl, use narrow upper bounds with promised transition period for any breaking changes.

(In some sense this system would allow versioning sub-parts of our SDK and softening the blow to any breaking changes by making pub auto-select SDK-wrapping packages that work with the currently used SDK)

#1941

[dcharkes]

The first mechanism is to decouple the protocol into core protocol and protocol extensions

👍 Yes we should!

Use package's semantic versioning system

These already have a meaning, the public Dart API. We'll still need to also satisfy that meaning.

Also we could do minior/patch version bumps to change the SDK constraint. So we don't need have the semantic version actually mean the protocol version. We only need to bump the SDK constraint afaics.

If we want to use a package semantic version to specify the protocol version we should have it separately from the Dart API version. (e.g. package:hook and package_hook_protocol.) But we don't actually want to pin the protocol version, because we want to support a range. Pub resolution needs to pick a single version, but we want to support a range. (Though if we make pub pick a specific version, we could possibly make the native_assets_builder speak exactly that version and support the version skew inside the package instead of inside the json format?! The native_assets_builder can see the package_config.json and see the ersion of package:hook_protocol.)

Those packages will get SDK constraints

The lower bound will need to double check that both Dart and Flutter (with some version of Dart) support that protocol. If there's a 3rd embedder that embeds Dart, but lacks behind in adopting a newer version of the protocol, we can't meaningfully use the Dart lower bound anymore. Because our Dart and Dart in Flutter might support a version of the protocol but that new embedder might not.

This smells like it's in the wrong place. The base protocol version depends on the embedder/launcher. The launcher should declare it. Not a package trying to approximate it via a a Dart version constraint.

If the SDK should declare it, it should be more like Flutter's pinned packages. But again, we don't want to pin the protocol version, that's not graceful. We want to support a range.

Basically, you want the SDK to have it's own dependencies.

# Dart / Flutter / other embedders:
dependencies:
  hooks_protocol: >=1.5.0 <= 1.9.0
  code_assets_protocol: ^1.0.0
  data_assets_protocol: ^1.0.0

# package:hook
version: 4.3.2

dependencies:
  hooks_protocol: >=1.5.0 <= 1.9.0

# package:code_assets
version: 1.0.0

dependencies:
  code_assets_protocol: ^1.0.0

# package with a hook
dependencies:
  hooks: ^4.3.2
  code_assets: ^1.0.0

This would cover the constraints more properly. And basically pub get has to take into account the implicit dependencies: of the SDK when solving and give a proper error message.

This approach means:

• For end users:
  • The right protocol versions are selected automatically by pub based on the SDK dependencies.
  • Get proper error messages on pub get instead of hook failures.
  • No issues with different embedders with the same Dart version speaking different ranges of the protocol
• For us:
  • Not having a process for publishing packages every new Dart/Flutter release
• For us and other embedders:
  • The embedders roll their dependencies: when they add support or drop support for the base protocol and protocol

The only issue that I see is that package_config.json produced by flutter is consumed by dart. So we can't have the Dart and Flutter supported protocol versions diverge. -> Actually, the fix for that is to take the dependencies: of both Dart and Flutter into account in the pub get for Flutter.

[dcharkes]

cc @jonasfj We've got more schemes going on trying to use pub to this versioning. 😄

[mkustermann]

The only benefit I see with your suggestion over the current approach with version numbers in the protocol is the error message surfacing.

Firstly my suggestion is to rely on existing version constraint solving instead of custom version numbers in json or C header files, etc. That's what I care about most and it does seem there's agreement around that, which is great!

The second part is how to piggy back on version constraint solving exactly. The benefit of the suggestion I made above is that we don't have to change pub's version constraint mechanism. But I'm entirely open to other ways to piggy back on our semantic constraint solving mechanism -- iff it can be done soon - as we don't want to artificially delay CLI. @jonasfj Would the pub team be willing to change version solving logic to have a magic list of Dart SDK dependencies on packages? If so, could that be done within ~ 3-4 months?

I don't believe that benefit is worth it to make the whole process toilsome (having to check what versions rolled into what, putting things in your calendar for around release dates) and fragile (forgetting to release a new version) due to specifying things in the wrong place.

Remembering what rolled into what is needed in any case. We should have always transition periods if we do breaking changes, so it's necessary to think when a new api rolled out, when the old one got deprecated and when we can remove the old thing. The publishing of the packages with higher sdk bounds would be automatic - as part of Dart SDK release process. So I'm not quite sure I see the problem with this. The only manual intervention would be to increase minimum sdk versions when new things get introduced vs updating numbers in a hard coded list - which seem somewhat similar in effort.

[dcharkes]

Basically @jonasfj already proposed in dart-lang/pub#3962 what I prosed in #93 (comment). 🔥 ❤
The only superficial difference is that my proposal is basically a compatibilities.yaml instead of incompatibilities.yaml, but that doesn't matter for the functionality.

Notes from discussion with @jonasfj:

• Having extra packages x_protocol makes things more complex, but doesn't give us anything. We use the semantic version to do both the Dart API and the protocol changes, we should be fine.
  • When adding something to the implementation in Dart/Flutter SDK we can change the version_compatibilities.yaml to include the stable version of the WIP we're rolling in
  • When removing support for an older version in the Dart/Flutter SDK, we change the lower bound in version_compatibilities.yaml to the last version of the package that's supported.
  • We'll need to keep a protocol-changelist per extension to make it easier to find in which package semantic versions what protocol extension versions are supported. (E.g. the package might rev version numbers much more often than that the protocol changes.)
  • @jonasfj suggested to still keep the version numbers in the JSON as well, because users can use dependency_overrides.

Notes from discussion with @jonasfj and @mkustermann:

• package:dart_c_api can be published as an SDK package right now. That would hard-tie it to the SDK.
  • Currently SDK packages are not very discoverable.
    • This can be mitigated by having an example
• With package_compatibilities.yaml, we'd be able to use a range (down to current major version).
  • Note: I'm not sure if that gives us anything. The newer versions of the include/ dir should always compile as well. And the newer versions of FFI bindings should also be backwards compatible.

If so, could that be done within ~ 3-4 months?

This would make me a happy man! 😄 🙏

Thanks @mkustermann @jonasfj! 🚀

[dcharkes]

Notes from a discussion with @jonasfj and @sigurdm.

• The logic in [native_assets_builder] Fail early on too old native_assets_cli #1923, is identical to having a packge_incompatibilities.yaml.
  • However, we're then not relying on existing version constraint solving.
  • This logic needs to be extended to be per package once we split the helper package up in package:hook, package:code_asset and package:data_asset. (The native assets builder will need to take constraints from the embedder.)
• If we want to support people not using package:native_assets_cli, but have their own package, then using package versions to specify incompatibility doesn't make any sense. Instead, there should be a version number in the JSON (per extension).
  • Addressing [native_assets_cli] Document the protocol #95 only makes sense if we do not use package:hook, package:code_assets etc as version numbers.
  • We should consider saying the Dart API is the thing that's versioned, and no-one should have their own JSON decoder/encoder.
• For making the Dart API the thing that's versioned (Instead of the JSON), there is only one option
  • (doesn't work) Using SDK packages. This doesn't work because of embedders supporting different asset types. The Dart standalone SDK asset types are not necessarily available in all other SDKs. And non-Dart asset types might want to be shared across SDKs.
  • (doesn't work) dart: libraries. Functionally equivalent to SDK package.
  • pub published packages: Can be shared among SDKs, enabling hooks with custom asset types to work in multiple SDKs supporting that asset types. (current approach)

[sigurdm]

For making the Dart API the thing that's versioned (Instead of the JSON), there are is only one option

• (doesn't work) Using SDK packages. This doesn't work because of embedders supporting different asset types. The Dart standalone SDK asset types are not necessarily available in all other SDKs. And non-Dart asset types might want to be shared across SDKs.

The basic asset mechanism could still be in the dart sdk (which is included in the flutter sdk) and the consumption of asset types is specific to each sdk and should be located there. The only thing that IMO should live on pub.dev, is helper packages for constructing these assets.

[dcharkes]

The basic asset mechanism could still be in the dart sdk (which is included in the flutter sdk) and the consumption of asset types is specific to each sdk and should be located there. The only thing that IMO should live on pub.dev, is helper packages for constructing these assets.

That could work 👍

We should give some thought to if other SDKs can introduce hooks that are not in the Dart standalone SDK. Because in that case package:hook should possibly be named package:build_hook (package:build_and_link_hook seems like the wrong name). And then some other SDK could introduce a package:post_pub_get_hook that the Dart SDK does not support for example. Such hook would have a JSON protocol that's not shared and not versioned with the protocol for build and link hooks. (And then we should also open up the pub checks to allow other hook names in the hook directory, which we currently disallow.)

If we expect to author the list of permissible hooks, and to have all of them in Dart (not have Flutter-only hooks), then we can indeed do it as an SDK package. (I'd feel somewhat uncomfortable committing to that, as I don't have a crystal ball.)

[sigurdm]

Some general thoughts about the idea of a package_incompatibilities.yaml file somewhere in the sdk:

• I think having the sdk needing to know about package versions of packages on pub.dev is a smell and should be avoided if at all possible. It is layering the architecture the wrong way.
• It does not allow the pub client to abstract away the sdk version (ie "solve pretending you are sdk v. 2.3") (sdk packages have this same drawback though).
• It cannot be used to fix incompatibilities "after the fact"

If we feel we really need something like this, I'd rather like some kind of generalization of the "retraction" feature, where package owners could specify incompatibilities after they are found. (I agree this feature would be more expensive to build - but it is also more generally useful).

[dcharkes]

Notes from another round of discussion with @mkustermann @jonasfj and @sigurdm:

Alternative currently pursuing

• No constraints on packages, the only thing that matters is the JSON schema (and its semantics).
• No version on the JSON. Instead, deprecate fields (and/or semantics) and remove after "a year".
  • This needs to go hand-in-hand with lazy-reading of the JSON.
  • Compared to versioning the JSON, this enables (a) removing deprecated fields out of order, and (b) not breaking someone even after "a year" when breaking schema or semantics if that part wasn't referenced.
• While something is an experiment, only ever run against dart-dev/main and flutter-master.
  • We should be able to clean things up after ~1 week (Dart's dev release cycle is 2x per week, and the changes need to have been rolled into both Flutter tools and Dart.)
  • When wanting to drop support for older SDKs in experimental, the lower SDK bound needs to be bumped to a dev release. (A dev release that also implies the Flutter version with that Dart version has the feature.)
    • We need to check if that is able to roll into flutter tools
    • We need to see how that works with having a pub-workspace with other packages on this repo working with Dart stable.
• Testing
  • Older hooks with the newest SDK can be tested by having example projects pinned to older versions of the helper packages (we already have this).
  • The newest hook with older SDKs can be tested by pinning an older version of the Dart/Flutter SDK on the GitHub CI in this repo and running the examples against it. (Only relevant when we are stable, not while we're an experiment.)

Downsides of this approach:

• Manually keeping track of what "version" of the protocol is in Dart and Flutter, no easy version number or date to inspect.

TODO:

• Rip out all the versioning checks.
• Migrate the changelog to a schema definition (with deprecations)
• Stop running anything against Dart stable
• Disable experiments on Dart stable https://dart-review.googlesource.com/c/sdk/+/406660
• Verify workflow with flutter tools

Alternatives considered

• Using JSON with version numbers is too strict.

  • Using a version number to version the JSON is too strict, it doesn't enable removing a field that no-one used early.
  • Using deprecating fields, and then after "a year" just removing them, and not incrementing a version is the same breaking change policy as we use for dart: APIs. And we'd like to align those processes.

• Using package constraints per SDK is problematic.

  • Adding a packages_incompatibilities.yaml per SDK means that the package resolution becomes SDK dependent. This means that the code doesn't run with a single resolution, which becomes highly inefficient and highly confusing. (The current situation with the Flutter SDK and the Dart standalone SDK is that most of the Dart tools simply work on the package_config.json produced by Flutter.)
  • We want to actually support having packages for different SDKs in a single pub workspace.
  • So, if the Flutter SDK supports different versions of the protocols than the Unity SDK, Jasper SDK, or Dart Angular SDK, the SDK running pub get would be unaware of the package_incompatibilities.yaml of the other SDKs, which leads to having to create a package_config.json per SDK. (Which means the code in your IDE being analyzed by the dart analyze is not what is actually running in flutter or jasper command.)

[dcharkes]

Small update from my explorations of using a JSON schema (#95) and dealing with version skew: We'll need separate schemas for hook writers and SDK authors.

Consider the following scenario: We start with Input.outputDirectory, then we introduce Input.outputDirectoryV2 and mark the first one as deprecated, and after 2 years we remove Input.outputDirectory.

Initial situation:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Input Schema (Writer and Reader - Start)",
  "type": "object",
  "properties": {
    "output_dir": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory" 
    }
  },
  "required": ["output_dir"],
  "additionalProperties": true
}

Transition period for hook writers, assume that one of the two output directories is available.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Input Schema (Reader - Transition) -> Compatible with both the Start and End schema for writers",
  "type": "object",
  "properties": {
    "output_dir": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory (deprecated: use output_dir_v2 instead)",
      "deprecated": true
    },
    "output_dir_v2": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory v2" 
    }
  },
  "oneOf": [
    { "required": ["output_dir"] },
    { "required": ["output_dir_v2"] }
  ],
  "additionalProperties": true
}

Transition period for SDK authors, provide both output directories.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Input Schema (Writer - Transition) -> Compatible with both the Start and End schema for readers",
  "type": "object",
  "properties": {
    "output_dir": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory (deprecated: use output_dir_v2 instead)",
      "deprecated": true
    },
    "output_dir_v2": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory v2" 
    }
  },
  "required": ["output_dir", "output_dir_v2"],
  "additionalProperties": true
}

Final breaking change, after 2 years.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Input Schema (Writer and Reader - End)",
  "type": "object",
  "properties": {
    "output_dir_v2": { 
      "type": "string", 
      "format": "uri",
      "description": "Output directory" 
    }
  },
  "required": ["output_dir_v2"],
  "additionalProperties": true
}

The scenario is identical for if the hook output changes in a breaking way.

@jonasfj You've been saying it's madness that we use the same serialization/deserialization for the hook-helper-package and the SDK-implementation. Now that I've been trying to express the contract in a schema, this becomes obvious. The implementation was written in a way that it dealt with the version skew inside the Dart code. However, we can't smooth over that difference when explicitly stating the contract in a schema.

For our documentation for hook writers, Dart+Flutter SDK consumers, we're mostly concerned about the schemas for the input-reader and output-writer.

However, since we want to enable other SDKs to implement the hook protocol as well, we should be explicit about the contract that these implementations must uphold as well. (And our own implementation in Dart and Flutter must uphold it.) This is the schemas for the input-writer and output-reader.

[mkustermann]

@jonasfj You've been saying it's madness that we use the same serialization/deserialization for the hook-helper-package and the SDK-implementation. Now that I've been trying to express the contract in a schema, this becomes obvious.

Luckily we have refactored the code and separated the reading and writing part of the json for this exact reason: The SDKs use the builders to create json and the hook authors use the readers to read it (and vice versa for the output).

[dcharkes]

Notes a discussion with @mkustermann:

• @mkustermann would like to use hard SDK lower bounds to ensure features are supported in produced input.jsons and consumed output.jsons
  • This would be achieved by the following workflow. We separate package:native_assets_cli and package:native_assets_builder completely. We can roll package:native_assets_builder into dartdev and flutter_tools so that the Dart and Flutter SDK support these new features. Then we can bump the SDK lower bound for package:native_assets_builder to the Dart dev release which ensures the feature is both in Dart and Flutter.
  • Workflow requirements: Being able to run tests in the Flutter code base against an unpublished package:native_assets_cli. The integration tests rely on having access to a package:native_assets_cli and a package:native_toolchain_c.
  • Workflow benefit: New versions of package:native_assets_cli can immediately drop any fallback behavior as the Dart and Flutter SDKs are guaranteed to have support for the new feature.
  • Workflow benefit: No different JSON schemas for hooks and SDKs.
  • Downside: This forces SDK authors (other SDKs than Dart and Flutter) to implement new features on the Dart roll. (Edit: @mosuem pointed out that some SDKs might not have a Dart roll as they are simply a Dart package. These SDKs would simply break.)
• @dcharkes would like to introduce new features as optional
  • This would enable publishing a package:native_assets_cli that would opt in to the new functionality if it is provided by an SDK.
  • Downside: package:native_assets_cli cannot clean up fallbacks eagerly.
  • Downside: Maintaining different JSON schemas for hooks and SDKs.
  • Workflow requirements: none.
  • Benefit: Other SDKs don't break.

Notes from a discussion with @mosuem:

• We might have SDKs built on top of Dart that do not explicitly roll Dart. They might just be a Dart package with bin/ directory.
  • This means these SDKs have no explicit rolls of Dart. So they cannot add support for new features as required field on the roll, they would do this afterwards, having a broken state of the world until they do.
  • This is not a current issue, but will if web SDKs start picking up Data and Wasm assets.
• @mosuem's suggestion: Have a mailing list with SDK authors that you can send a "breaking change announcement" to. This is a different list than hook authors/Dart users. (Adding a new required field to the input.json is non-breaking for hooks, but breaking for SDKs.)
  • Add the fallback logic in package:native_assets_cli and publish a new version with the old SDK constraint.
  • Send out a ping on the SDK-breaking-change list.
  • Once all SDKs support it, remove the fallback logic and bump the sdk constraint in package:native_assets_cli and publish a new version.

[mkustermann]

Downside: This forces SDK authors (other SDKs than Dart and Flutter) to implement new features on the Dart roll. (Edit: @mosuem pointed out that some SDKs might not have a Dart roll as they are simply a Dart package. These SDKs would simply break.)

It's important to disentangle this a bit. If one changes the protocol to

• add optional things and the package can work with and without - then there's no need for a package to increase the lower bound sdk constraint
• add guaranteed things and the package needs it in order to function, then a package should to increase the lower bound

Yes, the second point means that the increased lower bound SDK constraint may still not guarantee that the package works in a 3rd party SDK (a rare case in the first place) - but it's strictly better than the alternative: Users on older sdks, getting package resolutions to versions that won't work and getting weird build or runtime errors - instead of pub solving for an older version of the package that just works.

Imagine a newer version of package:crypto requires universal binaries support in the protocol which we introduce in version X. If it

• doesn't have a lower SDK bound on X, then users will get newer package:crypto (even if they don't update Flutter/Dart SDK) and it will just not work at build or runtime - a really bad state of affairs for users!
• does have a lower SDK bound on X, then users will get working versions of package:crypto - depending on their SDK

So if we introduce some new support and packages rely on it (cannot function without) they should have a corresponding lower bound SDK constraint. It's not perfect, but better than not having this constraint. Yes 3rd party SDKs may have trouble with the selected versions - but for the 99% case of Dart&Flutter SDK users it will be highly beneficial to have that lower bound SDK constraint and in the future we may eventually get support from Pub to do this in a better/cleaner way (**)

Now I can see that @dcharkes main point may be that a package wants to work in older and newer SDKs but optionally take advantage of features of newer SDKs. So it's hook would want to check if new features are available and use them and otherwise use some kind of fallback mechanism. Phrased in another way: The hook requires protocol X but if X+1 is available it may want to take advantage of the X+1 features. The SDK still only has one(!) version of the protocol, but a hook may want to speak protocol X or X+1. So it's not a question of protocol - it's a question of how a hook author can write code that works with different versions of the protocol. My thinking around this is

• The Dart & Flutter projects are evergreen (we do not maintain old versions - anyone wanting bugfixes / features / perf improvements has to upgrade to newest main/dev/beta/stable).
  => This strongly incentivzes our ecosystem to continiously upgrade SDKs (there's probably data for this)
• Similarly packages usually also don't maintain old versions (bug fixes, features, perf improvements aren't back ported to old versions and released as old version patch releases)
• It makes writing hooks complicated (to support multiple protocol versions)
• The package would still have a fallback path anyway. So the worst case is that it only uses the fallback and waits a little longer until it starts taking advantage of new protocol features (that have been released some time ago) - which isn't such a bad thing.

tl;dr I wouldn't optimize (or make things much more complicated) for this particular use case of a hook author wanting to talk multiple protocol versions.

(**) As discussed many times - this could be solved by having a separate package representing the protocol and SDKs (be flutter/dart sdks or package-based sdks) constraining versions of that protocol package - but it seems it was agreed to rule this out for now)

[dcharkes]

So if we introduce some new support and packages rely on it (cannot function without) they should have a corresponding lower bound SDK constraint.

Agreed, for these the lower SDK constraint should be bumped in such package. This would be the SDK constraint inside the package with the hook that requires this feature.

package:native_assets_cli would have an SDK constraint which is the lowest one it can parse and guarantees (in Dart and Flutter) with its API.

What I'm proposing is to not eagerly bump package:native_asstes_cli's SDK constraint. I understood your preference as eagerly bumping package:native_asstes_cli SDK constraint, thereby preventing users from wanting to write "hook would want to check if new features are available and use them and otherwise use some kind of fallback mechanism" to use the latest version of package:native_assets_cli and keeping their deps evergreen.

It makes writing hooks complicated (to support multiple protocol versions) [...] or make things much more complicated

I don't believe this to be the case. We haven't introduced features that don't have an almost trivial fallback for months. And almost all of these will be taken care of in the hook helper packages such as package:native_toolchain_c.

We'd adopt the new features in package:native_toolchain_c and other helper packages and hook writers don't even know they're benefitting of some more advantageous behavior.

[dcharkes]

Updates:

• The version number checking logic was removed from the packages

package:native_assets_cli would have an SDK constraint which is the lowest one it can parse and guarantees (in Dart and Flutter) with its API.

We've started having an Dart dev version lower bound on package:native_assets_cli and successfully rolled that into Dart and Flutter.

• package:native_assets_cli and package:native_toolchain_c are now pre-releases (0.123.456-789) due to the Dart SDK dev version constraint (enforced by pub).
  • Pub does not automatically resolve to these versions in flutter pub upgrade --major-versions. This will likely lead to users using the experiment not upgrading.
• package:native_assets_builder is a pre-release, this seems to be fine when rolling into flutter_tools.

[dcharkes]

Updates:

• As per [firehose] don't fail publish validation if we see the pub pre-release warning ecosystem#357, we no longer need to use pre-releases.

New status quo:

• No more version number in the protocol.
• All input/output is accessed lazily by the means of the new json_syntax_generator. So any breaking changes not accessed will not break hooks or SDKs.
• package:native_assets_cli can bump the lower bound SDK constraint to drop support for older SDKs. (The lower bound for the SDK is the dart dev release that rolled into Flutter for the flutter roll that added support. We assume that we virtually always roll into Dart first. We don't support other SDKs besides Flutter/Dart.)
• Dropping support for older hooks in SDKs is achieved by saying it is an experiment. If you update your SDK, you should also update your package:native_assets_cli. We give a ~2 weeks grace period before removing features that are experimental.
  • If hooks are stable, such changes need to go through the breaking change process.

Things considered and not adopted:

• Completely removing fallbacks and only publishing native_assets_cli when it's already supported in Dart and Flutter.
  • Not having to publish native_assets_cli before rolling into flutter_tools by using git dependencies in flutter_tools. That requires too many Flutter infra changes.
  • Not having to publish native_assets_cli before rolling into flutter_tools by copying all code into flutter_tools. This is too ugly.

Now that this is the new status quo, I'm going to close this issue (again).