[native_assets_cli] Access to dart_api.h and dart_api_dl.h + passing in the api version

[dcharkes]

When compiling native code from source, it would be nice if we'd compile against dart_api.h or dart_api_dl.h from the Dart/Flutter SDK that is invoking build.dart. This would prevent version skew between the native API and the Dart VM.

It would not prevent version skew with precompiled binaries, as they already have been compiled.

To prevent version skew with precompiled binaries we'd need to pass dart_api_version.h's version as an argument as well, so that the users can precompile a version per Dart API. This would be analogous to what python is doing. (@brianquinlan You pointed to examples of this earlier, but I can't seem to find where you did that.) Note that at this point you'd need an older Dart SDK to precompile a dylib to an older API version.

[brianquinlan]

To prevent version skew with precompiled binaries we'd need to pass dart_api_version.h's version as an argument as well, so that the users can precompile a version per Dart API. This would be analogous to what python is doing. (@brianquinlan You pointed to examples of this earlier, but I can't seem to find where you did that.) Note that at this point you'd need an older Dart SDK to precompile a dylib to an older API version.

I think that I showed you the downloadable files associated with a Python package, e.g. for tensorflow.

You can see that the names are like "tensorflow-2.15.0-cp311-cp311-macosx_12_0_arm64.whl"

So this is the file that will be downloaded if you install tensorflow 2.15 for Python 3.11.x on a ARM64 mac. 0.0.x version changes never break ABI compatibility but all other version changes are assumed to break ABI compatibility.

There is a limited API that is guaranteed to be ABI compatible for all of Python 3.x.x (3.0 was released 15 years ago and there are still no plans for a 4.0).

I'm not advocating for this, just explaining how it works ;-)

I would advocate for allowing some sort of binary packaging so that users don't need a compiler to use packages containing native code.

[dcharkes]

I would advocate for allowing some sort of binary packaging so that users don't need a compiler to use packages containing native code.

Yes, that is totally possible with the native assets feature. (And passing the dart_api_version would keep the precompiled capabilities on par with the on-demand-compiled capabilities when adding the path to dart_api_dl.h and friends.)

[dcharkes]

Preventing version skew between dart_api.h and generated FFIgen bindings could theoretically be solved by providing access to version in Dart, and then branching between two sets of generated bindings. (This is a similar issue to #1589.)

To align the API with access to flutter.h:

config
  .code
    .dart
      .cApi
        .headerPath
        .includeDir
        .version

This doesn't entirely cover it properly though. We have two APIs in reality:

1. dart_api.h is only available on Dart standalone, and not in Flutter. And it's not covered by version.h. not intended for dll writers.
2. dart_api_dl.h is available in both Dart and Flutter, and it is covered by version.h.

And API 1 doesn't really work on Windows when not also building Dart without a workaround: https://dart-review.googlesource.com/c/sdk/+/400582 not intended for dll writers.

[mkustermann]

Preventing version skew between dart_api.h ...

This is an embedder api intended for C code embedding the Dart VM. It's not intended to be used from native code called by Dart apps. We also reserve the right to change this API at-will. Why would we want to expose this via native assets?

The whole reason for dart_api_dl.h to exist is to provide a stable C API that packages can use to talk to the VM. If there's missing features, we can add it to that API.

[dcharkes]

Preventing version skew between dart_api.h ...

This is an embedder api intended for C code embedding the Dart VM. It's not intended to be used from native code called by Dart apps. We also reserve the right to change this API at-will. Why would we want to expose this via native assets?

👍

Maybe we should consider leaning into that and making two separate include directories, so that we don't give users the wrong idea by giving them a dart_api.h in the include dir.

[dcharkes]

From a discussion on #1937 and #93 (comment), we could solve this more nicely with a package if we can let the SDKs specify pinned versions for packages.

[dcharkes]

@jonasfj Mentioned we have Dart SDK packages. I'll explore doing it as an SDK package:

• The include/ dir.
• a hook/build.dart that compiles it into a dylib.
• generated bindings with package:ffigen.

This mandates that other packages use the dynamic linker to link against it. (Which we should have support for now: #190)

[jonasfj]

As @mkustermann mentions above, the point of dart_api_dl.h is to provide a stable C API.
So it probably doesn't matter if package:dart_api_dl is older than the current Dart SDK.

SDK-packages are mostly interesting when you want a package to ship with the SDK.

So if using an older version of package:dart_api_dl is fine, then you don't need to ship it in an SDK-package. Just make sure that package:dart_api_dl has the correct lower-bound SDK constraint when you publish it to pub.dev.

The only problem is if newer version of the Dart SDK stops supporting an older package:dart_api_dl version. But if that should happen, then:

• (A) The user will fix it with dart pub upgrade because upgrading package:dart_api_dl should fix the issue.
• (B) If we had package_incompatibilities.yaml (Native library support for Pub pub#39) then such breakages could be encoded in the SDK and made a bit more graceful.