[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s #49803

dcharkes · 2022-08-24T12:58:06Z

Given the following Dart code.

@FfiNative<Int64 Function(Int64, Int64)>('sum', asset: 'mylib/mylib')
external int sum(int a, int b);

Update 2022-10-19: We're taking a different approach. Below is no longer applicable.

~~The embedder can register a resolver function to look up symbols in assets.~~

/**
 * FFI native asset C function pointer resolver callback.
 *
 * \param asset The name of the asset.
 * \param symbol The name of the symbol.
 * \param error Returns NULL if lookup is successful, an error message
 *   otherwise. The caller is responsible for calling free() on the error
 *   message.
 *
 * \return Looks up the symbol in the asset and returns its address in memory.
 *         Returns NULL if the asset or symbol doesn't exist.
 */
typedef const void* (*Dart_NativeAssetResolver)(const char* asset,
                                                const char* symbol,
                                                char** error);

/**
 * Sets the callback used to resolve FFI native functions in assets for an
 * isolate group.
 *
 * The resolved functions are expected to be a C function pointer of the
 * correct signature (as specified in the `@FfiNative<NFT>()` function
 * annotation in Dart code).
 *
 * NOTE: This is an experimental feature and might change in the future.
 *
 * \param isolate_group An isolate group.
 * \param resolver A native function resolver.
 */
DART_EXPORT void Dart_SetNativeAssetResolver(
    Dart_IsolateGroup isolate_group,
    Dart_NativeAssetLookup resolver);

~~The Dart standalone embedder can receive a mapping from asset to path through a flag. (As follow up, this mapping will be provided automatically by packages which want to bundle native assets.)~~

Internal design doc

The text was updated successfully, but these errors were encountered:

dcharkes · 2022-10-19T09:38:52Z

Backend Implementation

@mraleph suggested to embed the resolution in kernel files and the Dart VM rather than let embedders do the resolution. This way we don't have to reimplement resolution in each embedder. Each embedder needs to provide the mapping as input to the kernel compiler instead. (This makes the first post obsolete.)

internal design doc

Dart API

@mraleph suggested to let assetId's default to library name.

If we combine that with our wish to make @Ffinatives more concise (#50097), and to do it in a non-breaking way, the Dart API would be as follows:

/// Annotation to be used for marking an external function as FFI native.
///
/// Example:
///
/// ```dart template:top
/// @FfiNative<Int64 Function(Int64, Int64)>('FfiNative_Sum', isLeaf:true)
/// external int sum(int a, int b);
/// ```
///
/// Calling such functions will throw an exception if no resolver
/// was set on the library or the resolver failed to resolve the name.
///
/// See `Dart_SetFfiNativeResolver` in `dart_api.h`
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.14')
@Deprecated('Use Native instead.')
class FfiNative<T> {
  final String nativeName;
  final bool isLeaf;
  const FfiNative(this.nativeName, {this.isLeaf: false});
}

/// Annotation to be used for marking an external function as native.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in
/// 1. the provided [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// Calling such function will throw an exception if the symbol fails to
/// resolve in all cases.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved.
  ///
  /// If not specified, defaults to the annotated function name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are identical.
  ///
  /// Prefer omitting [symbol].
  final String? symbol;

  /// The [asset] in which [symbol] is resolved.
  ///
  /// If not specified, defaults to [Asset] annotation on the `library`.
  /// If [Asset] on library is omitted, defaults to library name.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are identical.
  ///
  /// Prefer using the library name as `Asset` over specifying it.
  /// Prefer specifying [Asset] on `library` over specifying on [Native].
  final String? asset;

  /// Native function does not call back into Dart code or the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// Annotation to be used for specifying the default asset on a library.
///
/// If no annotation is provided, defaults to library name.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/b.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are identical.
///
/// Prefer using the library name as `Asset` over specifying it.
@Since('2.19')
class Asset {
  final String asset;

  const Asset(
    this.asset,
  );
}

cc @lrhn @mraleph @mkustermann @mit-mit

Some questions to answer:

Do we want the optional asset on Native? If not, we force people to write an extra file if they want to resolve in multiple assets.
Do things need different names for annotations or fields?

mkustermann · 2022-10-19T09:50:20Z

Do we want the optional asset on Native? If not, we force people to write an extra file if they want to resolve in multiple assets.

It's probably beneficial to allow one library to use natives from multiple native libraries.

@lrhn Do you think the name "asset" is too generic here?

lrhn · 2022-10-19T10:41:16Z

I have absolutely no idea what "asset" means. And that's after reading the documentation. So yes, too generic.

I think an asset is a string. But it can also be a library name (there are no examples of that, and I wouldn't recommend people using library names for anything).
It can be used to resolve a symbol in. Which seems to be a native thing.

I really need the ELI5 version of what's going on 😉.

Either describe what "asset" means in the documentation, or pick a name that describes it more precisely (if one exists).

dcharkes · 2022-10-19T10:53:20Z

Later, these assets will be coming from the bin/native.dart CLI and be included by Dart Dev (dart run / dart compile).

En example of a dart_app that depends on two packages which provide native assets:

dart_app$ dart run native list
Building package executable... (1.2s)
Built native:native.

- name: mylib_dylib # Can be a library name including `package:`.
  packaging: dynamic
  path:
    path_type: absolute
    uri: /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/mylib_dylib/native/lib/linux_x64/libmylib_dylib.so
  target: linux_x64
- name: mylib_source # Can be a library name including `package:`.
  packaging: dynamic
  path:
    path_type: absolute
    uri: /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so
  target: linux_x64

When doing

/// @Native<Int64 Function(Int64, Int64)>(asset: `mylib_source`)
/// external int sum(int a, int b)

calling sum will result in looking up sum in /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so in dart run.

I have no idea how to refer to that in the dart:ffi Natives documentation if we don't land the CLI and dartdev support in the same CL. Unfortunately, the idea was to land the backend logic separately. Dartdev would convert the CLI output to .dart_tools/native_assets.dart:

@pragma('vm:entry-point')
@pragma('vm:native-assets', {
  "linux_x64": {
    "mylib_dylib": [
      "absolute",
      "/usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/mylib_dylib/native/lib/linux_x64/libmylib_dylib.so"
    ],
    "mylib_source": [
      "absolute",
      "/usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so"
    ]
  }
})
class _NativeAssets {}

but this is an implementation detail that users should never see.

Any suggestions for what to put in the implementation for when dartdev and CLI support have not been landed?

dcharkes · 2022-10-19T10:54:45Z

I mean, we could add documentation on passing in a library named .dart_tools/native_assets.dart containing the above pragma. But do we want to expose users to that low level VM API?

lrhn · 2022-10-19T11:59:04Z

So, an asset is a named abstraction over a native library (which is itself a can of worms to define in a consistent cross-platform way). You can refer to the native library by its asset name.

The compiler and/or runtime provides a binding from asset name to native library, which can (and probably does) depend on the target platform. The compiler/runtme can then resolve/lookup identifiers against the native library, and use it to bind external Dart function declarations to native functions as their implementation.

The name can be any string.
Defaults include the current library's file name without the trailing .dart?

"# Can be a library name including package:." - That's not a "library name". A library name is what comes after library in a library declaration, like library foo.bar.baz;. Do you mean the package URI, the package URI's path or the library's file name (minus extension) only? (Make sure to define it even if the library file name does not end in just .dart, or in .dart at all.)

dcharkes · 2022-10-19T19:59:40Z

So, an asset is a named abstraction over a native library (which is itself a can of worms to define in a consistent cross-platform way). You can refer to the native library by its asset name.

The compiler and/or runtime provides a binding from asset name to native library, which can (and probably does) depend on the target platform. The compiler/runtme can then resolve/lookup identifiers against the native library, and use it to bind external Dart function declarations to native functions as their implementation.

Correct. You are good with words! 😄

However, we also support looking up symbols in the process or executable. In that case asset maps not really to a native library. So one can refer to a native library, or process, or executable by asset name.

The binding part is correct verbatim.

The name can be any string. Defaults include the current library's file name without the trailing .dart?

My bad, it should be "library uri", which includes package: and .dart (if the file name has that extension).

"# Can be a library name including package:." - That's not a "library name". A library name is what comes after library in a library declaration, like library foo.bar.baz;. Do you mean the package URI, the package URI's path or the library's file name (minus extension) only? (Make sure to define it even if the library file name does not end in just .dart, or in .dart at all.)

It should be URI indeed. Though that raises the question, what is the uri of a file outside lib/?

What is the difference between a package uri and library uri? Or does "library uri" not exist?

Take two:

/// A [Native] external function is implemented in native code.
///
/// The native external function is resolved in the native library bound to
/// [asset].
///
/// The compiler and/or runtime provides a binding from [asset] name to native
/// library, which depends on the target platform. The compiler/runtme can then
/// resolve/lookup identifiers against the native library, and use it to bind
/// external Dart function declarations to native functions as their implementation.
///
/// Use this class as annotation on `external` functions, to make them resolve
/// to native functions.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in
/// 1. the provided [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// Calling such function will throw an exception if the symbol fails to
/// resolve in all cases.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved.
  ///
  /// If not specified, defaults to the annotated function name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are identical.
  ///
  /// Prefer omitting [symbol].
  final String? symbol;

  /// The [asset] in which [symbol] is resolved.
  ///
  /// If not specified, defaults to [Asset] annotation on the `library`.
  /// If [Asset] on library is omitted, defaults to package uri.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are identical.
  ///
  /// Prefer using the package uri as `Asset` over specifying it.
  /// Prefer specifying [Asset] on `library` over specifying on [Native].
  final String? asset;

  /// Native function does not call back into Dart code or the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// An [Asset] is a named abstraction over a native library.
///
/// The compiler and/or runtime provides a binding from asset name to native
/// library, which depends on the target platform. The compiler/runtme can then
/// resolve/lookup identifiers against the native library, and use it to bind
/// external Dart function declarations to native functions as their implementation.
///
/// Use this class as annotation on a `library` to make all [Native]s in this
/// library use the specified [asset] (unless overwrittten with [Native.asset]).
///
/// If no annotation is provided, defaults to package uri.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/b.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are identical.
///
/// Prefer using the package uri as `Asset` over specifying it.
@Since('2.19')
class Asset {
  final String asset;

  const Asset(
    this.asset,
  );
}

lrhn · 2022-10-20T11:01:27Z

It should be URI indeed. Though that raises the question, what is the uri of a file outside lib/?

It's a file:-schemed URI then.

What is the difference between a package uri and library uri? Or does "library uri" not exist?

A "package URI" is a shorthand sometimes used for "URI whose scheme is package".
A "library URI" is (typically, in a Dart context) the URI used to denote a Dart library.

A library URI is usually, but not always, a package URI. The counter examples are the libraries in bin/ or /test, which do not have package URIs denoting them, and are therefore always denoted by file: URIs (or maybe http: URIs, if we serve the tests over HTTP for some reason). Those are rarely referred to except by nearby files, using relative paths.

My attempt:

/// Annotation specifying how to bind an external function to native code.
///
/// The annotation applies only to `external` function declarations.
///
/// A [Native]-annotated `external` function is implemented by native code.
/// The implementation is found in the native library denoted by an asset name.
///
/// The compiler and/or runtime provides a binding from asset name to native
/// library, which depends on the target platform.
/// The compiler/runtime can then resolve/lookup symbols (identifiers)
/// against the native library, to find a native function,
/// and bind an `external` Dart function declaration to that native function.
///
/// Use this annotation on `external` functions to specify that they
/// are resolved against an asset, and to, optionally, provide overrides
/// of the default symbol and asset names.
///
/// The type argument to the [Native] annotation must be a function type
/// representing the native function's parameter and return types.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in (in that order)
/// 1. the provided or default [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// At least one of those three *must* provide a binding for the symbol,
/// otherwise the method call fails.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved, if not using the default.
  ///
  /// If not specified, the default symbol used for native function lookup
  /// is the annotated function's name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are equivalent.
  ///
  /// Prefer omitting the [symbol] when possible.
  final String? symbol;

  /// The asset name in which [symbol] is resolved, if not using the default.
  ///
  /// If no asset name is specified,
  /// the default is to use an asset name specified using
  /// an [Asset] annotation on the current library's `library` declaration,
  /// and if there is no [Asset] annotation on the current library,
  /// the library's URI, as a string is used instead.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are all equivalent.
  ///
  /// Prefer using the library URI as an asset name over specifying it.
  /// Prefer using an [Asset] on the `library` declaration
  /// over specifying the asset name in a [Native] annotation.
  final String? asset;

  /// Whether the function is a leaf function.
  ///
  /// A leaf function must not run Dart code or call back into the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// Annotation specifying the default asset name for the current library.
///
/// The annotation applies only to `library` declarations.
///
/// The compiler and/or runtime provides a binding from _asset name_ to native
/// library, which depends on the target platform and architecture.
/// The compiler/runtime can resolve identifiers (symbols)
/// against the native library, looking up native function implementations
/// which are then used as the implementation
/// of `external` Dart function declarations
///
/// If used as annotation on a `library` declaration, all [Native]-annotated
/// external functions in this library will use the specified [asset] name
/// for native function resolution (unless overridden by [Native.asset]).
///
/// If no [Asset] annotation is provided, the current library's URI
/// is the default asset name for [Native]-annotated external functions.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/a.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are equivalent.
///
/// Prefer using the library URI as asset name when possible.
@Since('2.19')
class Asset {
  /// The default asset name for [Native] external functions in this library.
  final String asset;

  const Asset(
    this.asset,
  );
}

(@dcharkes: edited typo and punctuation)

dcharkes · 2022-10-20T12:03:00Z

Thanks @lrhn!

Should we have "(file package:a/b.dart)" instead of "(file package:a/a.dart)" in the examples where the asset is specified in the annotations as 'package:a/a.dart'?

lrhn · 2022-10-20T12:25:14Z

If you say that the two examples are equivalent, then they should be the same file. Otherwise they are not equivalent.

The bindings might be, but then that's what it should say. (Pedantry at work!)

dcharkes · 2022-10-20T12:35:05Z

Right, let's keep it this way.

dcharkes · 2022-10-21T14:27:30Z

Do we discourage annotations on libraries? @lrhn @mraleph

sdk/lib/ffi/ffi.dart:966: 1 errors
  error: The library directive must appear before all other directives. [4:7] (library_directive_not_first)

  > @Asset('package:a/a.dart')
  > library a;
  >
  > @Native<Int64 Function(Int64, Int64)>()
  > external int sum(int a, int b);

Adding // ignore_for_file: library_directive_not_first in the snippet works, but feels wrong.

Source: CI on https://dart-review.googlesource.com/c/sdk/+/265084.
(Also unnamed libraries are an experiment at the moment, so the library needs to have a name.)

Bug: #49803 Bug: #50097 Change-Id: Id5b52be88937bcf9245f98e71afa56f079f288f0 Cq-Include-Trybots: luci.dart.try:analyzer-linux-release-try Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/265085 Reviewed-by: Brian Wilkerson <[email protected]> Commit-Queue: Daco Harkes <[email protected]>

mraleph · 2022-10-24T11:15:23Z

Do we discourage annotations on libraries? @lrhn @mraleph

I think this looks like a bug in the script which checks code in the comments. We definetely use annotations on libraries in other contexts (e.g. our package:test has @TestOn annotation).

Maybe we should contact the maintainer to see what exactly goes wrong? (I am not entirely sure who maintains that).

Dartdoc code snippets should be able to have annotations before the `library` keyword. Bug: #49803 (comment) Change-Id: I7ec0b6db296274a1173d10d900f7af38a1f2b552 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/265321 Reviewed-by: Devon Carew <[email protected]> Auto-Submit: Daco Harkes <[email protected]> Commit-Queue: Daco Harkes <[email protected]> Commit-Queue: Devon Carew <[email protected]>

This reverts commit 97eaded. Reason for revert: It was not erroring on the annotation on the library, but on the implicit import statement above it due to template:top. #50291 (comment) Original change's description: > [tools] `verify_docs` don't error on library annotations > > Dartdoc code snippets should be able to have annotations before the > `library` keyword. > > Bug: #49803 (comment) > > Change-Id: I7ec0b6db296274a1173d10d900f7af38a1f2b552 > Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/265321 > Reviewed-by: Devon Carew <[email protected]> > Auto-Submit: Daco Harkes <[email protected]> > Commit-Queue: Daco Harkes <[email protected]> > Commit-Queue: Devon Carew <[email protected]> [email protected],[email protected],[email protected],[email protected] Change-Id: I379a1871a1acd4d1c66804c3948f38e45691fd55 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: #49803 (comment) Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/265324 Commit-Queue: Daco Harkes <[email protected]> Bot-Commit: Rubber Stamper <[email protected]> Reviewed-by: Alexander Thomas <[email protected]>

@pragma

Validates a yaml format encoding a native asset mapping, and synthesizes a component containing a pragma with this information. Yaml example: ``` format-version: [1,0,0] native-assets: linux_x64: 'package:foo/foo.dart': ['absolute', '/path/to/libfoo.so'] ``` Generated format example: ``` @pragma('vm:ffi:native-assets': { 'linux_x64' : { 'package:foo/foo.dart': ['absolute', '/path/to/libfoo.so'] } }) library; ``` TEST=pkg/vm/test/native_assets/synthesizer_test.dart TEST=pkg/vm/test/native_assets/validator_test.dart In a follow-up CL, we will consume the yaml from `gen_kernel` and consume the pragma in the VM for `@FfiNative`s. Bug: #49803 Change-Id: Ie8d93b38ff4406ef7485e5513807e89b2772164b Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/272660 Reviewed-by: Martin Kustermann <[email protected]> Commit-Queue: Daco Harkes <[email protected]>

@pragma

Adds support for `asset: 'asset-id'` in `FfiNative`s. Symbols in assets are resolved by 1. the provided [asset] (the bulk of this CL), 2. the native resolver set with `Dart_SetFfiNativeResolver` in `dart_api.h` (old behavior), and 3. the current process. Symbols in assets are resolved by the Dart VM through the mapping provided in the `vm:ffi:native_assets` library: ``` @pragma('vm:ffi:native-assets', { "linux_x64": { "<asset1>": [ "absolute", "<absolute-path>" ], "<asset2>": [ "relative", "<relative-path>" ], "<asset3>": [ "system", "<system-path>" ], "<asset4>": [ "process", ], "<asset5>": [ "executable", ], } }) library; ``` There are 5 asset path types. Symbols resolved in these asset types are resolved as follows. A. Absolute paths are dlopened, after which the symbol is looked up. B. Relative paths are resolved relative to `Platform.script.uri`, and dlopened, after which the symbol is looked up. C. System paths are treated as absolute paths. (But we might explicitly try to use `PATH` in the future.) D. Symbols looked up in Process assets are looked up in the current process. E. Symbols looked up in Executable assets are looked up in the current executable. The native assets mapping can be provided in three ways. X. In the invocation to `gen_kernel` with the `--native-assets` argument. This uses the gen_kernel entry point for compiling kernel. Y. By placing a `native_assets.yaml` next to the `package_config.json` used by the isolate group. This works for `dart <source file>` and `Isolate.spawnUri(<dart source file>)`. This uses the kernel_service entry point for compiling kernel. Z. By sending a String containing the native assets mapping when invoking the kernel_service directly from within the VM. Currently, only used for unit tests. TEST=tests/ffi/native_assets/asset_*_test.dart (X) TEST=tests/ffi/native_assets/infer_native_assets_yaml_*.dart (Y) TEST=runtime/vm/ffi/native_assets_test.cc (Z) The library is synthesized from yaml in pkg:vm as AST. Design doc: go/dart-native-assets Bug: #49803 Change-Id: I8bf7000bfcc03b362948efd3baf168838948e6b9 Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-mac-debug-arm64-try,vm-kernel-nnbd-mac-debug-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-win-debug-x64c-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-precomp-win-debug-x64c-try,vm-kernel-precomp-nnbd-mac-release-arm64-try Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/264842 Reviewed-by: Martin Kustermann <[email protected]>

See API design discussion in bugs below. TEST=tests/ffi/native_assets/* Design doc: http://go/dart-native-assets Bug: #49803 Bug: #50097 Change-Id: Id6e6eb94c6eb39ccaaa637448583a40ab6110d12 Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-precomp-win-debug-x64c-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,dart2wasm-linux-x64-d8-try Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/265084 Reviewed-by: Lasse Nielsen <[email protected]> Commit-Queue: Daco Harkes <[email protected]>

blaugold · 2023-01-29T14:34:46Z

Is the symbol for a native function going to be resolved lazily before the first call? I have a specific use case in mind where there are multiple variants of a native library and some symbols are not available in all variants. I know at runtime which variant is loaded and can guard against calling native functions that aren't available, but will I be able to use @Native with these functions?

dcharkes · 2023-01-30T08:08:16Z

Is the symbol for a native function going to be resolved lazily before the first call? I have a specific use case in mind where there are multiple variants of a native library and some symbols are not available in all variants. I know at runtime which variant is loaded and can guard against calling native functions that aren't available, but will I be able to use @Native with these functions?

@blaugold we haven't decided between dynamic linking and dynamic loading. You're asking for dynamic loading. Dynamic linking on the other hand will give an earlier error to devs when developing. Maybe we should support both. As of right now (WIP), it is dynamic loading.

dcharkes · 2023-02-15T11:09:44Z

The VM can do native assets resolution (when the kernel file is compiled with passing --native-assets <path>.

How the native assets mapping is produced by packages and consumed by Dart and Flutter tooling is in progress and tracked in:

☂️ [vm/ffi] Native assets feature #50565

This more closely resembles --release compilation in Flutter on Linux and Android. TEST=tests/ffi/native_assets/asset_absolute_test.dart Bug: #49803 Change-Id: I51ea100966393cd25b11a44dc2c40ce43552ae87 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/283184 Commit-Queue: Daco Harkes <[email protected]> Reviewed-by: Tess Strickland <[email protected]>

The VM can read native asset mappings from kernel. Previous CLs already added support to embed native asset mappings for one-shot compilation. This CL adds support for adding native assets mappings to kernel files created by the frontend_server with the incremental compiler. The frontend_server accepts a `--native-assets=<uri>` at startup and accepts a `native-assets <uri>` message on stdin as compilation command. The frontend_server caches the compiled native assets library. When a `reset` command is sent to request a full dill from the incremental compiler, the native assets mapping is taken from the cache and added to the final dill file. Split of DartSDK & flutter_tools prototype to land separately. TEST=pkg/frontend_server/test/native_assets_test.dart Bug: #49803 Bug: #50565 Change-Id: I6e15f177564b8a962e81261815e951e7c9525513 Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-try Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/282101 Reviewed-by: Jens Johansen <[email protected]> Commit-Queue: Daco Harkes <[email protected]>

eseidel · 2023-05-26T15:16:44Z

I found this today trying to understand https://api.dart.dev/stable/3.0.2/dart-ffi/Native/assetId.html. I agree with @lrhn that it's confusing. Particularly that the "assetId" is a dart package path by default. In my case, I'm moving from:
https://github.com/dart-lang/samples/blob/main/ffi/hello_world/hello.dart and direct lookup:

  final dylib = DynamicLibrary.open('libhello.so');
  final HelloWorld hello = dylib
      .lookup<NativeFunction<HelloWorldFunc>>('hello_world')
      .asFunction();

To trying to understand how I would do the same with @assetId. Would love if the example included something similarly simple/clear. :)

dcharkes · 2023-05-26T15:50:35Z

@eseidel this issue was tracking the implementation in the VM, not yet the implementation in dart itself.

That requires https://dart-review.googlesource.com/c/sdk/+/267340 to land (which will land it behind an experimental flag), and further work for Flutter.
The high level issue to track is: #50565.

Scripts such as https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/example/native_add/build.dart will populate the asset information, but these are ignored until the above mentioned CL lands.

eseidel · 2023-05-26T15:57:41Z

I see, you're saying that @assetId doesn't even work from stand alone dart yet, so there isn't anything to document.

I got to this by trying to move from the failure I was seeing running Flutter in my Simulator + FFI environment:
shorebirdtech/shorebird#532 (comment)

To replicating in dart for easier fixing:
#52530

But if @Native isn't supposed to work in stand-alone dart yet, that's fine, I'll debug other ways.

Dampfwalze · 2024-03-13T11:59:34Z

I tried to use DynamicLibrary.open(String path) using an asset id, but it seems like this is not supported. Will this API be deprecated in the future?

I would like to start using native assets for new projects, but many things still rely on this ffi API, so it would be helpful if this is supported.

I can get it to work using dart build (using --enable-experiment=native-assets), which bundles the asset next to the executable. I am then able to use the actual filename of the asset, completely bypassing the native asset resolution. But this does not work with dart run or dart test.

dcharkes · 2024-03-13T16:58:29Z

Hi @Dampfwalze!

I tried to use DynamicLibrary.open(String path) using an asset id, but it seems like this is not supported. Will this API be deprecated in the future?

Native assets should be accessed with @Native() external functions, not with DynamicLibrary.open.

No, we don't intend on deprecating DynamicLibrary, there are still valid use cases.
However, you need to branch in your Dart code to find the correct path/name of the dylib, which is non-trivial.
The native assets feature lets you abstract over this. (It "just works" in every OS and in both Dart and Flutter.)

I would like to start using native assets for new projects, but many things still rely on this ffi API, so it would be helpful if this is supported.

What relies on this API? Can these things be migrated to use @Native() external functions instead? What is your use case?

I can get it to work using dart build (using --enable-experiment=native-assets), which bundles the asset next to the executable. I am then able to use the actual filename of the asset, completely bypassing the native asset resolution. But this does not work with dart run or dart test.

It doesn't work indeed. It is not a supported use case. Native assets are designed to work with @Native() external functions.

P.S. Feel free to open new GitHub issues instead of resurrecting (related) old issues. (And feel free to tag me on anything native assets related!)

dcharkes added area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. library-ffi labels Aug 24, 2022

dcharkes self-assigned this Aug 24, 2022

dcharkes mentioned this issue Oct 10, 2022

Implement support for creating isolate independent native functions code in dart:ffi #47778

Open

dcharkes mentioned this issue Oct 19, 2022

[vm/ffi] Make FfiNatives more concise #50097

Open

dcharkes changed the title ~~[vm/ffi] Native asset resolution by the embedder for FfiNatives~~ [vm/ffi] Native asset resolution by the Dart VM for FfiNatives Oct 19, 2022

dcharkes mentioned this issue Oct 25, 2022

intent to implement: dart:crypto #50290

Closed

This was referenced Nov 28, 2022

☂️ [vm/ffi] Native assets feature #50565

Open

Remove static fields in FfiNative's #47625

Closed

dcharkes closed this as completed Feb 15, 2023

Dampfwalze mentioned this issue Mar 14, 2024

Resource files with native assets #55195

Open

Dampfwalze mentioned this issue Apr 1, 2024

Support for @Native fzyzcjy/flutter_rust_bridge#1849

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s #49803

[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s #49803

dcharkes commented Aug 24, 2022 •

edited

Loading

dcharkes commented Oct 19, 2022 •

edited by mraleph

Loading

mkustermann commented Oct 19, 2022

lrhn commented Oct 19, 2022

dcharkes commented Oct 19, 2022 •

edited

Loading

dcharkes commented Oct 19, 2022

lrhn commented Oct 19, 2022

dcharkes commented Oct 19, 2022 •

edited

Loading

lrhn commented Oct 20, 2022 •

edited by dcharkes

Loading

dcharkes commented Oct 20, 2022

lrhn commented Oct 20, 2022

dcharkes commented Oct 20, 2022

dcharkes commented Oct 21, 2022

mraleph commented Oct 24, 2022

blaugold commented Jan 29, 2023

dcharkes commented Jan 30, 2023

dcharkes commented Feb 15, 2023

eseidel commented May 26, 2023 •

edited

Loading

dcharkes commented May 26, 2023

eseidel commented May 26, 2023

Dampfwalze commented Mar 13, 2024

dcharkes commented Mar 13, 2024 •

edited

Loading

[vm/ffi] Native asset resolution by the Dart VM for FfiNatives #49803

[vm/ffi] Native asset resolution by the Dart VM for FfiNatives #49803

Comments

dcharkes commented Aug 24, 2022 • edited Loading

dcharkes commented Oct 19, 2022 • edited by mraleph Loading

Backend Implementation

Dart API

mkustermann commented Oct 19, 2022

lrhn commented Oct 19, 2022

dcharkes commented Oct 19, 2022 • edited Loading

dcharkes commented Oct 19, 2022

lrhn commented Oct 19, 2022

dcharkes commented Oct 19, 2022 • edited Loading

lrhn commented Oct 20, 2022 • edited by dcharkes Loading

dcharkes commented Oct 20, 2022

lrhn commented Oct 20, 2022

dcharkes commented Oct 20, 2022

dcharkes commented Oct 21, 2022

mraleph commented Oct 24, 2022

blaugold commented Jan 29, 2023

dcharkes commented Jan 30, 2023

dcharkes commented Feb 15, 2023

eseidel commented May 26, 2023 • edited Loading

dcharkes commented May 26, 2023

eseidel commented May 26, 2023

Dampfwalze commented Mar 13, 2024

dcharkes commented Mar 13, 2024 • edited Loading

[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s #49803

[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s #49803

dcharkes commented Aug 24, 2022 •

edited

Loading

dcharkes commented Oct 19, 2022 •

edited by mraleph

Loading

dcharkes commented Oct 19, 2022 •

edited

Loading

dcharkes commented Oct 19, 2022 •

edited

Loading

lrhn commented Oct 20, 2022 •

edited by dcharkes

Loading

eseidel commented May 26, 2023 •

edited

Loading

dcharkes commented Mar 13, 2024 •

edited

Loading