[native_assets_cli] Code Assets depending on other code assets

[dcharkes]

Dynamic libraries can try to load other dynamic libraries when they are dlopened.

For this case, the build.dart needs to add the transitive closure of dynamic libraries as assets.

When we want to start doing tree shaking of whole dylibs (not static linking), we need to take into account the dependencies between dylibs.
This means we need to extend the CLI protocol to be able to communicate the dependencies between assets.

Also, if the dylib paths inside the dylib need to be rewritten in Flutter: install_name_tool -change <abs-deps-path> @executable_path/../Frameworks/lib…dylib dependent.dylib (We already do something like this in Flutter for the main dylib name itself.)

• [ ] Add dependencies to protocol
• Use install_name_tool to patch up moved dylibs in Flutter
• Add example / test on this repo for Dart standalone.
• Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency.

Concrete use case: libexif depends on libintl.

Thanks for reporting @mkustermann

[blaugold]

I need support for this with a use case where I have a library, but for some features of the library I need some native glue code so that I can use them with Dart. The library is only available as a prebuilt binary, so I can't compile the library and the glue code into one binary. The library with the glue code needs to be linked to the primary library. Loading the primary library from the glue code library works for most configurations (only Flutter + macOS has been a problem), but that's probably coincidental.

@dcharkes I'm happy to work on this if you could give me some pointers.

[dcharkes]

One of the challenges here is that the embedder renames/repackages dylibs. So relying on the C/C++ dynamic loader with a fixed char* name doesn't work well. (I presume this is the cause of the issues for MacOS?)

And Flutter does something else than Dart. So you'd need a different file path when called from Flutter than when called from Dart.

Some options:

• Should we have a dart_api.h function that allows you to get the absolute dlopen path for an asset ID maybe? And then in your code you can call dlopen with the resolved path?
• Or maybe cleaner, just have such function in Dart. And then you pass the absolute dlopen path from Dart to native code and do dlopen in native code?

Both of these assume that you use "dynamic loading" e.g. nothing happens at the link step in the native compiler.

[ ] Add dependencies to protocol

I believe this is not needed. The way we have designed the link hooks should allow for simply reporting both the main and support dylib (based on the eventual tree-shaking info / resources.json)

@blaugold Can you elaborate on your use case? Are you doing dynamic loading? e.g. doing dlopen at runtime?

[blaugold]

One of the challenges here is that the embedder renames/repackages dylibs. So relying on the C/C++ dynamic loader with a fixed char* name doesn't work well. (I presume this is the cause of the issues for MacOS?)

That's precisely the issue.

Can you elaborate on your use case? Are you doing dynamic loading? e.g. doing dlopen at runtime?

I have two libraries: cblite and cblitedart. cblite is downloaded by the build hook. cblitedart contains the glue code to be able to use cblite and is built with CBuilder by the build hook. I have generated @Native bindings through ffigen for both libraries, so they are being loaded by whatever API Dart is using to load libraries for @Native bindings.

Here is the build hook:

import 'package:cbl_native_assets/src/support/edition.dart';
import 'package:cbl_native_assets/src/version.dart';
import 'package:logging/logging.dart';
import 'package:native_assets_cli/native_assets_cli.dart';
import 'package:native_toolchain_c/native_toolchain_c.dart';

import 'cblite_builder.dart';

const _edition = Edition.enterprise;

final _logger = Logger('')
  ..level = Level.ALL
  // ignore: avoid_print
  ..onRecord.listen((record) => print(record.message));

void main(List<String> arguments) async {
  await build(arguments, (config, output) async {
    output.addDependencies([
      config.packageRoot.resolve('hook/build.dart'),
      config.packageRoot.resolve('lib/src/version.dart'),
    ]);

    const cbliteBuilder = CbliteBuilder(
      version: cbliteVersion,
      edition: _edition,
    );

    await cbliteBuilder.run(
      buildConfig: config,
      buildOutput: output,
      logger: _logger,
    );

    final cbliteLibraryUri = output.assets
        .whereType<NativeCodeAsset>()
        .map((asset) => asset.file)
        .singleOrNull;

    final cblitedartBuilder = CBuilder.library(
      name: 'cblitedart',
      assetName: 'src/bindings/cblitedart.dart',
      language: Language.cpp,
      std: 'c++17',
      cppLinkStdLib: config.targetOS == OS.android ? 'c++_static' : null,
      defines: {
        if (_edition == Edition.enterprise) 'COUCHBASE_ENTERPRISE': '1',
      },
      flags: [
        if (cbliteLibraryUri != null)
          ...switch (config.targetOS) {
            OS.iOS => [
                '-F${cbliteLibraryUri.resolve('../').toFilePath()}',
                '-framework',
                'CouchbaseLite',
              ],
            _ => [
                '-L${cbliteLibraryUri.resolve('./').toFilePath()}',
                '-lcblite',
              ]
          },
        if (config.targetOS == OS.iOS) '-miphoneos-version-min=12.0',
        if (config.targetOS == OS.android) ...['-lc++abi']
      ],
      includes: [
        'src/vendor/cblite/include',
        'src/vendor/dart/include',
        'src/cblitedart/include',
      ],
      sources: [
        'src/cblitedart/src/AsyncCallback.cpp',
        'src/cblitedart/src/CBL+Dart.cpp',
        'src/cblitedart/src/Fleece+Dart.cpp',
        'src/cblitedart/src/Sentry.cpp',
        'src/cblitedart/src/Utils.cpp',
        'src/vendor/dart/include/dart/dart_api_dl.c',
      ],
    );
    await cblitedartBuilder.run(
      config: config,
      output: output,
      logger: _logger,
    );
  });
}

The full package is also available publicly on GitHub.

On macOS + Flutter I get the following exception:

flutter: 00:00 +0: (setUpAll) [E]
flutter:   Invalid argument(s): Couldn't resolve native function 'CBLDart_Initialize' in 'package:cbl_native_assets/src/bindings/cblitedart.dart' : Failed to load dynamic library 'cblitedart.framework/cblitedart': Failed to load dynamic library 'cblitedart.framework/cblitedart': dlopen(cblitedart.framework/cblitedart, 0x0001): Library not loaded: @rpath/libcblite.3.dylib
    Referenced from: <3369C631-E4BB-3CC9-9B88-55F5E82A5804> /Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/cblitedart.framework/Versions/A/cblitedart
    Reason: tried: '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/FlutterMacOS.framework/Versions/A/./libcblite.3.dylib' (no such file), '/usr/local/lib/./libcblite.3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/./libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/FlutterMacOS.framework/Versions/A/../../../libcblite.3.dylib' (no such file), '/usr/lib/swift/libcblite.3.dylib' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/MacOS/Frameworks/libcblite.3.dylib' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libcblite.3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libcblite.3.dylib' (no such file).

flutter:   dart:ffi-patch/ffi_patch.dart                                                                                                               Native._ffi_resolver.#ffiClosure0
  dart:ffi-patch/ffi_patch.dart 1523:20                                                                                                       Native._ffi_resolver_function
  package:cbl_native_assets/src/bindings/cblitedart.dart                                                                                      CBLDart_Initialize
  package:cbl_native_assets/src/bindings/base.dart 315:43                                                                                     BaseBindings.initializeNativeLibraries.<fn>
  package:ffi/src/arena.dart 150:33                                                                                                           withZoneArena.<fn>
  dart:async/zone.dart 1399:13                                                                                                                _rootRun
  dart:async/zone.dart 1301:19                                                                                                                _CustomZone.run
  dart:async/zone.dart 1826:10                                                                                                                _runZoned
  dart:async/zone.dart 1763:10                                                                                                                runZoned
  package:ffi/src/arena.dart 149:12                                                                                                           withZoneArena
  package:cbl_native_assets/src/bindings/base.dart 302:5                                                                                      BaseBindings.initializeNativeLibraries
  package:cbl_native_assets/src/support/isolate.dart 52:19                                                                                    initPrimaryIsolate.<fn>
  package:cbl_native_assets/src/support/errors.dart 59:14                                                                                     runWithErrorTranslation
  package:cbl_native_assets/src/support/isolate.dart 51:3                                                                                     initPrimaryIsolate
  ===== asynchronous gap ===========================
  package:cbl_native_assets/src/couchbase_lite.dart 28:9                                                                                      CouchbaseLite.init.<fn>
  ===== asynchronous gap ===========================
  /Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/integration_test/cbl_e2e_tests/test_binding.dart 108:7  CblE2eTestBinding._setupTestLifecycleHooks.<fn>
  ===== asynchronous gap ===========================
  dart:async/future.dart 670:3                                                                                                                Future._kTrue
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1254:12                                                                                                                _CustomZone.bindUnaryCallbackGuarded.<fn>

[dcharkes]

Thanks @blaugold!

Where on https://github.com/cbl-dart/cbl-dart/ is cblitedart loading libcblite.3.dylib? Is it a dlopen in C/C++ (dynamic loading)? Or is it a reference to a symbol and the C dynamic linker adds the machine code that is triggering the loading of libcblite.3.dylib (dynamic linking).

As a workaround, can you force loading of libcblite first by doing a Native.addressOf on one of it's symbols before invoking a function with a @Native from cblitedart. (This workaround might not work though due to the install_name being changed.)

[blaugold]

Where on https://github.com/cbl-dart/cbl-dart/ is cblitedart loading libcblite.3.dylib? Is it a dlopen in C/C++ (dynamic loading)? Or is it a reference to a symbol and the C dynamic linker adds the machine code that is triggering the loading of libcblite.3.dylib (dynamic linking).

Sorry, I thought that was implied. 🙈 cblitedart is linked to cblite (here in the build hook), so cblite is implicitly loaded when I try the invoke a function from cblitedart in Dart.

As a workaround, can you force loading of libcblite first by doing a Native.addressOf on one of it's symbols before invoking a function with a @Native from cblitedart. (This workaround might not work though due to the install_name being changed.)

Ah, yes, I'm already doing that. If I remember correctly, that was a workaround that helped on iOS.

[dcharkes]

I think conceptually what we have to do is: not only update the install_path with the install_name_tool, but also all the paths that the dynamic linker will look at. Probably in the style of https://stackoverflow.com/questions/33991581/install-name-tool-to-update-a-executable-to-search-for-dylib-in-mac-os-x

For that we either need to (a) come up with an algorithm/heuristic to decide which dynamic linking paths we need to update or (b) explicitly specify these paths in the NativeCodeAsset somewhere (path -> assetId. And then flutter_tools which does the install_name change can then do path -> new path.)

@blaugold Can you explore this with install_name_tool and otool in flutter_tools ?

Ah, yes, I'm already doing that. If I remember correctly, that was a workaround that helped on iOS.

Hopefully that should not be needed anymore afterwards. 🙂

[blaugold]

@blaugold Can you explore this with install_name_tool and otool in flutter_tools ?

@dcharkes Thanks for the pointers! I'll see how far I can get. 🙂

[blaugold]

@dcharkes I've created a PR: flutter/flutter#153054. Let me know what you think.