[native_assets_cli] Assets depending on other assets

[dcharkes]

trafficstars

Dynamic libraries can try to load other dynamic libraries when they are dlopened.

For this case, the build.dart needs to add the transitive closure of dynamic libraries as assets.

When we want to start doing tree shaking of whole dylibs (not static linking), we need to take into account the dependencies between dylibs. This means we need to extend the CLI protocol to be able to communicate the dependencies between assets.

Also, if the dylib paths inside the dylib need to be rewritten in Flutter: install_name_tool -change <abs-deps-path> @executable_path/../Frameworks/lib…dylib dependent.dylib (We already do something like this in Flutter for the main dylib name itself.)

• ~~[ ] Add dependencies to protocol~~
• [ ] Use install_name_tool to patch up moved dylibs in Flutter
• [ ] Add example / test on this repo for Dart standalone.
• [ ] Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency.

Concrete use case: libexif depends on libintl.

Thanks for reporting @mkustermann

[blaugold]

I need support for this with a use case where I have a library, but for some features of the library I need some native glue code so that I can use them with Dart. The library is only available as a prebuilt binary, so I can't compile the library and the glue code into one binary. The library with the glue code needs to be linked to the primary library. Loading the primary library from the glue code library works for most configurations (only Flutter + macOS has been a problem), but that's probably coincidental.

@dcharkes I'm happy to work on this if you could give me some pointers.

[dcharkes]

One of the challenges here is that the embedder renames/repackages dylibs. So relying on the C/C++ dynamic loader with a fixed char* name doesn't work well. (I presume this is the cause of the issues for MacOS?)

And Flutter does something else than Dart. So you'd need a different file path when called from Flutter than when called from Dart.

Some options:

• Should we have a dart_api.h function that allows you to get the absolute dlopen path for an asset ID maybe? And then in your code you can call dlopen with the resolved path?
• Or maybe cleaner, just have such function in Dart. And then you pass the absolute dlopen path from Dart to native code and do dlopen in native code?

Both of these assume that you use "dynamic loading" e.g. nothing happens at the link step in the native compiler.

[ ] Add dependencies to protocol

I believe this is not needed. The way we have designed the link hooks should allow for simply reporting both the main and support dylib (based on the eventual tree-shaking info / resources.json)

@blaugold Can you elaborate on your use case? Are you doing dynamic loading? e.g. doing dlopen at runtime?

[blaugold]

One of the challenges here is that the embedder renames/repackages dylibs. So relying on the C/C++ dynamic loader with a fixed char* name doesn't work well. (I presume this is the cause of the issues for MacOS?)

That's precisely the issue.

Can you elaborate on your use case? Are you doing dynamic loading? e.g. doing dlopen at runtime?

I have two libraries: cblite and cblitedart. cblite is downloaded by the build hook. cblitedart contains the glue code to be able to use cblite and is built with CBuilder by the build hook. I have generated @Native bindings through ffigen for both libraries, so they are being loaded by whatever API Dart is using to load libraries for @Native bindings.

Here is the build hook:

import 'package:cbl_native_assets/src/support/edition.dart';
import 'package:cbl_native_assets/src/version.dart';
import 'package:logging/logging.dart';
import 'package:native_assets_cli/native_assets_cli.dart';
import 'package:native_toolchain_c/native_toolchain_c.dart';

import 'cblite_builder.dart';

const _edition = Edition.enterprise;

final _logger = Logger('')
  ..level = Level.ALL
  // ignore: avoid_print
  ..onRecord.listen((record) => print(record.message));

void main(List<String> arguments) async {
  await build(arguments, (config, output) async {
    output.addDependencies([
      config.packageRoot.resolve('hook/build.dart'),
      config.packageRoot.resolve('lib/src/version.dart'),
    ]);

    const cbliteBuilder = CbliteBuilder(
      version: cbliteVersion,
      edition: _edition,
    );

    await cbliteBuilder.run(
      buildConfig: config,
      buildOutput: output,
      logger: _logger,
    );

    final cbliteLibraryUri = output.assets
        .whereType<NativeCodeAsset>()
        .map((asset) => asset.file)
        .singleOrNull;

    final cblitedartBuilder = CBuilder.library(
      name: 'cblitedart',
      assetName: 'src/bindings/cblitedart.dart',
      language: Language.cpp,
      std: 'c++17',
      cppLinkStdLib: config.targetOS == OS.android ? 'c++_static' : null,
      defines: {
        if (_edition == Edition.enterprise) 'COUCHBASE_ENTERPRISE': '1',
      },
      flags: [
        if (cbliteLibraryUri != null)
          ...switch (config.targetOS) {
            OS.iOS => [
                '-F${cbliteLibraryUri.resolve('../').toFilePath()}',
                '-framework',
                'CouchbaseLite',
              ],
            _ => [
                '-L${cbliteLibraryUri.resolve('./').toFilePath()}',
                '-lcblite',
              ]
          },
        if (config.targetOS == OS.iOS) '-miphoneos-version-min=12.0',
        if (config.targetOS == OS.android) ...['-lc++abi']
      ],
      includes: [
        'src/vendor/cblite/include',
        'src/vendor/dart/include',
        'src/cblitedart/include',
      ],
      sources: [
        'src/cblitedart/src/AsyncCallback.cpp',
        'src/cblitedart/src/CBL+Dart.cpp',
        'src/cblitedart/src/Fleece+Dart.cpp',
        'src/cblitedart/src/Sentry.cpp',
        'src/cblitedart/src/Utils.cpp',
        'src/vendor/dart/include/dart/dart_api_dl.c',
      ],
    );
    await cblitedartBuilder.run(
      config: config,
      output: output,
      logger: _logger,
    );
  });
}

The full package is also available publicly on GitHub.

On macOS + Flutter I get the following exception:

flutter: 00:00 +0: (setUpAll) [E]
flutter:   Invalid argument(s): Couldn't resolve native function 'CBLDart_Initialize' in 'package:cbl_native_assets/src/bindings/cblitedart.dart' : Failed to load dynamic library 'cblitedart.framework/cblitedart': Failed to load dynamic library 'cblitedart.framework/cblitedart': dlopen(cblitedart.framework/cblitedart, 0x0001): Library not loaded: @rpath/libcblite.3.dylib
    Referenced from: <3369C631-E4BB-3CC9-9B88-55F5E82A5804> /Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/cblitedart.framework/Versions/A/cblitedart
    Reason: tried: '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/FlutterMacOS.framework/Versions/A/./libcblite.3.dylib' (no such file), '/usr/local/lib/./libcblite.3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/./libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/FlutterMacOS.framework/Versions/A/../../../libcblite.3.dylib' (no such file), '/usr/lib/swift/libcblite.3.dylib' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/Frameworks/libcblite.3.dylib' (no such file), '/Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/build/macos/Build/Products/Debug/cbl_e2e_tests_native_assets_flutter.app/Contents/MacOS/Frameworks/libcblite.3.dylib' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libcblite.3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libcblite.3.dylib' (no such file).

flutter:   dart:ffi-patch/ffi_patch.dart                                                                                                               Native._ffi_resolver.#ffiClosure0
  dart:ffi-patch/ffi_patch.dart 1523:20                                                                                                       Native._ffi_resolver_function
  package:cbl_native_assets/src/bindings/cblitedart.dart                                                                                      CBLDart_Initialize
  package:cbl_native_assets/src/bindings/base.dart 315:43                                                                                     BaseBindings.initializeNativeLibraries.<fn>
  package:ffi/src/arena.dart 150:33                                                                                                           withZoneArena.<fn>
  dart:async/zone.dart 1399:13                                                                                                                _rootRun
  dart:async/zone.dart 1301:19                                                                                                                _CustomZone.run
  dart:async/zone.dart 1826:10                                                                                                                _runZoned
  dart:async/zone.dart 1763:10                                                                                                                runZoned
  package:ffi/src/arena.dart 149:12                                                                                                           withZoneArena
  package:cbl_native_assets/src/bindings/base.dart 302:5                                                                                      BaseBindings.initializeNativeLibraries
  package:cbl_native_assets/src/support/isolate.dart 52:19                                                                                    initPrimaryIsolate.<fn>
  package:cbl_native_assets/src/support/errors.dart 59:14                                                                                     runWithErrorTranslation
  package:cbl_native_assets/src/support/isolate.dart 51:3                                                                                     initPrimaryIsolate
  ===== asynchronous gap ===========================
  package:cbl_native_assets/src/couchbase_lite.dart 28:9                                                                                      CouchbaseLite.init.<fn>
  ===== asynchronous gap ===========================
  /Users/terwesten/dev/cbl-dart/cbl-dart/packages/cbl_e2e_tests_native_assets_flutter/integration_test/cbl_e2e_tests/test_binding.dart 108:7  CblE2eTestBinding._setupTestLifecycleHooks.<fn>
  ===== asynchronous gap ===========================
  dart:async/future.dart 670:3                                                                                                                Future._kTrue
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1254:12                                                                                                                _CustomZone.bindUnaryCallbackGuarded.<fn>

[dcharkes]

Thanks @blaugold!

Where on https://github.com/cbl-dart/cbl-dart/ is cblitedart loading libcblite.3.dylib? Is it a dlopen in C/C++ (dynamic loading)? Or is it a reference to a symbol and the C dynamic linker adds the machine code that is triggering the loading of libcblite.3.dylib (dynamic linking).

As a workaround, can you force loading of libcblite first by doing a Native.addressOf on one of it's symbols before invoking a function with a @Native from cblitedart. (This workaround might not work though due to the install_name being changed.)

[blaugold]

Where on https://github.com/cbl-dart/cbl-dart/ is cblitedart loading libcblite.3.dylib? Is it a dlopen in C/C++ (dynamic loading)? Or is it a reference to a symbol and the C dynamic linker adds the machine code that is triggering the loading of libcblite.3.dylib (dynamic linking).

Sorry, I thought that was implied. 🙈 cblitedart is linked to cblite (here in the build hook), so cblite is implicitly loaded when I try the invoke a function from cblitedart in Dart.

As a workaround, can you force loading of libcblite first by doing a Native.addressOf on one of it's symbols before invoking a function with a @Native from cblitedart. (This workaround might not work though due to the install_name being changed.)

Ah, yes, I'm already doing that. If I remember correctly, that was a workaround that helped on iOS.

[dcharkes]

I think conceptually what we have to do is: not only update the install_path with the install_name_tool, but also all the paths that the dynamic linker will look at. Probably in the style of https://stackoverflow.com/questions/33991581/install-name-tool-to-update-a-executable-to-search-for-dylib-in-mac-os-x

For that we either need to (a) come up with an algorithm/heuristic to decide which dynamic linking paths we need to update or (b) explicitly specify these paths in the NativeCodeAsset somewhere (path -> assetId. And then flutter_tools which does the install_name change can then do path -> new path.)

@blaugold Can you explore this with install_name_tool and otool in flutter_tools ?

Ah, yes, I'm already doing that. If I remember correctly, that was a workaround that helped on iOS.

Hopefully that should not be needed anymore afterwards. 🙂

[blaugold]

@blaugold Can you explore this with install_name_tool and otool in flutter_tools ?

@dcharkes Thanks for the pointers! I'll see how far I can get. 🙂

[blaugold]

@dcharkes I've created a PR: https://github.com/flutter/flutter/pull/153054. Let me know what you think.

[dcharkes]

@dcharkes I've created a PR: flutter/flutter#153054. Let me know what you think.

Awesome!

While reviewing I had some more thoughts on how to have dynamic dependencies more usable, documented and tested:

• Add example / test on this repo for Dart standalone (where it should work without any additional changes.)
• Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency. (If we don't have a way to do it with the current APIs.) This would then also be usable in the integration test in https://github.com/flutter/flutter/pull/153054.

(Feel free to leave these TODOs open.)

[blaugold]

(Feel free to leave these TODOs open.)

👍 I'll try and find some time to work on those.

While updating the integration tests in https://github.com/flutter/flutter/pull/153054 I found that it's currently not possible to use CBuilder.flags to pass flags to the linker when building for Windows. MSVC requires that all linker flags are passed after other compiler flags and separated from those with /link. Maybe CBuilder could also support LinkerOptions.

Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency.

CBuilder could get an option similar to List<Linkable> linkTo, which adds the necessary linker flags (would be nice if LinkerOptions had something like a merge method for combining with user provided linker flags). Linkable would be an interface that builders implement that contribute linkable native code assets, such as CBuilder.

[dcharkes]

(Feel free to leave these TODOs open.)

👍 I'll try and find some time to work on those.

While updating the integration tests in flutter/flutter#153054 I found that it's currently not possible to use CBuilder.flags to pass flags to the linker when building for Windows. MSVC requires that all linker flags are passed after other compiler flags and separated from those with /link. Maybe CBuilder could also support LinkerOptions.

I don't believe we have looked into supporting anything else than Linux yet w.r.t. linking.

• https://github.com/dart-lang/native/issues/1376

What compiler flags need adding?

Please feel free to submit PRs for the missing OSes! 😄

Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency.

CBuilder could get an option similar to List<Linkable> linkTo, which adds the necessary linker flags

That makes sense. (What does Linkable contain besides a Uri?)

(would be nice if LinkerOptions had something like a merge method for combining with user provided linker flags).

Do you envision having some helper code producing LinkerOptions objects per dylib that you want to pass to the dynamic linker for the final dylib?

Linkable would be an interface that builders implement that contribute linkable native code assets, such as CBuilder.

Currently, the Builders stream assets to the Output. So they are not composable. And the output is not meant to be read be subsequent builder invocations. The Builders are designed to have all their config in the constructor and then only stream to output. Would it be possible to have the information in a good way in the builder constructor calls? If you know the output dylib path for the first lib, you can pass that in to the first builder as target path, and to the second builder as include/link path.

(If there's a need to change the design of builders to allow information flowing from one to the other, I'd be open to that. But I'd like a good reason to deviate from the current design where the constructor is static configuration as object literal, and the output only streams to the hook output.)

Edit: It looks like I'm reviewing out of order. Based on https://github.com/dart-lang/native/pull/1413 I think the current design with Builders being configured in the constructor should still work. Let me know if you hit any issues. 👍

[blaugold]

I think I have to take a step back and consider all the scenarios in which dynamic linking of libraries built by a build hook needs to work:

• Packaged Flutter app (flutter build)
• Packaged Dart app (dart build)
• Flutter tests (flutter test)
• Dart tests (dart test)
• Dart script (dart run)

This is my understanding of what embedders currently do with the libraries built by a build hook:

OS	Packaged Flutter app	Packaged Dart app	Flutter tests	Dart tests	Dart script
macOS	Repackage library into multi-arch framework in Frameworks directory of app package	Not sure if a fat binary is created before packaging
iOS	Same as macOS
Linux	Copy to lib directory of app package
Android	Copy library into jniLibs; Libraries end up in lib directory of app package
Windows	Copy to root of app package, next to app executable
Common		Copy library into lib directory of app package	Copy library into build/native_assets/{os}	Directly use library from build hook output directory	Directly use library from build hook output directory

Dynamic linking in build hooks

Currently a build hook can get dynamic linking right so that loading of the linked library works in most scenarios.

As an aside w.r.t. https://github.com/dart-lang/native/issues/1425, in Dart tests and scripts, libraries built by other hooks, would be difficult to use because libraries from different packages are not copied to the same directory.

macOS and iOS

Requirements:

• place all libraries next to each other in the build hook output directory.
• build the library that is dynamically linked to with install name @rpath/libfoo.dylib.
• build libraries that link to the dynamic library with -L{build_directory} -lfoo.

When the linked library needs to be loaded, the dynamic linker will substitute @rpath with the rpath directories that have been set for the loading library (rpaths of the executable are also considered) and attempt to load the library from one of those directories.

The app executables that are created for Flutter macOS and iOS apps, the flutter-tester executable and the standalone Dart executable all have rpaths for @loader_path/., @loader_path/../../.. and @executable_path/Frameworks. These rpaths cover all the scenarios in which the library might be loaded, as long as the libraries are always placed next to each other.

Run-Path Dependent Libraries

Linux and Android

Requirements:

• place all libraries next to each other in the build hook output directory.
• build libraries that link to the dynamic library with -rpath=$ORIGIN/. -L{build_directory} -lfoo.

The $ORIGIN variable is expanded to the directory of the loading library and is added to the list of directories that the dynamic linker searches for libraries. By using $ORIGIN/. and placing all libraries next to each other in all scenarios, the dynamic linker will be able to find the linked library.

ld.so - dynamic linker/loader

Windows

Requirements:

• place all libraries next to each other in the build hook output directory.
• pass the import library (foo.lib) on the command line when compiling the library that links to the dynamic library.

One of the default locations the dynamic linker will look for foo.dll is next to the executable. This is why dynamic linking works for packaged Flutter apps.

Directories that should be searched can be added to the PATH environment variable. Alternatively, AddDllDirectory can be used at runtime to add directories to the search path, if modifying the PATH environment variable is not an option.

Dynamic-link library search order

---

Possible next steps:

• For all scenarios, copy the libraries to a common directory. Then the layout of the build hook output directory doesn't matter.
• For macOS and iOS, modify the linked and linking libraries to have the correct install names and rpaths.
  • For Flutter https://github.com/flutter/flutter/pull/153054 already would do that. Not for flutter tester though, which we should probably change, so that build hooks don't have to build all libraries with an install name of @rpath/libfoo.dylib. If we don't do this and the build hook does not set an explicit install name, the compiler sets the absolute path of the output file as the install name, and at runtime, the linked library will be loaded from the build hook output directory instead of build/native_assets/{os}.
  • Also update install names and rpaths in standalone Dart.
• For Linux and Android, add -rpath=$ORIGIN/. to linking libraries (or just all libraries?).
• For standalone Dart on Windows, add the common directory to the DLL search paths with SetDllDirectory.

[dcharkes]

Thanks for this very detailed write-up @blaugold! ❤️

As an aside w.r.t. #1425, in Dart tests and scripts, libraries built by other hooks, would be difficult to use because libraries from different packages are not copied to the same directory.

I had the same realization. If we want to support that, we need to either copy all libs into the same dir, or have all directories as library paths.

[...] the standalone Dart executable all have rpaths for @loader_path/. [...]

Ah right, I might have played around with that years back.

For all scenarios, copy the libraries to a common directory. Then the layout of the build hook output directory doesn't matter.

This would entail probably also entail addressing:

• https://github.com/dart-lang/native/issues/1425

(Or adding a big fat disclaimer somewhere.)

It would indeed be good to just always copy for all running modes. That way we ensure that naming conflicts (or our way of resolving them) already show up for dart test before even trying flutter run.

If we don't do this and the build hook does not set an explicit install name, the compiler sets the absolute path of the output file as the install name, and at runtime, the linked library will be loaded from the build hook output directory instead of build/native_assets/{os}.

If we choose to not copy things around, that could be fine for Dart standalone (and flutter pub run test). But it's cleaner to unify our approaches and always copy.

Also update install names and rpaths in standalone Dart.

We should consider if this logic can be part of the native_assets_builder package so that we don't have to duplicate it across flutter_tools and dartdev. One challenge there is that all these different ways of running code package things differently. E.g. in a framework, not in a framework, etc. I am open to suggestions here.

For Linux and Android, add -rpath=$ORIGIN/. to linking libraries (or just all libraries?).

This also depends on the directory layout of how an app is packaged. If we standardize on everything being in one directory, we can standardize the rpath. The question is again whose responsibility this would be. It would be nice if the native assets builder can do things, but in the end it is the embedder (flutter_tools, dartdev) that decides the directory layout of where the dylibs are compared to the exe and the other dylibs. We can't bake the logic into package:native_assets_builder or package:native_toolchain_c if we don't mandate a directory layout.

I'm a bit hesitant to mandate a directory layout. For example, we can imagine embedding dylibs inside a zip file and dlopening with offsets in an embedder. See the documentation on https://dart-review.googlesource.com/c/sdk/+/361881.

So, I think for now this would be best implemented in flutter_tools and dartdev. And then we need to think if we can share the implementation across those two by moving some helper functions in a shared location (package:native_assets_builder/linker_helpers.dart?) later.

For standalone Dart on Windows, add the common directory to the DLL search paths with SetDllDirectory.

This is a function that should be called inside dart.exe and dartaotruntime.exe? What should that directory be? The "script path" (the dart script file or the kernel snapshot, or the aotsnapshot, or the exe if used with dart compile exe) parent dir? That we use for relative path resolution of dynamic libraries.

I don't see any call to that function in the Flutter code base though. I guess that's because we don't have a precompiled flutter.exe or flutter_aot.exe? So the dynamic linker just looks next to the exe.

[blaugold]

For Linux and Android, add -rpath=$ORIGIN/. to linking libraries (or just all libraries?).

This also depends on the directory layout of how an app is packaged. If we standardize on everything being in one directory, we can standardize the rpath. The question is again whose responsibility this would be. It would be nice if the native assets builder can do things, but in the end it is the embedder (flutter_tools, dartdev) that decides the directory layout of where the dylibs are compared to the exe and the other dylibs.

It's actually surprisingly difficult to modify the rpath or dependency strings of elf binaries, as opposed to what is possible with install_name_tool on macOS. There is a similar tool called patchelf, but it's not part of the compiler tool chain and does not seem to be reliable. Another tool is chrpath. It's also not part of the compiler tool chain and only allows you to change an already existing rpath. It cannot add one if the binary does not already contain one. I've considered using the compiler to create a new dylib from an existing one with --whole-archive but you would have to know the linker options that were used to create the source dylib.

This is a function that should be called inside dart.exe and dartaotruntime.exe? What should that directory be? The "script path" (the dart script file or the kernel snapshot, or the aotsnapshot, or the exe if used with dart compile exe) parent dir? That we use for relative path resolution of dynamic libraries.

Yes, dart.exe and dartaotruntime.exe would have to call SetDllDirectory (or probably better AddDllDirectory) with the directory where all the native asset libraries have been copied to. For dart run and dart test that could be a directory in .dart_tool and for the executable created by dart build the lib directory relative to it.

[dcharkes]

For Linux and Android, add -rpath=$ORIGIN/. to linking libraries (or just all libraries?).

This also depends on the directory layout of how an app is packaged. If we standardize on everything being in one directory, we can standardize the rpath. The question is again whose responsibility this would be. It would be nice if the native assets builder can do things, but in the end it is the embedder (flutter_tools, dartdev) that decides the directory layout of where the dylibs are compared to the exe and the other dylibs.

It's actually surprisingly difficult to modify the rpath or dependency strings of elf binaries, as opposed to what is possible with install_name_tool on macOS. There is a similar tool called patchelf, but it's not part of the compiler tool chain and does not seem to be reliable. Another tool is chrpath. It's also not part of the compiler tool chain and only allows you to change an already existing rpath. It cannot add one if the binary does not already contain one.

Interesting. Thanks for the research!

• build libraries that link to the dynamic library with -rpath=$ORIGIN/. -L{build_directory} -lfoo.

If a user does this in their hook/builder.dart, will it work in all scenarios? If yes we should maybe standardize on that (either in documentation or in the package as default flags).

(If that doesn't work, we might need to resort to having some configuration in the build config. I'd like to avoid that. Then a next question would be: How do other build systems that wrap existing build systems deal with linked dependencies on Linux.)

I've considered using the compiler to create a new dylib from an existing one with --whole-archive but you would have to know the linker options that were used to create the source dylib.

Yeah, that doesn't sound like a portable solution.

This is a function that should be called inside dart.exe and dartaotruntime.exe? What should that directory be? The "script path" (the dart script file or the kernel snapshot, or the aotsnapshot, or the exe if used with dart compile exe) parent dir? That we use for relative path resolution of dynamic libraries.

Yes, dart.exe and dartaotruntime.exe would have to call SetDllDirectory (or probably better AddDllDirectory) with the directory where all the native asset libraries have been copied to. For dart run and dart test that could be a directory in .dart_tool and for the executable created by dart build the lib directory relative to it.

To know the directory (or directories), we should probably add that info to native_assets.yaml. Then the embedder (Dart standalone or Flutter) who's bundling the native assets mapping can also specify the dll directories instead of trying to guess in the VM or in the embedder without knowing what command is being run. And the most natural time to add the dll directories would probably be just before loading the first dll we're loading with @Native external functions. For keeping the responsibilities in the right place, it should probably be done as another callback in NativeAssetsApi that the VM calls before the first dlopen. (Relevant code: https://dart-review.googlesource.com/c/sdk/+/361881 )

[blaugold]

• build libraries that link to the dynamic library with -rpath=$ORIGIN/. -L{build_directory} -lfoo.

If a user does this in their hook/builder.dart, will it work in all scenarios?

With these link options, the layout in build hook output directory does not matter if we standardize on copying all native code assets to a single directory.

If yes we should maybe standardize on that (either in documentation or in the package as default flags).

Yeah, probably both, so that someone implementing a builder that wraps another build system is aware.

[dcharkes]

With these link options, the layout in build hook output directory does not matter if we standardize on copying all native code assets to a single directory.

Standardize on copying all into a single directory, and standardize on not renaming on naming conflicts!

• https://github.com/dart-lang/native/issues/1425

[dcharkes]

Thanks a ton @blaugold! ❤️

• Add a way to to build two dylibs with CBuilder where one depends on the dynamic linker to load the other as dependency.

I think this is the only thing left to do for this now. And we have some WIP in:

• https://github.com/dart-lang/native/pull/1423