[native_assets_cli] Sharing work between build hooks

[dcharkes]

trafficstars

Build hooks are per OS/arch. However, if assets are identical across multiple invocations, and the work to produce those assets is non-trivial, such work should be shared.

Examples:

• Data assets which are "compiled" (svg -> tessellated, 3d model -> bytes, translation json).
• Native code assets which are downloaded as fat binaries (e.g. containing already MacOS arm64 and MacOS x64)

Non-examples:

• Native code assets different per OS and per arch.
• Data assets that are on disk already and are simply reported.

We have multiple avenues of sharing the work:

1. Add documentation that the actual work should happen in .dart_tool/<your-package-name>.
2. Make the output directory per asset type, and give the same output directory for data assets for all invocations.

Both of these options require hook writers to think about locking due to concurrent invocations of the build hook for different OSes/architectures. So whatever we come up with for https://github.com/dart-lang/native/issues/1319, should be made available in an API to hook writers.

Option (2) doesn't support sharing native code assets, so in for the second example, hook writers would still use option (1) even if we go for option (2).

[dcharkes]

A variant of 1. that's more user-friendly:

BuildConfig.sharedOutputDirectory that automatically lives in .dart_tool/native_assets_builder/<the-users-package-name>

[dcharkes]

API design:

// Copyright (c) 2024, the Dart project authors.  Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'dart:io';

import 'package:native_assets_cli/locking.dart';
import 'package:native_assets_cli/native_assets_cli.dart';

const assetName = 'asset.txt';

Future<void> main(List<String> args) async {
  await build(args, (config, output) async {
    final sharedDirectory = config.outputDirectoryShared;
    final assetPath = sharedDirectory.resolve(assetName);
    await runUnderDirectoryLock(
      Directory.fromUri(sharedDirectory),
      () async {
        final packageName = config.packageName;

        if (!config.dryRun) {
          // Pretend we're downloading assets here.
          await File.fromUri(assetPath).writeAsString('Hello world!');
        }

        output.addAsset(
          DataAsset(
            package: packageName,
            name: assetName,
            file: assetPath,
          ),
        );
      },
    );
  });
}

We'll need to add outputDirectoryShared to the BuildConfig and LinkConfig.

[dcharkes]

await runUnderDirectoryLock(

Thinking about this, the shared directory should be locked in the native assets builder, due to https://github.com/dart-lang/native/issues/1534. Otherwise, the native assets builder could be copying the assets while another hook invocation is trying to delete them at the same time. The native assets builder should only be deleting the hook after it has copied the necessary files to a temp/target location.

[dcharkes]

Note from discussion with @HosseinYousefi:

config.outputDirectoryShared is shared across all invocations of the same hook, config.outputDirectory is shared across only invocations that are 100% identical config. But we might have some cases in which things are not identical in a subset of configurations: for example debug/release doesn't make a difference in some native code, or images are only different per OS. The minimally shared approach is a safe default because users don't accidentally share the output dir when they shouldn't (for example if debug/release does make a difference but they only considered os/arch). However, the safe default could also be unnecessarily slow/wasteful. So we want to provide an easy way for users to go somewhere in between minimally and maximally shared.

The API could look something like:

Directory HookConfig.getSharedOutputDirectory({
  bool splitOnTargetOS: false, 
  bool splitOnTargetArchitecture: false,
  bool splitOnAndroidNdkVersion: false,
  // ...
});

This feel natural because one has to last what config aspects are important to branch on. However, users are more likely to accidentally share and have weird behavior.

The opposite would be quite weird:

Directory HookConfig.getOutputDirectory({
  bool shareAcrossTargetOS: false, 
  bool shareAcrossTargetArchitecture: false,
  bool shareAcrossAndroidNdkVersion: false,
  // ...
});

It would require users to list all the config params they don't care about. And this doesn't work with (a) us adding new parameters, and (b) some other embedder adding config parameters that we don't even know about.

To make things work for params that we don't know about:

Directory HookConfig.getOutputDirectory(List<Object> splitOn);

config.getOutputDirectory([config.targetOS, config.targetArchitecture, ...]);

[dcharkes]

We have input.outputDirectoryShared, and we have decided to invoke hooks per asset type.

• https://github.com/dart-lang/native/issues/1739