sdk reduce end-to-end AOT compilation time for large applications

This is an umbrella issue to track efforts around reduction of critical path when building large Flutter applications in AOT mode

Sep 02 '20 13:09 mraleph

This looks duplicate to https://github.com/dart-lang/sdk/issues/42442.

Sep 02 '20 16:09 alexmarkov

@alexmarkov this is more of an umbrella issue, not limited to just TFA but to general end-to-end performance.

Sep 02 '20 16:09 mraleph

One way to do this would be to support breaking things up into smaller increments of compilation.

This would allow for fanning out compilation steps to many different worker machines, and would also potentially help with incrementally compiling smaller changes between builds.

I'm more interested in faster incremental builds for the sake of performance tuning/debugging - right now, changing a single line of code requires repeating the entire process of kernel compiling/tfa/snapshotting, which can be very expensive on large codebases.

Oct 25 '21 17:10 dnfield

We could parallelize gen_snapshot step in a relatively straightforward fashion though it would potentially require us to forgo some of the late stage tree shaking (might not be that relevant for code quality after all).

Parallelizing TFA is an open problem - it is intrisically non-modular global step. You should compare it to similar global optimizations steps like LTO in LLVM or ProGuard et al

Oct 25 '21 17:10 mraleph

Google concerns: b/203690870

Oct 25 '21 18:10 xster

Perhaps something like what LLVM did with ThinLTO would be applicable for TFA? http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html

Oct 25 '21 23:10 dnfield

Hi,

Please guide me anyway to debug AOT progress? Our bank application build AOT too long.

stuck at step frontend_server.dart.snapshot

debug without AOT: 51 seconds (with 16 Gb RAM)

release with AOT: 51 minutes (with 16 Gb RAM) 31 minutes (with 32 Gb RAM)

But problem is frontend_server.dart.snapshot --verbose dont print anything helpful so we can not know where is problem. Is it possible to print what dart file is processing? So we can see where it stuck.

Thank you.

Sep 22 '22 03:09 quyenvsp

@quyenvsp which Flutter version are you on? (could you post flutter doctor -v?).

It is probably hitting some pathological case in the TFA.

Currently the best way to diagnose that requires a bit of manual work - it's described in https://github.com/dart-lang/sdk/issues/47546#issuecomment-955078084

get Dart SDK sources (as explained at dart-lang/sdk/wiki/Building#getting-the-source)
Checkout the version of Dart matching the version of Flutter you are using. I would be able to tell you the version if you post flutter doctor -v output.

Get exact command line arguments for frontend_server.dart.snapshot (e.g. by doing flutter build apk --release -v) and then run it from sources in the following way ($DART is the path to Dart SDK sources):

# $DART is the path to Dart SDK sources
# $ARGS are the arguments that are normally passed to frontend_server.dart.snapshot
$ dart -Dglobal.type.flow.print.stats=true --disable-dart-dev --packages=$DART/.packages $DART/pkg/frontend_server/bin/frontend_server_starter.dart $ARGS

/cc @alexmarkov (we should consider allowing users to set global.type.flow.print.stats=true through OS environment variables, rather than through Dart defines).

Sep 22 '22 07:09 mraleph

@mraleph

Thank so much for guide.

We have 2 bank application project almost similar (one just little smaller) but TFA time is more different 377947ms >< 1997843ms (~5 time slower).

Both do same Ubuntu machine (32Gb of ram), using Flutter stable 2.10.5 Dart 2.16.2 (Dart SDK sources), free memory before build each. one project build just waste 8Gb ram, the slow project take 17Gb ram

While we continue find where is problem, hope global.type.flow.print.stats=true result below of 2 project helpful. (Also have perf report but size is 200Mb and 600Mb, will upload if you need)

fine_project_result.txt slow_project_result.txt

We see a huge different 258907308 >< 738198250 invocations queried in cache, but it not detailed yet. Please is these anyway counter per packages/classes? Something like package/class A 1000 invocations, package/class B 2000 invocations,...By this way we easy know where is problem then find more on it.

Sep 25 '22 10:09 quyenvsp

@quyenvsp Thank you for the report. These huge compilation times are usually caused by a large amount of auto-generated Dart source code. Do you use Dart source auto-generation in your app?

By looking at the logs, it seems like the second app is ~3x times larger (10676547 vs. 28089433 summaries analyzed). That may explain the difference in the compilation times. So far I do not see anything out of the ordinary from those logs, except maybe that in both apps analysis hits the hard limit of 5000 maximum invocations cached per selector. You can try to vary (decrease) constants in the _SelectorApproximation class to see if it helps:

https://github.com/dart-lang/sdk/blob/17bd00df50ad3ddfee4dcea69d6de5bae41aba58/pkg/vm/lib/transformations/type_flow/analysis.dart#L762-L775

Also, you can try to run frontend server with -Dglobal.type.flow.print.timings=true option (as @mraleph explained above) to get a list of methods where analysis spent the most time. This option should print 3 tables of methods - Top summaries by number of times analyzed, Top summaries by dirty analysis time (including callees), in microseconds and Top summaries by pure analysis time (excluding callees), in microseconds. The most interesting are the last and the first tables.

Sep 27 '22 16:09 alexmarkov

hi, I'm facing the same issue, and most of my time is spent on the gen_snapshot, which takes nearly 8 minutes. 2023-10-24 16:31:32:218 : [ +1 ms] executing: gen_snapshot_arm64 --deterministic --save-obfuscation-map=build/ios/framework/Release/app_iOS_symbols.json --snapshot_kind=app-aot-assembly --assembly=/Volumes/data/workspace/.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/snapshot_assembly.S --dwarf-stack-traces --resolve-dwarf-paths --save-debugging-info=build/ios/framework/Release/app.ios-arm64.symbols --obfuscate /Volumes/data/workspace/.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/app.dill 2023-10-24 16:33:47:397 : ##[error][+135179 ms] Warning: The generated assembly code contains unobfuscated DWARF debugging information. 2023-10-24 16:33:47:398 : ##[error][ ] To avoid this, use --strip to remove it. 2023-10-24 16:33:47:646 : [ +247 ms] executing: xcrun cc -arch arm64 -miphoneos-version-min=11.0 -isysroot /Applications/Xcode_14.3.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS16.4.sdk -c /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/snapshot_assembly.S -o /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/snapshot_assembly.o 2023-10-24 16:38:04:903 : [+257257 ms] executing: xcrun clang -arch arm64 -miphoneos-version-min=11.0 -isysroot /Applications/Xcode_14.3.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS16.4.sdk -dynamiclib -Xlinker -rpath -Xlinker @executable_path/Frameworks -Xlinker -rpath -Xlinker @loader_path/Frameworks -fapplication-extension -install_name @rpath/App.framework/App -o /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/App.framework/App /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/snapshot_assembly.o 2023-10-24 16:38:05:763 : [ +859 ms] executing: xcrun dsymutil -o /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/App.framework.dSYM /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/App.framework/App 2023-10-24 16:38:08:158 : [+2395 ms] executing: xcrun strip -x /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/App.framework/App -o /Volumes/data/workspace/ /.dart_tool/flutter_build/206753438bffcc2db0ccae02ede2c598/arm64/App.framework/App 2023-10-24 16:38:09:312 : [+1153 ms] aot_assembly_release: Complete Are there any good way for outputting trace information, so I can better identify where the longest delays occur? Also, does gen_snapshot currently support incremental compilation? If not, are there any other methods to improve the compilation speed? Currently, our project takes close to 30 minutes to compile. Thank you.

Oct 24 '23 12:10 yaminet1024

@yaminet1024 how big is your code? You can try to run gen_snapshot_arm64 with --print_precompiler_timings and paste the output. You can also run gen_snapshot_arm64 with --trace-compiler, redirect it to 2>/tmp/compilation_trace.txt and then process the log using the following command

$ cat /tmp/compilation_trace.txt | perl -n -e 'm/--> \'([^\']+)\'.*time: (\d+)/ && do{$n=$1;$t=$2; $n =~ s/ /_/g; print ($n, " ", $t, "\n")}' | sort -n -k 2 > /tmp/sorted.txt

You can look at the methods that take longest to compile. If there are some outliers - we would like to know if there is something special about them (e.g. huge autogenerated code is one common suspect).

It would be nice to know histogram of compilation timings, how many methods are compiled in total (wc -l /tmp/sorted.txt should give that).

Oct 24 '23 13:10 mraleph

After some searching I believe this is the right issue to comment on... I am seeing large compile times for a Flutter app using quite a bit of generated serializer code. I'm not sure what is considered big in terms of an app.

I am seeing 25 minute release builds, here is output from verbose logs:

[  +86 ms] gen_dart_plugin_registrant: Complete
[   +7 ms] kernel_snapshot: Starting due to {}
[  +10 ms] Embedding native assets mapping /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/native_assets.yaml in kernel.
[   +6 ms] /Users/michaelgolfi/development/flutter/bin/cache/dart-sdk/bin/dartaotruntime --disable-dart-dev /Users/michaelgolfi/development/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot --sdk-root /Users/michaelgolfi/development/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk_product/ --target=flutter --no-print-incremental-dependencies -Ddart.vm.profile=false -Ddart.vm.product=true --delete-tostring-package-uri=dart:ui --delete-tostring-package-uri=package:flutter --aot --tfa --target-os ios --packages /Users/michaelgolfi/work/mobile/.dart_tool/package_config.json --output-dill /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/app.dill --depfile /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/kernel_snapshot.d --source file:///Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart -Dflutter.dart_plugin_registrant=file:///Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/dart_plugin_registrant.dart --native-assets /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/native_assets.yaml --verbosity=error package:phyla/main.dart
[+1096958 ms] kernel_snapshot: Complete
[+1782 ms] aot_assembly_release: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/michaelgolfi/development/flutter/packages/flutter_tools/lib/src/build_system/targets/ios.dart,/Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/app.dill,/Users/michaelgolfi/development/flutter/bin/internal/engine.version,/Users/michaelgolfi/development/flutter/bin/internal/engine.version}
[   +1 ms] targetingApplePlatform = true
[        ] extractAppleDebugSymbols = true
[        ] Will strip AOT snapshot manually after build and dSYM generation.
[        ] executing: /Users/michaelgolfi/development/flutter/bin/cache/artifacts/engine/ios-release/gen_snapshot_arm64 --deterministic --snapshot_kind=app-aot-assembly --assembly=/Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/snapshot_assembly.S /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/app.dill
[+59343 ms] executing: sysctl hw.optional.arm64
[   +3 ms] Exit code 0 from: sysctl hw.optional.arm64
[        ] hw.optional.arm64: 1
[        ] executing: /usr/bin/arch -arm64e xcrun cc -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk -c /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/snapshot_assembly.S -o /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/snapshot_assembly.o
[+105579 ms] executing: /usr/bin/arch -arm64e xcrun clang -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.5.sdk -dynamiclib -Xlinker -rpath -Xlinker @executable_path/Frameworks -Xlinker -rpath -Xlinker @loader_path/Frameworks -fapplication-extension -install_name @rpath/App.framework/App -o /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/App.framework/App /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/snapshot_assembly.o
[ +100 ms] executing: /usr/bin/arch -arm64e xcrun dsymutil -o /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/App.framework.dSYM /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/App.framework/App
[ +751 ms] executing: /usr/bin/arch -arm64e xcrun strip -x /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/App.framework/App -o /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/App.framework/App
[ +124 ms] aot_assembly_release: Complete
[        ] release_ios_bundle_flutter_assets: Starting due to {}

@mraleph I will try your suggestions above. So far I have tried to move most of the serializers into their own path dependency in the past, in the hopes it would cache the builds since they change less often than the main app.

Jun 27 '24 15:06 michael-golfi

I found that much time was spent on serializers generated from ferry_generator. I managed to reduce some of the generated code with some config and shaved 500s from release build time, an improvement of about 40%.

The shell script above has a missing quote somewhere. I gave up on it and wrote some Python to sort the files by time instead.

"""
  /Users/michaelgolfi/development/flutter/bin/cache/artifacts/engine/ios-release/gen_snapshot_arm64 \
  --print_precompiler_timings \
  --trace-compiler \
  --deterministic \
  --snapshot_kind=app-aot-assembly \
  --assembly=/Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/arm64/snapshot_assembly.S \
    /Users/michaelgolfi/work/mobile/.dart_tool/flutter_build/9dcd2357a6d80e978997d358d9163d28/app.dill 2>compilation_trace.txt
"""

def process_file(input_file, output_file):
    data = []

    with open(input_file, "r") as file:
        for line in file:
            if "-->" in line:
                parts = line.strip().split("time:")
                if len(parts) > 1:
                    time_part = (
                        parts[1].strip().split(" ")[0]
                    )  # Extract the time before 'us'
                    name_part = parts[0].split("'")[1]  # Extract the function name
                    name_part = name_part.replace(
                        " ", "_"
                    )  # Replace spaces with underscores
                    data.append((name_part, int(time_part)))

    # Sort data by the time, which is the second item in the tuple
    data.sort(key=lambda x: x[1])

    with open(output_file, "w") as file:
        for name, time in data:
            file.write(f"{name} {time}\n")


# Usage
process_file("compilation_trace.txt", "sorted.txt")

Jun 27 '24 19:06 michael-golfi

The ferry_generator generated code does seem to hit some sort of non-linearity in TFA.

I extracted a benchmark core from another issue I was looking at before (related to build_runner performance): https://github.com/mraleph/flutter_ferry_aot_stress_test. I can clearly see that as input grows TFA speed (input size divided by time taken) decreases: 0.49 kb/ms, then 0.25 kb/ms, then 0.11 kb/ms

@alexmarkov could you take this for a spin?

Jun 28 '24 10:06 mraleph

If helpful, I posted a repro on a build_runner issue a couple years ago.

Jun 28 '24 12:06 michael-golfi

What could cause the compilation of snapshot_assembly.o to take an extremely long time, much longer than compiling the snapshot itself?

[   +8 ms] /Users/****/.src/flutter/bin/cache/dart-sdk/bin/dartaotruntime --disable-dart-dev /Users/****/.src/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot --sdk-root /Users/****/.src/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --target=flutter --no-print-incremental-dependencies -Ddart.vm.profile=true -Ddart.vm.product=false --delete-tostring-package-uri=dart:ui --delete-tostring-package-uri=package:flutter --aot --tfa --target-os ios --packages /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/package_config.json --output-dill /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/program.dill --depfile /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/kernel_snapshot_program.d --source file:///Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart -Dflutter.dart_plugin_registrant=file:///Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/dart_plugin_registrant.dart --verbosity=error package:house591/main.dart

[+57447 ms] kernel_snapshot_program: Complete

[+4044 ms] native_assets: Starting due to {}

[  +95 ms] No packages with native assets. Skipping native assets compilation.

[        ] Writing native_assets.yaml.

[   +6 ms] Writing /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/native_assets.yaml done.

[        ] native_assets: Complete

[   +1 ms] kernel_snapshot_native_assets: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/native_assets.yaml,/Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/package_config_subset,/Users/****/.src/flutter/packages/flutter_tools/lib/src/build_system/targets/common.dart,/Users/****/.src/flutter/bin/internal/engine.version,/Users/****/.src/flutter/bin/internal/engine.version,/Users/****/.src/flutter/bin/internal/engine.version,/Users/****/.src/flutter/bin/internal/engine.version}

[        ] kernel_snapshot_native_assets: Complete

[        ] kernel_snapshot: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/program.dill,/Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/native_assets.dill}

[  +76 ms] kernel_snapshot: Complete

[ +952 ms] aot_assembly_profile: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/****/.src/flutter/packages/flutter_tools/lib/src/build_system/targets/ios.dart,/Users/****/.src/flutter/bin/internal/engine.version,/Users/****/.src/flutter/bin/internal/engine.version,/Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/app.dill}

[   +2 ms] targetingApplePlatform = true

[        ] extractAppleDebugSymbols = true

[        ] Will strip AOT snapshot manually after build and dSYM generation.

[        ] executing: /Users/****/.src/flutter/bin/cache/artifacts/engine/ios-profile/gen_snapshot_arm64 --deterministic --snapshot_kind=app-aot-assembly --assembly=/Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/snapshot_assembly.S /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/app.dill

[+105598 ms] executing: sysctl hw.optional.arm64

[   +6 ms] Exit code 0 from: sysctl hw.optional.arm64

[        ] hw.optional.arm64: 1

[        ] executing: /usr/bin/arch -arm64e xcrun cc -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS18.2.sdk -c /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/snapshot_assembly.S -o /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/snapshot_assembly.o

[+1396474 ms] executing: /usr/bin/arch -arm64e xcrun clang -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS18.2.sdk -dynamiclib -Xlinker -rpath -Xlinker @executable_path/Frameworks -Xlinker -rpath -Xlinker @loader_path/Frameworks -fapplication-extension -install_name @rpath/App.framework/App -o /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/App.framework/App /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/snapshot_assembly.o

[ +347 ms] executing: /usr/bin/arch -arm64e xcrun dsymutil -o /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/App.framework.dSYM /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/App.framework/App

[+1406 ms] executing: /usr/bin/arch -arm64e xcrun strip -x /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/App.framework/App -o /Users/****/source/repo/****_workspace/apps/****flutter/.dart_tool/flutter_build/194d86c3747d0e1c6eb4d33ecb8d9677/arm64/App.framework/App

[ +329 ms] aot_assembly_profile: Complete

Feb 27 '25 06:02 yiiim

In my Flutter project, when generating the iOS framework, the command “usr/bin/arch -arm64e xcrun cc -v -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS18.2.sdk -c /Users/xxx/Desktop/.dart_tool/flutter_build/2d151adfb8aeaa558257e5cae0b2158d/arm64/snapshot_assembly.S -o /Users/xxx/Desktop/.dart_tool/flutter_build/2d151adfb8aeaa558257e5cae0b2158d/arm64/snapshot_assembly.o” takes 40+ minutes to execute and there is no output from the console during the execution, can anyone tell me how to optimize the time spent on this process?

Mar 07 '25 06:03 zhangjianbin

@zhangjianbin @yiiim You should probably file a bug with Apple or LLVM folks and ask them why it takes so long to assemble. Seems like a bug. These .S files are effectively streams of bytes so they should not take so long.

Maybe we should stope relying on clang for this and just make our own Mach-O writer. It is not a rocket science.

Mar 07 '25 09:03 mraleph

llvm/llvm-project#68445 We should indeed consider not using clang.

Mar 07 '25 10:03 yiiim

@zhangjianbin @yiiim You should probably file a bug with Apple or LLVM folks and ask them why it takes so long to assemble. Seems like a bug. These .S files are effectively streams of bytes so they should not take so long.

Maybe we should stope relying on clang for this and just make our own Mach-O writer. It is not a rocket science.

@mraleph Clang is slower when compiling a large assembly source code Currently, the snapshot_assembly.S generated in my project is over 600 MB. Can the overall process be modified to generate smaller .S files to produce .o files, and then merge these generated .o files in the end?

Mar 10 '25 09:03 zhangjianbin

@zhangjianbin we can't split that file into smaller files because we need precise ordering of functions to be maintained.

We should just write our own Mach-O object file writer... It should be relatively straightforward.

cc @alexmarkov @rmacnak-google @sstrickl Any takers?

Mar 10 '25 11:03 mraleph

A side benefit here might be that we can enable sharding on the Mac AOT bots. Sharding is disabled because XCode is only reliably installed on the main step, not the shards, and IIRC XCode is needed for the assemble step because it pointlessly looks for some system library, which direct generation of Mach-O from the VM could skip.

Mar 10 '25 17:03 rmacnak-google

@zhangjianbin we can't split that file into smaller files because we need precise ordering of functions to be maintained.

We should just write our own Mach-O object file writer... It should be relatively straightforward.

cc ... @sstrickl Any takers?

Sure, I could work on this, especially since I've already had to look into the Mach-O object file format before for creating standalone executables.

/cc @athomas

Mar 12 '25 09:03 sstrickl

In my Flutter project, when generating the iOS framework, the command “usr/bin/arch -arm64e xcrun cc -v -arch arm64 -miphoneos-version-min=12.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS18.2.sdk -c /Users/xxx/Desktop/.dart_tool/flutter_build/2d151adfb8aeaa558257e5cae0b2158d/arm64/snapshot_assembly.S -o /Users/xxx/Desktop/.dart_tool/flutter_build/2d151adfb8aeaa558257e5cae0b2158d/arm64/snapshot_assembly.o” takes 40+ minutes to execute and there is no output from the console during the execution, can anyone tell me how to optimize the time spent on this process?

@yiiim @zhangjianbin

As a workaround we internally started building snapshot_assembly.S using xcode 15.4 xctoolchain. This was a blocker for migration to Xcode 16, as the build times of snapshot_assembly.S suddenly skyrocketed from ~30 to ~100 minutes (M2 Pro).

prepare toolchain:

cp -r /Applications/Xcode15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain ~/Library/Developer/Toolchains
cd ~/Library/Developer/Toolchains
mv XcodeDefault.xctoolchain Xcode15.4.xctoolchain
/usr/libexec/PlistBuddy -c "Add CompatibilityVersion integer 2" Xcode15.4.xctoolchain/ToolchainInfo.plist
/usr/libexec/PlistBuddy -c "Set Identifier clang.Xcode15.4" Xcode15.4.xctoolchain/ToolchainInfo.plist

update flutter_tools to use overriden toolchain: https://github.com/flutter/flutter/blob/fbe64c68d8a63a9837a9903c956435731e4182e8/packages/flutter_tools/lib/src/macos/xcode.dart#L225

  Future<RunResult> cc(List<String> args) => _run('--toolchain', <String>[
    'clang.Xcode15.4',
    'cc',
    ...args,
  ]);

Apr 01 '25 13:04 2ZeroSix

Thanks to @2ZeroSix for providing a workaround to the slow compilation issue with Xcode 16. I have built a wrapper based on that workaround, hoping it can help those in need.

Install condor_cli

dart pub global activate condor_cli

Copy xctoolchain

Note down the corresponding Xcode name, for example, if it is /Applications/Xcode-15.4.0.app, then use Xcode-15.4.0 for the following command.

condor optimize-build xctoolchain-copy --xcode Xcode-15.4.0

Make cc redirectable

# If using the default flutter, you don't need to pass the flutter parameter
condor optimize-build redirect-cc

# If you want to specify a Flutter version managed by fvm
# condor optimize-build redirect-cc --flutter fvm spawn 3.24.5

Set the environment variable CONDOR_TOOLCHAINS to the Xcode name mentioned above.

export CONDOR_TOOLCHAINS=Xcode-15.4.0

If the CONDOR_TOOLCHAINS is not set, the default cc will be used.

Apr 06 '25 13:04 LinXunFeng

This change dramatically reduces time spent in cc for darwin builds, it's barely visible in build timeline, toolchain hack is not required anymore on main branch https://github.com/dart-lang/sdk/commit/8505c558da004e747f6f0ef5c00ba722891320d6

I've applied it on top of 3.3.4 and our ios release builds were accelerated by ~30 minutes, profile by ~54 minutes (so without toolchain hack on xcode 16 it would be ~100 and ~130 minutes improvement)

@rmacnak-google thank you

Apr 23 '25 08:04 2ZeroSix

sdk sdk copied to clipboard

reduce end-to-end AOT compilation time for large applications

Install condor_cli

Copy xctoolchain

Make cc redirectable

sdk
sdk copied to clipboard