flutter.engine: init

[RossComputerGuy]

Description of changes

Fixes #201574

This PR is mostly working, only thing to resolve now is a ninja issue. However, I believe this PR is just about ready for review and this PR has everything needed to make building the Flutter engine possible.

Things done

• Built on platform(s)
  • [x] x86_64-linux
  • [x] aarch64-linux
  • [ ] x86_64-darwin
  • [ ] aarch64-darwin
• [ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• [ ] Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• [x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• [ ] Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • [ ] (Package updates) Added a release notes entry if the change is major or breaking
  • [ ] (Module updates) Added a release notes entry if the change is significant
  • [ ] (Module addition) Added a release notes entry if adding a new NixOS module
  • [ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• [ ] Fits CONTRIBUTING.md.

[RossComputerGuy]

Flutter engine now starts ninja!

[RossComputerGuy]

flutter-engine-jit_release> Done. Made 953 targets from 324 files in 520ms
flutter-engine-jit_release> building
flutter-engine-jit_release> Building flatc
flutter-engine-jit_release> [88/88] LINK ./flatc_party/libcxx/libcxx.stamp.o_rust.o[KooK[K[KKK
flutter-engine-jit_release> Building blobcat
flutter-engine-jit_release> [565/565] LINK ./blobcat/impeller/blobcat/blobcat_lib.stampder.oter.o[K[Kble.o[Kfee25/src/third_party/icu/common/icudtl.dat icudtl.dat
flutter-engine-jit_release> Building gen_snapshot
flutter-engine-jit_release> [514/514] LINK ./gen_snapshott/runtime/libdart_precompiler.ailer.stampKger.ook.oer_x86.o[Klper.o
flutter-engine-jit_release> Building impellerc
flutter-engine-jit_release> [1415/1415] LINK ./impellercmpeller/compiler/compiler_lib.stampmpmp_val.validate_builtins.oes.oK.os.o.oinux:clang_x64)ang_x64)
flutter-engine-jit_release> Building Flutter Engine library
flutter-engine-jit_release> [113/1313] ACTION //flutter/impeller/entity:gen_blob_embed_gles_entity_shaders(//build/toolchain/linux:clang_x64)
flutter-engine-jit_release> FAILED: gen/flutter/impeller/entity/gles/entity_shaders_gles.h gen/flutter/impeller/entity/gles/entity_shaders_gles.c
flutter-engine-jit_release> python3 ../../../../../../../build/flutter-engine-src-857bd6b74c5eb56151bfafe91e7fa6a82b6fee25/src/flutter/impeller/tools/xxd.py --symbol-name entity_shaders_gles --output-header /nix/store/k987lqbgl09glaf3xfrckbb8ybpahjsh-flutter-engine-jit_release-857bd6b/lib/flutter/out/jit_release/gen/flutter/impeller/entity/gles/entity_shaders_gles.h --output-source /nix/store/k987lqbgl09glaf3xfrckbb8ybpahjsh-flutter-engine-jit_release-857bd6b/lib/flutter/out/jit_release/gen/flutter/impeller/entity/gles/entity_shaders_gles.c --source /nix/store/k987lqbgl09glaf3xfrckbb8ybpahjsh-flutter-engine-jit_release-857bd6b/lib/flutter/out/jit_release/gen/flutter/impeller/entity/genlib_gles_entity_shaders.shaderblob
flutter-engine-jit_release> Traceback (most recent call last):
flutter-engine-jit_release>   File "/nix/store/k987lqbgl09glaf3xfrckbb8ybpahjsh-flutter-engine-jit_release-857bd6b/lib/flutter/jit_release/../../../../../../../build/flutter-engine-src-857bd6b74c5eb56151bfafe91e7fa6a82b6fee25/src/flutter/impeller/tools/xxd.py", line 95, in <module>
flutter-engine-jit_release>     main()
flutter-engine-jit_release>   File "/nix/store/k987lqbgl09glaf3xfrckbb8ybpahjsh-flutter-engine-jit_release-857bd6b/lib/flutter/jit_release/../../../../../../../build/flutter-engine-src-857bd6b74c5eb56151bfafe91e7fa6a82b6fee25/src/flutter/impeller/tools/xxd.py", line 51, in main
flutter-engine-jit_release>     assert os.path.exists(args.source)
flutter-engine-jit_release> AssertionError
flutter-engine-jit_release> [124/1313] CXX obj/flutter/impeller/renderer/backend/gles/gles.pipeline_library_gles.o
flutter-engine-jit_release> ninja: build stopped: subcommand failed.

Mostly working, we just need to fix this issue now. I checked the assert and it looks like renaming the stuff in configurePhase breaks stuff. I'll fix that and we should have the engine working.

[RossComputerGuy]

Commits have been reordered and squashed, it's now tidy.

[RossComputerGuy]

Working on adding the GTK library to be built, I had to use touch on all files because it tries zipping the files and there's a timestamp error when touch isn't used.

[RossComputerGuy]

Got some improvements for cross-compiling. Should be possible to build on aarch64-linux now. We can also produce builds for WASM. All --target-os values are supported except for Fuchsia. Just have a few bugs to fix before I push.

[RossComputerGuy]

It's probably not entirely working correctly but it almost is. I've gtg but I'll be back in like an hour to continue. If there's any fixes @milahu can work on while I'm gone, please do and comment back the fixes.

[RossComputerGuy]

I'm back and I've figured it out. I'll have the fixes out in a little bit. Some dependencies have common ones so we just gotta symlink them.

[RossComputerGuy]

Engine now builds (again) but we're using our own toolchain now and the better cross-compiling setup has been added in.

[RossComputerGuy]

I just refactored a bunch of stuff so it's possible to use custom versions and have multiple versions of the engine in the repos. flutter-engine just links to flutter-engineVersions.latest which is the latest stable version we should use. This will be useful for programs which need different engine versions.

[RossComputerGuy]

Cross compiling should work better now

[RossComputerGuy]

Haven't found a solution to fixing the missing linked libraries

$ ldd result/lib/flutter/out/debug/libflutter_engine.so
	linux-vdso.so.1 (0x00007ffc45f23000)
	libm.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libm.so.6 (0x00007fec9fdab000)
	libc.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libc.so.6 (0x00007fec9fba2000)
	/nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007feca2136000)
$ ldd result/lib/flutter/out/debug/libflutter_linux_gtk.so
	linux-vdso.so.1 (0x00007ffc2c3e6000)
	libgtk-3.so.0 => not found
	libgdk-3.so.0 => not found
	libatk-1.0.so.0 => not found
	libgio-2.0.so.0 => not found
	libgobject-2.0.so.0 => not found
	libglib-2.0.so.0 => not found
	libepoxy.so.0 => not found
	libm.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libm.so.6 (0x00007f15dcc2c000)
	libc.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libc.so.6 (0x00007f15dca23000)
	/nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f15deffc000)

[RossComputerGuy]

I've set platforms to have only x86_64-linux since CIPD is missing dependencies we need for other platforms. Until we can find a workaround, flutter-engine will only support that.

[RossComputerGuy]

Just fixed the pkg-config files in the main package.

[algorithmiker]

I am not sure, but I think this, once merged, might deserve an own changelog entry. Do others agree?

[milahu]

changelog? just add nix comments when doing something unexpected

im more worried about the use of binaries and FHS. ideally we use clang_15 from nixpkgs, compile missing packages like cipd, and patch FHS paths. ideally...

[RossComputerGuy]

Ok, the engine now works with the Flutter tool.

[RossComputerGuy]

I might refactor this to use FHS soon. I just need to find the time.

[RossComputerGuy]

Ok, I have finished rewriting it to use FHS and it builds correctly on my setup. Just need to verify with ofborg. Would be neat to see this merged soon.

[RossComputerGuy]

Rebased and squashed some of the commits to simplify the log.

[algorithmiker]

I agreed that this would be neat. is depot_tools a package set?

[algorithmiker]

How is this going to be integrated to our flutter package btw? (soon to be flutter maintainer here)

[RossComputerGuy]

I agreed that this would be neat. is depot_tools a package set?

depot_tools is a special Git repo of a collection of config files, scripts, and other things which allows for downloading source code from Google's own infrastructure and stuff. It's not really a package set but it also kinda is like one.

How is this going to be integrated to our flutter package btw?

The engine ships with a few executables which can be used independently of the tool. However, there's 2 ways of using custom engine builds with the tool.

1. Specify the source and built engine through environmental variables
2. Specify the source and built engine through command line arguments

I was thinking we could add a nullable attribute to the package creation method for the Flutter tool. If it is null, then the tool will default to downloading the engine like normal. If it is not null, that must be the engine derivation. Then we could use ${engine}/lib/flutter-engine/out/${engine.targetDir} and ${engine.src} for the values we pass as environmental variables and the tool should just work.

[RossComputerGuy]

I've merged depot_tools into the package.

[RossComputerGuy]

Fixed the reproducibility issue with the source fully now. Hopefully we can get this merged soon.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/2069

[RossComputerGuy]

Still trying to fix the reproducibility issue of the engine source code so I'm not quite yet ready to merge this yet.

[RossComputerGuy]

Still not reproducible source for some strange reason, I've tried this outside of Nix and directly on my system. Ran nix hash path --type sha256 and got the same hash, diff recursively also shows 0 changes.

[MGlolenstine]

@RossComputerGuy Would Diffoscope help in this case? It got recommended when I tried to figure out the difference between two derivations, as I couldn't figure out the differences.

[RossComputerGuy]

@RossComputerGuy Would Diffoscope help in this case? It got recommended when I tried to figure out the difference between two derivations, as I couldn't figure out the differences.

I tried it but it took forever and about a third of the way in, it crashed.

[hacker1024]

@RossComputerGuy Would Diffoscope help in this case? It got recommended when I tried to figure out the difference between two derivations, as I couldn't figure out the differences.

I tried it but it took forever and about a third of the way in, it crashed.

Perhaps you can turn off some of the analysis. ELF diffs can be quite large.