iree-org
Embed executable sources in vmfb files for portable Tracy captures
We can view executable sources for dispatch functions in Tracy thanks to https://github.com/openxla/iree/pull/9994, https://github.com/openxla/iree/pull/13333, and https://github.com/openxla/iree/pull/13500.
-
The definition of "executable sources" in a multi-level compiler can have multiple values. We use whatever is snapshot into MLIR source locations, which practically equals the input .mlir locations (e.g. Python files / lines) by default and linalg sources prior to codegen when
--iree-hal-dump-executable-sources-to={path}is set. (You could also snapshot PTX, spirv, LLVM dialect MLIR, x86 assembly, etc.)
-
Source locations are stored as absolute file paths that are not portable across machines right now. This means that you can't view sources on platforms like Android without manually editing the .vmfb and copying files across.
Tracy 0.9 (https://github.com/wolfpld/tracy/releases/tag/v0.9) added this feature:
You can now provide custom source file contents through a profiler callback. This enables profiler access to files contained in your custom data files, etc.
We can use that callback together with embedded sources to support viewing sources on Android and other non-host platforms.
- Prior discussion about this work on Discord here.
- Work-in-progress patch here: https://github.com/ScottTodd/iree/commit/deb4956d78460f25c63c7cc449c68b78abc819d1 . Note the comments from the discussion:
we'll want to keep the loc and instead have an additional field for files - that way we can have multiple entry points that reference lines in the same file (N:M) then we'd just walk src_files and register them all and let the existing src_locs reference them by file name will be important when linking multiple source files together and such
cc @KoolJBlack @dcaballe (sounded like there was some interest in improving Tracy workflows on Android, this could help and would be 1-3 days of work I think)
This source snapshotting also helps connect locations with sampled assembly:
With --iree-hal-dump-executable-sources-to={path}:
Without:
Nice! Would be cool if we could make embedded elfs work with this too - not sure tracy has enough API hooks for us to do that though.
Yeah. With embedded (--iree-llvmcpu-link-embedded=true), I don't see any sampling or assembly for executable code:
I do at least see the instrumented source though.
To clarify (since I just confused myself about this):
- If you compile on machine A and run on machine A, the .tracy file will be portable to machine B (including C/C++ and MLIR sources)
- If you compile on machine A and run on machine B, the .tracy file will not be portable to machine C (without a fix for this)
The other element with this issue is decoupling the "I want to dump executable files to disk" workflow from the "I want debug information in my .vmfb so I get useful traces" workflow.
@benvanik or I might help land a fix for this, as reliable and easy profiling across different systems is high priority.