How to reduce test cases down (llvm-reduce support for wasm object files?)

[juj]

trafficstars

I am looking at reducing a large wasm test case down (hundreds of .o files, see https://github.com/llvm/llvm-project/issues/55111 ) to the smallest repro part.

If the test case is LLVM IR, I can use llvm-reduce. But if the test case is Wasm object files, then llvm-reduce cannot be applied.

Is there a way to reduce down test cases consisting of wasm object files?

Slightly related, if I have two or more wasm object files, is there a command to merge those all to a single .o file? I find a mention of wasm-merge, and using ld.lld to merge, but not sure if either are applicable today? @tlively can you recall?

[kripken]

Sounds like the testcase is part LTO (LLVM IR in object files) and part non-LTO (wasm object files containing wasm)?

Unfortunately, for the non-LTO part, we don't have a good solution for reduction atm. Binaryen's wasm-reduce won't work since it doesn't have support for relocations in wasm object files

wasm-merge was a tool we had once in Binaryen, but it did not handle relocations either.

I believe wasm-ld can do a "partial link" (not sure of the right term) to link object files to a single big object file, however. That would at least help with the number of files. cc @sbc100 for details and maybe other ideas?

Can you use LTO for all that build and still see the issue, perhaps? Or does it only happen when some files are non-LTO?

[juj]

Sounds like the testcase is part LTO (LLVM IR in object files) and part non-LTO (wasm object files containing wasm)?

That is correct. I was able to llvm-reduce the LLVM IR part, but haven't been able to reduce the wasm object file part of the input.

Unfortunately, for the non-LTO part, we don't have a good solution for reduction atm. Binaryen's wasm-reduce won't work since it doesn't have support for relocations in wasm object files

Oh interesting, I was actually not aware of the wasm-reduce tool. By relocations here are you referring to -fPIC command line setting? We should not be using -fPIC iirc, so I think I could give wasm-reduce a try.

Can you use LTO for all that build and still see the issue, perhaps? Or does it only happen when some files are non-LTO?

That's the question I have been trying to resolve atm. What makes this tricky is that the code comes from multiple build systems built at different times, so getting all code to build as LLVM bitcode is a bit tricky. Though I am working on this.

[sbc100]

wasm-ld can combine object files using -r/--relocatabe.

Sadly there is no support in binaryen for wasm object files (wasm files with relocations) so binaryen is probably not going to a tool that can help you here.

If you are debugging a crash during LTO you should be able reproduce it by pass -save-temp to the linker and have it write out the bitcode file that cases that crash (i.e. the LTO input file, which I think this s single bitcode file).. you should be able reproduce the crash using llc on the that single bitcode file, and then use llvm-reduce to make it smaller.

[sbc100]

BTW if the crash is in the bitcode compiler then wasm object files should be not relevant. The bitcode compile just takes as input bitcode/LTO files.

[juj]

Thanks @sbc100 - at least the wasm object files are required in the repro, otherwise the crash does not occur. Not sure if that is to do with the build failing in missing symbols if the wasm object files are not specified.

for wasm object files (wasm files with relocations)

Oh gotcha, wasm object files all have relocations enabled.. that of course makes sense.

If I had a tool to strip the bodies of individual functions by name (e.g. wasm-drop-function-body a.o --function=_foo --function=_bar or similar) , I think that might be a good first step - I think I'd then be able to whip up a python based engine that would then exhaustively reduce function bodies from wasm object files until only minimal number of nonempty functions remain.

[sbc100]

The only inputs the bitcode compiler are bitcode files, so whenever there is a crash in the bitcode compiler there should be set of pure-bitcode files that can reproduce it. The only way the wasm object files could trigger a crash in the compiler would be by the by the dependecies that they indirectly introduce. I.e. they can add dependencies based on the symbols they need.

My understanding is that it should always be possible to reproduce a wasm-ld crash that occurs in the bitcode compiler, without using wasm-ld but taking the combined bitcode file and using llc on the that directly. In other words it should always be possible to come up with pure-bitcode reproducer. But its been a while since I've had to do it so my memory of exactly how that to do it is a little rusty.

IIRC, the key is having wasm-ld spit out the combined bitcode file and trying to reproduce the crash in llc directly.

[juj]

My understanding is that it should always be possible to reproduce a wasm-ld crash that occurs in the bitcode compiler, without using wasm-ld but taking the combined bitcode file and using llc on the that directly. In other words it should always be possible to come up with pure-bitcode reproducer.

That makes sense, though unfortunately I haven't been able to repro the crash on just the bitcode.

The only way the wasm object files could trigger a crash in the compiler would be by the by the dependecies that they indirectly introduce

I presume that is the key here. If I try to drop any of the wasm object files, I don't get the crash repro anymore, but instead I get a bunch of missing symbols errors, so I presume the wasm object files get scanned before running the bitcode compiler, and that affects what gets run on the bc content(?)

When using llvm-reduce on this, I have actually observed *four* different crash call stacks now in LLVM that ask "PLEASE submit a bug report". LLVM crashes in different ways:

1. LLVM crashes in llvm::PatternMatch::match: https://github.com/llvm/llvm-project/issues/55111#issuecomment-1110237667
2. LLVM crashes in ::classof: https://github.com/llvm/llvm-project/issues/55111#issuecomment-1127761286
3. LLVM crashes in ::getValueID: https://github.com/llvm/llvm-project/issues/55111#issuecomment-1127788302
4. LLVM crashes in LLVM ERROR: only one .init_array section fragment supported: https://github.com/llvm/llvm-project/issues/55504

So it looks like this set of input files is a ripe ground for finding new LLVM 15 git branch regressions (LLVM 14 compiles the codebase ok) - having the ability to reduce the wasm side of this for developers like @fhahn to be able to focus diagnosing their core area would be beneficial.