[JIT] CodeGen verification testing

[TIHan]

Description

@markples and I have been working hard on having the ability to write tests that verify code-gen for X64 and ARM64. Our goal is to make writing these tests easy as possible and be able to incorporate them with already existing tests.

The tool we are using for verifying code-gen is LLVM's FileCheck. We created a new NuGet package in dotnet/llvm-project called Microsoft.NETCore.Runtime.JIT.Tools that contains LLVM's FileCheck - we consume this package now. See https://llvm.org/docs/CommandGuide/FileCheck.html for more information on how FileCheck is used.

We have also created a tool called, SuperFileCheck, that wraps FileCheck and uses Roslyn to allow us to write tests more easily and look readable in C#. For example:

    [MethodImpl(MethodImplOptions.NoInlining)]
    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
    {
        // ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0
        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
    }

The test above verifies the method call, AdvSimd.CompareEqual, will emit the correct ARM64 instruction, cmeq with two operands and the zero. <prefix>-FULL-LINE: <instr> is a new construct in SuperFileCheck that translates to FileCheck 's syntax, <prefix>: {{* ^}}<instr>{{$}} - this allows to match a single line exactly without having to use {{* ^}} or {{$}}.

Things to consider:

• DOTNET_JitDisasm will need to be a bit more reliable in its output as we create more of these kinds of tests - especially start and end anchors for methods. I think this is fine considering we are making DOTNET_JitDisasm public in release versions.

Acceptance Criteria

• [x] CI disasmchecks are run and passing
• [x] Write documentation
• [x] Write a few more tests
• [x] Cleanup

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

@markples and I have been working hard on having the ability to write tests that verify code-gen for X64 and ARM64. Our goal is to make writing these tests easy as possible and be able to incorporate them with already existing tests.

The tool we are using for verifying code-gen is LLVM's FileCheck. We created a new NuGet package in dotnet/llvm-project called Microsoft.NETCore.Runtime.JIT.Tools that contains LLVM's FileCheck - we consume this package now. See https://llvm.org/docs/CommandGuide/FileCheck.html for more information on how FileCheck is used.

We have also created a tool called, SuperFileCheck, that wraps FileCheck and uses Roslyn to allow us to write tests more easily and look readable in C#. For example:

    [MethodImpl(MethodImplOptions.NoInlining)]
    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
    {
        // ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0
        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
    }

The test above verifies the method call, AdvSimd.CompareEqual, will emit the correct ARM64 instruction, cmeq with two operands and the zero. <prefix>-FULL-LINE: <instr> is a new construct in SuperFileCheck that translates to FileCheck 's syntax, <prefix>: {{* ^}}<instr>{{$}} - this allows to match a single line exactly without having to use {{* ^}} or {{$}}.

Things to consider:

• DOTNET_JitDisasm will need to be a bit more reliable in its output as we create more of these kinds of tests - especially start and end anchors for methods. I think this is fine considering we are making DOTNET_JitDisasm public in release versions.

Acceptance Criteria

• [ ] CI disasmchecks are run and passing
• [ ] Write documentation
• [ ] Write a few more tests
• [ ] Cleanup

Author:	TIHan
Assignees:	TIHan
Labels:	area-CodeGen-coreclr
Milestone:	-

[BruceForstall]

Random note: should the syntax for things that will specify checking use a unique, searchable, prefix that makes it super clear it's for FileCheck, e.g.:

// ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0
=>
// FILE-CHECK-ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0

?

[TIHan]

should the syntax for things that will specify checking use a unique, searchable, prefix that makes it super clear it's for FileCheck, e.g.:

It's fair to ask, but I think it's fine the way it is given that the FileCheck syntax is pretty known especially within LLVM. You also have to modify the test .csproj to be explicit that you want to use FileCheck on a single file; that at least gives some explicitness to using the tool.

[TIHan]

@dotnet/jit-contrib This is ready for review.

[trylek]

There seemed to be a general naming convention that the csproj file for each test could specify CLRTestXXX and then they would get transformed here (if needed) into something not starting with CLRTest, mostly by moving the CLR later in the name.

I see, I didn't realize that CLRTestBatchPostCommands is supposed to come from the individual test projects. Frankly speaking I haven't found a single test using this property and there's apparently no equivalent Unix version CLRTestBashPostCommands either, perhaps the easiest way to clean this up would be by deleting the unused property.

[TIHan]

Current CI failures look unrelated to these changes.

[TIHan]

Current CI failures are from: https://github.com/dotnet/runtime/issues/76041

There are no failures related to this PR - merging this in.