Extract compiler test cases to individual Pony files

[plietar]

The compiler test suite in test/libponyc forms a nice corpus of both valid and invalid sample Pony programs. However, the Pony code snippets are contained inside C files, making them impractical to reuse for other purposes.

My suggestion is to move to a layout where each test case is a separate .pony file. The file can have annotations at the top specifying specific features about the program that need to be tested.

For example this test case

TEST_F(SubTypeTest, IsTypeparamSubIntersect)
{
  const char* src =
    "class B\n"
    "  fun f[A: (I32 & Real[A])](a: A, b: (I32 & Real[A])) =>\n"
    "    None";

  TEST_COMPILE(src);

  pass_opt_t opt;
  pass_opt_init(&opt);

  ASSERT_TRUE(is_subtype(type_of("a"), type_of("b"), NULL, &opt));
  ASSERT_TRUE(is_subtype(type_of("b"), type_of("a"), NULL, &opt));

  pass_opt_init(&opt);
}

would become something like (strawman syntax) :

// compile-pass
// subtype(type_of(a), type_of(b))
// subtype(type_of(b), type_of(a))
class B
  fun f[A: (I32 & Real[A])](a: A, b: (I32 & Real[A])) =>
     None

This is similar to how the test cases for the Rust compiler are organised.

I'll need to extract these programs anyway for my project, and I'm happy to work on the test runner to integrate with the existing test infrastructure.

[sylvanc]

I like this. I don't think we need an RFC, as this doesn't impact users, just compiler hackers. I marked this as under discussion rather than ready for work to show we can have a conversation about the syntax for what should be tested and how, but if you want to start on this with the strawman syntax, go for it!

@Praetonus has been working on the tests recently (particularly JIT compiling) so he may have some good ideas.

[Praetonus]

This is an interesting problem. A good starting point would be to use package_add_magic_path to compile arbitrary packages in the tests suite.

[jemc]

Something to keep in mind while thinking about this:

A while ago I introduced the TEST_ERRORS_N series of macros (where N is a number of expected), so that we could test for specific error messages that were expected. This helps to prevent situations where a TEST_ERROR case is failing to compile for the wrong reasons, but we don't notice since the test still passes for any error. I've been meaning to go through all the cases of TEST_ERROR and replace them with TEST_ERRORS_N, so that our tests are more specific and robust.

So our new system should account for that way of testing for errors, and not just the TEST_ERROR way of doing it.