[P4Testgen] Add a p4test-like target to testgen.

[vlstill]

@fruffy, I think you have mentioned this already at some point and I think this would make sense. Currently, all the testgen P4 testing happens on one of the targets which are actually not too simple (e.g. BMV2) and one might not actually want them build, especially if one is working on another testgen target. I think it would make sense to have a testgen target with a very minimal architecture and simple, testing-oriented output format that would be used basically just for testing and bug reproduction. Then we could have tests similar to the frontend/midend compiler tests that just record the expected outputs (from testgen in this case).

I think an architecture with minimal parser/control/deparser and no externs defined by the architecture would be in order for this. The output could be either something like STF, or (I thing this would be even better) a JSON-based format that would be easier to inspect on failures.

@fruffy, and others interested in testgen, what do you think about this? Is this something we want to pursue.

[fruffy]

That would definitely be nice. The biggest hurdle is patching together the package, which can be in arbitrary order. With https://github.com/p4lang/p4c/pull/4267 it should be easier, since we can just relax the arch specification and take the package as is for generating tests.

However, I have no plans to pursue this for now since I am working on a couple other rewrites.

#4295 is somewhat related to this idea since you could just pass in an arbitrary architecture specification.

[jnfoster]

Just to make sure I understand: the idea is to use p4testgen as a concrete interpreter? Then we can run tests natively without having to start up a separate software model.

This is obviously not useful for testing a real target, but could be useful for tracking bugs and such in the compiler.

If that's right, sounds cool!

[vlstill]

@jnfoster I'm not sure I understand your comment, so I will try to explain myself a bit more.

The idea is to have a testgen target that does not correspond to any real target, but is kind of a minimal example and it useful for testing. My original ideas were about testgen testing, but actually it could be very useful also for compiler testing (since the testgen uses a lot of the common passes).

My idea was to define a very simple arch (and a fixed one, so I was not thinking about having flexibility there as @fruffy's comment seems to suggest). Maybe something like this:

#include <core.p4>

struct metadata_t {
    bit<8> ingress_port;
    bit<8> egress_port;
    bit<1> dropped;
}

extern void assume(in bool check);
extern void assert(in bool check);

parser ParserT<H, M>(packet_in pkt, out H hdrs, out M md_user, inout metadata_t meta);
// I don't know where to put parser_error so I put it in the main control as an extra argument,
// similar to how PNA has it in intrinsic md_user in pre-control
control ControlT<H, M>(inout H hdrs, inout M md_user, inout metadata_t meta, in error parser_error);
control DeparserT<H, M>(packet_out pkt, in H hdr, in M md_user, inout metadata_t meta);

package TestPackage<H, M>(ParserT<H, M> p, ControlT<H, M> c, DeparserT<H, M> d);

And then a testgen target for this arch with a simple backend that just dumps the inputs and outputs (and trace) to some format (probably JSON).

I can see two use cases:

1. simple reproduction of core testgen bugs in a minimal format, making it (almost) certain these bugs are not in the specific target but in the core itself.
   • In this case, one would just write a P4 code and demostrate testgen fails on it.
2. testing testgen and the compiler
   • In this case, one would write a P4 program, let testgen generate the test cases and inspect them if the results are as expected. Then one would commit these as expected results.
   • It would be possible to use assume/assert to guide (and check) behavior on specific values.

[jnfoster]

Oh, I see. Yes, a simple, fixed instance of p4testgen could make a lot of sense.

[fruffy]

My idea was to define a very simple arch (and a fixed one, so I was not thinking about having flexibility there as @fruffy's comment seems to suggest). Maybe something like this:

Adding a fixed target can easily be done. We also already have a folder with custom includes here. We could just add a package definition there.

But this fixed target still demands a certain package structure, which could be constraining. A lot of the p4test test files in testdata have arbitrary package structure, for example.

I could imagine we could define something like --target p4test --arch testgen_simple or --target p4test --arch testgen_multipipe to work around this and test different kinds of architectures.

[vlstill]

But this fixed target still demands a certain package structure, which could be constraining. A lot of the p4test test files in testdata have arbitrary package structure, for example.

I see, that is something I did not realize at first. Still this would be useful even with a single architecture.

I could imagine we could define something like --target p4test --arch testgen_simple or --target p4test --arch testgen_multipipe to work around this and test different kinds of architectures.

I indent to start small and we can make this into a more complex facility later (even include the architecture spec #4295).

[fruffy]

Now that the library extension is in this should be much easier to prototype.