What we can get from rewriting pgrx-tests

[workingjubilee]

trafficstars

We could

• [ ] rewrite it as a custom test harness: https://github.com/tcdi/pgx/issues/534
• [x] make it respect attributes: https://github.com/tcdi/pgx/issues/599
• [ ] make it respect --no-default-features
• [x] make it https://github.com/tcdi/pgx/issues/324
• [ ] allow writing tests in places other than in-library unit tests in mod tests
• [ ] support Postgres-style regression testing https://github.com/tcdi/pgx/issues/284
• [ ] support https://github.com/tcdi/pgx/issues/525
• [ ] hopefully make it a lot faster by reducing the use of Mutex: https://github.com/tcdi/pgx/blob/1a89778046547c52c714fe6885069eab5638f965/pgx-tests/src/framework.rs#L27-L43

Of these, I am most concerned about --no-default-features:

• It is needed for providing correct dev- and user-experience that adheres to Rust convention.
• It blocks using pgx-tests to test features in PGX-dependent libraries

And making #[pg_test] fully respect attributes and actually work if you e.g. cfg out a test:

• Not doing so blocks actually interacting with other Rust code.
• It is an ergonomics impairment that costs hours each time I have run into it.

A major concern is to not regress existing diagnostics or execution time. Tests executed via pgx-tests are very slow-going by nature. Even if we recover speed from rewriting, I expect a lot of tests will in many cases still require entire seconds, even minutes, to complete. Every byte of data that can be extracted from the tests is precious, as people wait quite a long while for them. I somewhat recently had to significantly annotate the diagnostics just to be able to discern why things were failing consistently enough that I could fix them to make the beginnings of the postgrestd feature, and thus PL/Rust, happen.

[Hoverbear]

I think putting some attention to pgx-tests would help improve our ability to accomplish our own goals as well as provide users an improved experience. :)