Discussion: Correctness Checking Between PPLs

[JasonPekos]

Rationale:

It would be handy to have some sort of infrastructure for ensuring model implementations in different PPLs are equivalent.

Idea:

Evaluate the log-density over the reference draws, to see if the implementations line up. Intuitively, this may be possible with, e.g.

• BridgeStan for Stan
• logjoint in DynamicPPL for Turing and so on.

Problems:

• Stan seems to only allow log density calculations on unconstrained space. For the log posteriors to be the same, the PPLs need to be using the same bijections, both for this, as well as for things like ordered transforms.
• For more complicated models, factors out of our control --- like slightly different results from diffeq solvers --- could make the results no longer line up. This should not impact most models.
• Not all models have reference draws, but in this case we could use something like "evaluation draws". doesn't seem like a big issue.