Do we want to make a version of two-argument categorical that collapses out the choice?

[axch]

For example, here's a way to make a 2-D clustering model with a fixed number of clusters:

[assume c1_mu (mvn (array 0 0) (id_matrix 2))]
[assume c1_sig (inv_wishart ...)]
c2, c3 likewise
[assume weights (dirichlet ...)]
[assume point (make_c_categorical weights
  (list (lambda () (mvn c1_mu c1_sig))
        (lambda () (mvn c2_mu c2_sig))
        (lambda () (mvn c3_mu c3_sig))))]

Here observe (point) (vector foo bar) is expected to collapse out the cluster choice (which are independent given the cluster parameters). This is analogous to what is expected in Stan (HMC can now be applied to the weights and the cluster parameters).

The more typical (to date) way to do Gaussian clustering in Venture is to collapse out the cluster parameters and the weights, and leave the assignments uncollapsed:

[assume c1 (make_cmvn ...)]
[assume c2 (make_cmvn ...)]
[assume c3 (make_cmvn ...)]
[assume cluster (make_dir_cat ... (list c1 c2 c3))]
[assume point ((cluster))]

The inferential difference for local M-H type inference programs is that in the former, it will take O(3*#points) work to evaluate a proposal to the weights or the cluster parameters, but on the other hand, there are only O(3) variables to do inference on. In the latter, by contrast, there are O(#points) variables to infer, but evaluating a proposal to one takes only O(3) work.

Also

• HMC applies to the former model but not the latter.
• Enumeration applies to the latter model but not the former.
• Rejection should apply to the former; trying it on the latter produces "Can't do rejection sampling when observing resimulation of unknown code", but maybe that should be adjusted in light of the operator SPs in question all being instances of the same class (see #472).
• Assessing the predictive distribution of the former model is naturally deterministic; trying to get the obvious deterministic assessment of the one-point predictive in the latter model is kind of a pain (which suggests that we should also have a way to make a model that acts like the latter for training and like the former for prediction; and/or explicitly maintaining completed enumerations in the trace (not just for enumerative Gibbs) whence to compute predictive probabilities).

In any case, I think it would be nice to be able to implement the former collapsing style. This seems to require the made SP from make_c_categorical to assess by querying the assessments of its constituents, which seems nontrivial.

Can this be further collapsed by integrating out the weights too, or not?

[axch]

The project management decision is whether/when to do this. This also calls for design, and possibly math or planning.