Comments on Structure Post

[ToddMorrill]

Here are a few reactions to your initial post in the series, "Structure." Feel free to disregard these comments if you think they detract from the article.

• Is it worth saying why you're conditioning on $\mathbf{x}_{0}$ in $q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)$? - Is it because there's too much uncertainty in $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t)$ without $\mathbf{x}_0$? This seems to hint at that. There may be a deeper reason why (as hinted at here), which would be illuminating if you can clearly explain the true reason (ideally algebraically).
• I think there is value in spelling out your setup of $q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)$. I don't know if it's as simple as, "The re-arranging to arrive at the last line above follows from Bayes' Rule." I started with the following

$$ q(\mathbf{x}_{1:T} \mid \mathbf{x}_0) = q(\mathbf{x}_1 \mid \mathbf{x}_0) \prod_{t=2}^T q(\mathbf{x}_t\mid \mathbf{x}_{t-1}, \mathbf{x}_0) $$

then used Bayes' rule on the terms in the product, manipulated a little, and the final step for me was to unroll the product with a few terms to notice all the cancellation that happens. Again, this video covers the derivation in full.

• A small stylistic choice might be to make the endpoints of your products and sums explicit, e.g. $$\prod_{t=2}^T \text{and} \sum_{t=1}^T$$
• You briefly introduce $\phi$ in your derivations of the ELBO. Might as well skip that.
• Right after you say "convenient form," I think the first line of math has some typos in the limits of integration (should be the integral over the domain, not the probability) and you might as well just skip the substitution of that first product after the integral since you switch to expectation notation right after (it's also inconsistent with your setup that always conditions on $\mathbf{x}_{0}$).

Thanks for putting this series together! Very nicely done and I'm looking forward to the next few posts. I think other diffusion beginners like me will get a lot out of your series.