Request: ADVI delta_ELBO_mean delta_ELBO_med to diagnostic_writer

[lseffer]

Summary:

Feature request: Add convergence metrics delta_ELBO_mean, delta_ELBO_med to ADVI diagnostic writer

Description:

My first issue here. Searched but found no issues specifically related to this.

I am fitting models to large nested data and I'm using ADVI. Meanfield algorithm is giving me good results when it converges. The problem I'm having is I need to automate the fitting, and sometimes the meanfield algorithm does not converge in the amount of iterations and wall time I set as the maximum.

It would be nice if the convergence conditions delta_ELBO_mean and delta_ELBO_med would be included in the output by the diagnostic writer here https://github.com/stan-dev/stan/blob/master/src/stan/variational/advi.hpp#L491 . Then it would allow for automatically checking that the convergence conditions have been met. Right now I have no choice but to re-create the convergence checks myself (in Python in this case). Another solution might be to redirect stdout (haven't tried) but that is also overly complicated in comparison to the feature request.

Possibly related (?) https://github.com/stan-dev/stan/issues/2474 .

[bob-carpenter]

Thanks for submitting an issue.

Do you plan to submit a pull request to fix this? If not, it might take a while. We're rebuilding a model base class, then refactoring the algorithms, then the services, then the interfaces. So maybe this'd get prioritized for after that, maybe 9 to 12 months unless someone knows how to do it easily. It can be done now and we'll carry over the work, but we're trying to avoid duplicating work going forward, even if it means delaying features.

[lseffer]

Thanks for the quick response. I can do a PR, hoping it's as simple as just adding the values to the writer.

[bob-carpenter]

I think some other folks are working on this, too, so it may take some coordination. I'm pinging @yao-yl and @bbbales2 who I think were involved in the discussion.

[yao-yl]

I thought delta_ELBO_mean and delta_ELBO_med have already been printed?

[lseffer]

@yao-yl They are printed to console, but not written anywhere (Pystan). This makes it hard and hacky to automate.

In PyStan the stanmodel.vb() method can produce a diagnostic csv with these columns iter,time_in_seconds,ELBO. After looking through the C++ writer code in PyStan I concluded that the missing metrics need to be added to the writer in advi.hpp.

For the record I have re-created the convergence checks in Python so this isn't such an alarming issue, but anyone wishing to automate an ADVI model needs to "reinvent" the wheel so to say.

[bob-carpenter]

Great. Makes sense to store them. We need to get more regular forms of output rather than making users scrape console output.