pytorch
[Feature Request] Support transforming observation noise in `Log` outcome transform
🚀 Feature Request
The Log outcome transform currently raises a NotImplementedError if it is used with known observation noise. This limits its usability. The transform should be updated to support observation noise.
Motivation
I'm interested in using this transform through Ax, where we may have both known and unknown observation noise.
What is the assumption we'd be making on the noise though? Typically we assume Gaussian noise on the data at the modeling level; but log-transforming observations under Gaussian noise means that the noise in the transformed space isn't Gaussian anymore. Would we interpret the variance as the variance of a log-Normal RV instead?
Side note: this issue is of course also present - albeit implicitly - in the case of unobserved noise levels that we infer as part of the likelihood. So we kind of are making that assumption already implicitly. I think it's ok to enable this for the Log transform but we should be explicit about it.
Gaussian noise is a common assumption we make throughout the code base, so I think it is ok to continue with that.
Sure, but do we make that assumption at the level of raw inputs or at the level or the transformed inputs?
If the former then making the Gaussian noise assumption at the transformed level would be wrong (it won't matter too much if the variance is small relative to the absolute outcome values, but could be way off if the metric is near zero).
If the latter then we'd essentially be making the assumption of log-normal observation noise on the raw inputs.
The former scenario could be quite problematic. The latter I think is fine, but we'd want to make that assumption explicit (e.g. in the docstring of the Log transform).