dowhy
dowhy copied to clipboard
py-why
Improvement documentation | Refutation results
I am missing a piece of documentation that summarizes how the results of a refutation procedure should be interpretet.
Assuming there is a causal effect, what does a significant p-value of a specific procedure (e.g., random common cause) mean?
I would prefer a table like the following one:
- Refutation Method: e.g., "random common cause"
- Short description: e.g., "introduces a random common cause to test the robustness of a model". At best, there are links to scientific studies related to this approach.
- Interpretation: e.g., significant p-value increase the robustness of the estimation. A non-significant p-value reduces the robustness of the estimation.
Something along these lines would be great for interpreting of the results.
Agreed, @Klesel . This page would be a good place in the docs to add that: https://www.pywhy.org/dowhy/v0.9.1/user_guide/effect_inference/refute.html
I would like to consider using the same H0 for all refutation tests:
H0: Estimated effect is equal to new effect
Thus, adding a random common cause should be significant in case of a robust model.
In contrast, when using a dummy outcome, the p-value should not be significant in the case of a robust model.
I agree this is very much needed. I think it would also be nice to have some literature references where available and links to other methods which are similar. There is a treasure trove of 'refuters' that might be under a different name!
As a side note: Would we consider common robustness tests like the Hausman test a refuter?
I like this idea a lot. Let me start a PR with a common template and we can all edit the docs to add more info.
@padarn are you referring to the Durbin-wu-hausman test? Yes, that can definitely be a refuter.
Right @amit-sharma. Do you think we should add some of these. Might be worth taking some inspiration from some of the econometrics focused packages like statsmodels.
yes, adding them will be great. would you like to start a PR for this, perhaps by using statsmodels? We already have statsmodels as a dependency.
Sure sounds good!
On Mon, 13 Feb 2023 at 6:45 PM, Amit Sharma @.***> wrote:
yes, adding them will be great. would you like to start a PR for this, perhaps by using statsmodels? We already have statsmodels as a dependency.
— Reply to this email directly, view it on GitHub https://github.com/py-why/dowhy/issues/847#issuecomment-1427720875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGRPN7NSMVJRWEES4ZNHX3WXIGF5ANCNFSM6AAAAAAUSYB5DI . You are receiving this because you were mentioned.Message ID: @.***>
--
By communicating with Grab Holdings Limited and/or its subsidiaries, associate companies and jointly controlled entities (collectively, “Grab”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/ https://grab.com/privacy/
This email contains confidential information that may be privileged and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email. Please notify Grab immediately if you have received this by mistake and delete this email from your system. Email transmission may not be secure or error-free as any information could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab does not accept liability for any errors or omissions in this email that arise as a result of email transmission. All intellectual property rights in this email and any attachments shall remain vested in Grab, unless otherwise provided by law
FYI I looked into this the statsmodels tests (at least for Hausman) are really specifically embedded into the IV estimator. @amit-sharma so you have a suggestion for how this might make as a general refuter? Maybe more generally, do we already have a 'estimator specific' refuter concept?
Agree, the Hausman test would make sense as an estimator-specific refuter. We are trying to add support for estimator-specific refuters. The general idea is that estimators should be able to specify the refuters specific to them, so that the downstream refute_estimate can automatically run all common refuters and the specific ones. At the same time, each etimator-specific refuter should check for the estimator class, and if it is not supported, raise an Exception.
So I would suggest you can go ahead with creating the refuter. In the function, you can check the Estimator class to be IV. If it is not, you can raise a ValueError (note that we've refactored refuters as functions now, see the placebo_treatment_refuter.py file as an example)
That makes sense.
We are trying to add support for estimator-specific refuters. The general idea is that estimators should be able to specify the refuters specific to them, so that the downstream refute_estimate can automatically run all common refuters and the specific ones. At the same time, each etimator-specific refuter should check for the estimator class, and if it is not supported, raise an Exception
That sounds good. Do you have any PRs open on this? Or is it something I can find in discord?
There are no PRs yet, just something that we have discussed in the PyWhy meetings. @emrekiciman do you know if we already have this on Discord?
Hi all. I would he happy to contribute to the docu, if I can. I actually tried to answer this question myself, in note form here: https://github.com/py-why/dowhy/issues/929 If this is correct, then at least I understand it enough to write up! Otherwise, keen to understand my error.
