dowhy icon indicating copy to clipboard operation
dowhy copied to clipboard

Increase the default number of simulations

Open Klesel opened this issue 2 years ago • 1 comments

I would like to suggest changing the default values for the number of simulations:

Currently, the number of simulations for the Confidence Intervals is set to 100:

    # The default number of simulations for statistical testing
    DEFAULT_NUMBER_OF_SIMULATIONS_STAT_TEST = 1000
    # The default number of simulations to obtain confidence intervals
    DEFAULT_NUMBER_OF_SIMULATIONS_CI = 100
    # The portion of the total size that should be taken each time to find the confidence intervals
    # 1 is the recommended value
    # https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf
    # https://projecteuclid.org/download/pdf_1/euclid.ss/1032280214

This is too little. I suggest to use at least 399 instead. IMHO, 999 would be even better and preferable as default.

This paper can be used as reference. https://www.econstor.eu/bitstream/10419/67820/1/587473266.pdf

I already open another issue: https://github.com/py-why/dowhy/issues/809 relates to the same part of the code. I would be probably most convenient to change both.

Klesel avatar Jan 30 '23 18:01 Klesel

Good point, and thanks for providing the reference. It was set to 100 primarily to avoid computational burden for a new user. Let's set it to 399. Feel free to open a PR (do also add the paper link as an inline comment to help future readers of the code).

amit-sharma avatar Jan 31 '23 11:01 amit-sharma