causaltune icon indicating copy to clipboard operation
causaltune copied to clipboard

SimpleParamService doesn't filter estimator list by problem if dataset large and "all" estimators requested

Open TimoFlesch opened this issue 2 years ago • 0 comments

The following code block is intended to filter the list of all implemented estimators by the problem (backdoor or iv). It doesn't do this if the dataset exceeds 1000 rows:

        if patterns == "all":
            if data_rows <= 1000:
                return [e for e in self.estimator_names if problem_match(e, problem)]
            else:
                warnings.warn(
                    "Excluding OrthoForests as they can have problems with large datasets"
                )
                return [e for e in self.estimator_names if "OrthoForest" not in e]

As example, compare the following output: 1/ ac.cfg.estimator_names_from_patterns("backdoor","all",100) yields:

['backdoor.auto_causality.models.Dummy',
 'backdoor.auto_causality.models.NewDummy',
 'backdoor.econml.metalearners.SLearner',
 'backdoor.econml.metalearners.TLearner',
 'backdoor.econml.metalearners.XLearner',
 'backdoor.econml.metalearners.DomainAdaptationLearner',
 'backdoor.econml.dr.ForestDRLearner',
 'backdoor.econml.dr.LinearDRLearner',
 'backdoor.econml.dr.SparseLinearDRLearner',
 'backdoor.econml.dml.LinearDML',
 'backdoor.econml.dml.SparseLinearDML',
 'backdoor.econml.dml.CausalForestDML',
 'backdoor.auto_causality.models.TransformedOutcome',
 'backdoor.econml.orf.DROrthoForest',
 'backdoor.econml.orf.DMLOrthoForest']

2/ ac.cfg.estimator_names_from_patterns("backdoor","all",1001) yields:

['backdoor.auto_causality.models.Dummy',
 'backdoor.auto_causality.models.NewDummy',
 'backdoor.econml.metalearners.SLearner',
 'backdoor.econml.metalearners.TLearner',
 'backdoor.econml.metalearners.XLearner',
 'backdoor.econml.metalearners.DomainAdaptationLearner',
 'backdoor.econml.dr.ForestDRLearner',
 'backdoor.econml.dr.LinearDRLearner',
 'backdoor.econml.dr.SparseLinearDRLearner',
 'backdoor.econml.dml.LinearDML',
 'backdoor.econml.dml.SparseLinearDML',
 'backdoor.econml.dml.CausalForestDML',
 'backdoor.auto_causality.models.TransformedOutcome',
 'iv.econml.iv.dr.LinearDRIV',
 'iv.econml.iv.dml.OrthoIV',
 'iv.econml.iv.dml.DMLIV',
 'iv.econml.iv.dr.SparseLinearDRIV',
 'iv.econml.iv.dr.LinearIntentToTreatDRIV']

TimoFlesch avatar Sep 23 '22 10:09 TimoFlesch