autofeat icon indicating copy to clipboard operation
autofeat copied to clipboard

Cannot reproduce results

Open RomuloPaiva01 opened this issue 3 years ago • 1 comments

Every time I call fit_transform I get different results.

I noticed that np.random.permutation changes the random_state, so I used np.random.RandomState(seed=seed).permutation() to solve.

I also noticed that np.random.seed(i) is used in run_select_features, but it changes the random state in the same way, so I can always convert back to the random_state that I had.

Even with those changes, and always getting the same random_state after calling fit_transform, I always end up with different results.

RomuloPaiva01 avatar Aug 18 '20 21:08 RomuloPaiva01

Yes, randomness is used in a lot of places in the code, both explicitly in places you've mentioned as well as internally (e.g. in some of the models). And it is crucial for the feature selection to use lots of randomness everywhere to make sure a robust subset of features is selected.

If you find a way to catch all instances where randomness is used and make it possible to pass a single random seed to the model to make the results reproducible, I'd love to accept a pull request! :)

cod3licious avatar Aug 21 '20 10:08 cod3licious