[question] Feature importance?

[jarlva]

Hi, Let's say we have a bunch of features (> 100) and the model is trained and working (but took forever to train). Is there a way to tell which features are more important than others or not used at all by the model? So that the next training is more efficient/accurate? Can something like https://captum.ai be used with sample-factory? The only way I know is the process of elimination. But that can take an exorbitant amount of time. I understand that this is beyond the scope of the repository. Any piece of information is appreciated.

[alex-petrenko]

Sorry, the topic of interpretability in RL is rather obscure to me. I didn't even know about https://captum.ai/ From their tutorial on the website, I can't see why this can't be used in SF. Looks like this concerns evaluation time, so maybe just writing a modified enjoy script would do it?

Can you explain what exactly the Captum thing is supposed to tell about your model?

[jarlva]

As is, AI models are by nature a "black box". Meaning - we don't know how the model's logic work and which of the features we gave it are actually used. Feature importance, as its name implies, is able to tell us (in certain cases) the measure of the importance of the features we used. Since not all features are always needed. So, the next time we train a model for the same problem we can use just the features that are of value to AI. Thus saving a lot of training time.

In regard to your note about changing enjoy. If I use hundreds of features it would take forever to filter out the unimportant ones. If I had this technology, after one successful training it will tell me which features not to use next time (to optimize the model). Thus saving a lot of time.

It would be amazing to combine captum (or another feature importance project) with SF. It would create a very powerful combination that would shorten the time even further in order of magnitude.