Exclude Features from 2nd Order Features Search

[JoshuaC3]

When training an EBM the parameter, interactions=10, helps improve the predictive power of the model. However, in certain scenarios, this can cause problems, see #184 as a single example. This is particularly true in regulated spaces and/or trying to understand feature importance and their affects on predictions.

To mitigate this we can set interactions=0, however, this is likely to reduce the predictive power of most models. Therefore, I propose a parameter which allows features to be excluded from the 2nd Order Feature Search. This would allow the creation of 2nd order features while also safeguarding the use of constrained or regulated features.

An simple and rough example the parameter could work as follows: second_order_exclusion=[0, 1, 5, 21, 22] or second_order_exclusion=['feat_name_0', 'feat_name_1', ..., 'feat_name_22'].

[interpret-ml]

Hey @JoshuaC3,

This is a nice idea! As you point out, it's currently not possible to use the automated interaction detection with any restrictions -- it's currently all or nothing. The main discussion point internally has been around the API for introducing this.

In the past, we had a community contribution (https://github.com/interpretml/interpret/pull/66) which enabled support for "first_order_exclusion", i.e. allowing a feature to be skipped in training main effects but still be included in pairs. It's currently supported as a mains init argument in EBMs, and is used here: https://github.com/interpretml/interpret/blob/ac5e727498cbdcb6911444420343454b0f328a46/python/interpret-core/interpret/glassbox/ebm/ebm.py#L952-L957)

What we've mainly been discussing is:

• The mains flag is currently inclusion-based -- you need to specify all the features you want included, not the ones you want skipped. On the other hand, it seems more common and practical to specify exclusions (as you've also suggested). Having two arguments that work in opposite ways is unintuitive/confusing, so we might need to change how the mains flag is currently used.
• Because the existing mains and proposed second_order_exclusion parameters are so similar, we're brainstorming ways of unifying them as a general feature_exclusions parameter which allows specifying both first and second order terms to skip. There's quite a lot of ways to approach this joint parameter design (e.g. a dictionary for mains/pairs, list of lists for each stage, etc.). and we're working through the relative tradeoffs of each. If none of these designs work out, we can always fall back to two separate parameters.

Wanted to share an update and hear your thoughts on this! We'll continue to update this thread as we make progress on our side.

-InterpretML Team

[JoshuaC3]

Personally, the problem I see in having an inclusion based parameter, second_order_constraints is it is slightly less intuitive - I would assume there to be fewer second-order-only variables than first-only or both.

This is how LightGBM does its feature interactions: interaction_constraints. This is a different type of constraint, namely, restricting any set features being in the same tree. If they are listed interaction_constraints=[[1, 2], [1, 3]] for example, this would restrict 2 and 3 from being in the same tree and use only [1, 2] and [1, 3] together.

That said, I feel the most intuitive solution would be rename and rework mains (quite an ambiguous name) to behave as first_order_exlusions and then add second_order_exclusions as proposed above. If ever desired and/or required, add an interaction_constraints which would either supersede/subside to second_order_exclusions, or throw a warning when there was a conflict.

Ultimately, whether you chose to go with inclusions or exclusions I am not so bothered. If it is named well then it would be clear how it works, especially if it has an example. I would be very happy to put together a notebook example to showcase this if/when it gets implemented.

[francescopisu]

Is there any update on this regard ?

[mtl-tony]

I recently tried implementing this since I had a problem in which I wanted certain variable to never interact and it seemed to work for me. With the updates of 0.3 it's a relatively easy adjustment. I was going to do a pull request when I cleaned my code a bit but I can share where to change it for now.

https://github.com/interpretml/interpret/blob/33162306fe82e2c61d75db49b3c493f8b8099925/python/interpret-core/interpret/glassbox/ebm/ebm.py#L607-L618

From there I added a 'interaction_constraints' list in the EBMModel class. https://github.com/interpretml/interpret/blob/33162306fe82e2c61d75db49b3c493f8b8099925/python/interpret-core/interpret/glassbox/ebm/ebm.py#L192

Then what you need to do is instead of passing combinations(range(n_features_in), 2) at line 613 you pass what fits your needs. I used the interaction_constraints and feature_names_in to make my combinations.

I'll probably do a pull request in the next few days.

[paulbkoch]

Hi @mtl-tony, @francescopisu & @JoshuaC3 -- I would like to propose a single parameter called "exclude=[...]" which will replace the existing "mains" parameter, and also handle the second order constraint case. You could use it in the following ways:

exclude=[7, 9] # exclude mains of feature 7 and feature 9, but allow pairs that include these. Also accept tuples

exclude=[(7, x) for x in range(0, n_features)] # exclude pairs that include feature 7

exclude=[(7, 9), (7, 11)] # exclude specific pairs

exclude= [7] + [(7, x) for x in range(0, n_features)] # exclude feature 7 entirely

[paulbkoch]

The exclude parameter which handles both main effects and pair exclusions has been implemented in the develop branch. It will be in the next v0.4.0 release.

It works as described in my last message above.

[tszyan-bain]

hi @paulbkoch

Thanks for releasing this feature.

Just to clarify:

1. when higher order interactions are included, do I need to specify them too? For example exclude = [(7,9,11)] for a 3rd order interaction exclusion.
2. Does the order of the index matter? What I mean is whether exclude=[(7,9)] will exclude also (9,7) as well as (7,9)?

[paulbkoch]

Hi @tszyan-bain --

1. If you specify the interactions explicitly and then exclude them it should work for pairs, triples, etc. There's no way to currently get 3rd order interactions other than explicitly specifying them, so the only way to get to this scenario for 3rd order interactions would be to specify and then exclude them.

2. The order does not matter for exclusions. exclude=[(7,9)] will exclude both (7,9) and (9,7).