captum

captum copied to clipboard

I'm encountering an AttributeError while using LayerLRP with the SwitchTransformersAttention module in my custom transformer model. The error occurs when I attempt to compute the attributions using LayerLRP and seems to be related to the assigned LRP rule.

Here is the full error traceback: Traceback (most recent call last): File "/home/xinting/clones/Switch_Magic/get_de_en_LRP.py", line 83, in attributions = layer_lrp.attribute( File "/opt/conda/lib/python3.10/site-packages/captum/attr/_core/layer/layer_lrp.py", line 233, in attribute relevances = self._get_output_relevance(output) File "/opt/conda/lib/python3.10/site-packages/captum/attr/_core/layer/layer_lrp.py", line 282, in _get_output_relevance return self._get_single_output_relevance(self.layer, output) File "/opt/conda/lib/python3.10/site-packages/captum/attr/_core/layer/layer_lrp.py", line 258, in _get_single_output_relevance normalized_relevances = layer.rule.relevance_output AttributeError: 'EpsilonRule' object has no attribute 'relevance_output'

The error suggests that EpsilonRule lacks an attribute relevance_output, which Captum's LayerLRP seems to be attempting to access. The error occurs when attempting to use LayerLRP for relevance propagation through the SwitchTransformersAttention layer of my custom transformer model, specifically when the assigned rule does not seem to meet the expected requirements.

Below is my code snippet:

from captum.attr import LayerLRP from captum.attr._utils.lrp_rules import EpsilonRule import torch model_name = 'google/switch-base-8' tokenizer = AutoTokenizer.from_pretrained(model_name) model = SwitchTransformersForConditionalGeneration.from_pretrained(model_name)

for name, module in model.named_modules(): if isinstance(module, SwitchTransformersAttention): setattr(module, 'rule', EpsilonRule())

target_layer = model.decoder.block[0].layer[0].SelfAttention layer_lrp = LayerLRP(model, layer=target_layer)

attributions = layer_lrp.attribute( inputs=input_embeds, additional_forward_args=(attention_mask, decoder_inputs_embeds), attribute_to_layer_input=False, verbose=True )

Any help or suggestions would be appreciated!