How to calculate Bert FLOPs

[ZLKong]

Hi,

I have a very rookie question. How can I calculate the FLOPs of BERT model? I tried to use thop,

macs, params = profile(model, inputs=(input, ), 
                        custom_ops={YourModule: count_your_model})

but I don't know how what is the input and custom_ops={YourModule: count_your_model}

For example, I want to run the models given by Huggingface. https://github.com/huggingface/transformers/tree/master/examples/text-classification

CUDA_VISIBLE_DEVICES=1 python run_glue.py \
  --model_type bert \
  --model_name_or_path /tmp/fintune_CoLA_output-bert/ \

I tried to put the macs, params = profile(model, inputs.....) command line in run_glue.py, but I'm not sure where to put it. I get errors like: [WARN] Cannot find rule for <class 'torch.nn.modules.sparse.Embedding'>. Treat it as zero Macs and zero Params. [WARN] Cannot find rule for <class 'torch.nn.modules.normalization.LayerNorm'>. Treat it as zero Macs and zero Params.

File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/transformers/trainer.py", line 677, in _training_step model, inputs=inputs, custom_ops={ File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/thop/profile.py", line 188, in profile model(*inputs) File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/transformers/modeling_bert.py", line 1144, in forward inputs_embeds=inputs_embeds, File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/zhk20002/anaconda2/envs/Py3.6/lib/python3.6/site-packages/transformers/modeling_bert.py", line 691, in forward input_shape = input_ids.size() AttributeError: 'str' object has no attribute 'size'

Do you have a general code like this where I can test out the Flops of models such as BERT, RoBERTa, DistilBERT by just changing the --model_type?

Thanks!

Tony

[autoliuweijie]

There is no a general code that can test the FLOPs of all models, because different models are implemented differently.

If you want to get the FLOPs of a model, you should add the following code to the models' script:

flops, params = profile(model, inputs, verbose=False)

the model is a torch instance, the inputs is the input tensor for this model.

[ZLKong]

Thanks! It worked!

[ZLKong]

There is no a general code that can test the FLOPs of all models, because different models are implemented differently.

If you want to get the FLOPs of a model, you should add the following code to the models' script:

flops, params = profile(model, inputs, verbose=False)

the model is a torch instance, the inputs is the input tensor for this model.

Hi,

In your paper, is the total FLOPS of BERT 21785M? It looks very small. Is thop capable of calculating FC layers? I see that FLOPs depend on input sequence length. What is the FLOPs for one token?

Thanks, ZK

[llsj14]

https://arxiv.org/pdf/2003.10555.pdf It seems that this paper also represents the inference FLOPs of BERT-BASE as 29 BFLOPs. so 21785M looked quite affordable for me.