help on object has no attribute 'name'

[ooodragon94]

trying to prune efficientnet by lukemelas getting AttributeError: 'SwishImplementationBackward' object has no attribute 'name' I tried a lot of solutions that came up in my mind but all failed I also tried to see some documentations on

<class 'AccumulateGrad'>
<class 'ViewBackward'>
<class 'ViewBackward'>
<class 'MeanBackward1'>
<class 'ViewBackward'>

But I couldn't find them. Swish looks like this btw

class SwishImplementation(torch.autograd.Function):
    @staticmethod
    def forward(ctx, i):
        result = i * torch.sigmoid(i)
        ctx.save_for_backward(i)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        i = ctx.saved_variables[0]
        sigmoid_i = torch.sigmoid(i)
        return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))


class MemoryEfficientSwish(nn.Module):
    def forward(self, x):
        return SwishImplementation.apply(x)

I appreciate any help on this... (spending about 5 hours on this haha.. I'm pretty sure I'm going to have another after this.. but I want to give it a try)

Or if anyone succeeded in filter pruning on efficientnet, I would love to hear your experiences...

thanks

[VainF]

Hi, @ooodragon94 , SwishImplementation is not a standard module in pytorch. I'm trying to skip those customized ops and treat them as element-wise OP.

[ooodragon94]

OMG thanks if I may, may I ask when it would be done

[VainF]

BTW, I found that EfficientNet contains group conv which is not supported yet.

[ooodragon94]

oh... then no future plan on supporting it?

[VainF]

I'm working on this. maybe tomorrow?

[ooodragon94]

oh my god that is fast... thanks!

[VainF]

Hi @ooodragon94 , please try the latest version

from efficientnet_pytorch import EfficientNet
import torch_pruning as tp
import torch
import torch.nn as nn
import random

model = EfficientNet.from_pretrained("efficientnet-b0", advprop=True)
model.eval()
example_inputs = torch.randn(1,3,224,224)
prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d )
prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ]
ori_size = tp.utils.count_params( model )
DG = tp.DependencyGraph().build_dependency( model, example_inputs=example_inputs )
for layer_to_prune in prunable_modules:
    # select a layer
    if isinstance( layer_to_prune, nn.Conv2d ):
        prune_fn = tp.prune_conv
    elif isinstance(layer_to_prune, nn.BatchNorm2d):
        prune_fn = tp.prune_batchnorm

    ch = tp.utils.count_prunable_channels( layer_to_prune )

    rand_idx = random.sample( list(range(ch)), min(10, ch//2))
    plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx)
    plan.exec()

print(model)
with torch.no_grad():
    out = model( example_inputs )
print( "  Params: %s => %s"%( ori_size, tp.utils.count_params(model) ) )
print( "  Output: ", out.shape )
print("------------------------------------------------------\n")

[VainF]

My output:

Loaded pretrained weights for efficientnet-b0
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919549b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919548c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919547d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919546e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919545f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919589b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919588c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919587d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919586e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919585f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928129b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928128c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928127d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928126e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928125f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812410>. It will be treated as element-wise op
EfficientNet(
  (_conv_stem): Conv2dStaticSamePadding(
    3, 2, kernel_size=(3, 3), stride=(2, 2), bias=False
    (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
  )
  (_bn0): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
  (_blocks): ModuleList(
    (0): MBConvBlock(
      (_depthwise_conv): Conv2dStaticSamePadding(
        2, 2, kernel_size=(3, 3), stride=[1, 1], groups=2, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        2, 4, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        4, 2, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        2, 4, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(4, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (1): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        4, 46, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(46, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        46, 46, kernel_size=(3, 3), stride=[2, 2], groups=46, bias=False
        (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(46, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        46, 2, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        2, 46, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        46, 2, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (2): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        2, 94, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        94, 94, kernel_size=(3, 3), stride=(1, 1), groups=94, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        94, 3, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        3, 94, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        94, 2, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (3): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        2, 94, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        94, 94, kernel_size=(5, 5), stride=[2, 2], groups=94, bias=False
        (static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        94, 3, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        3, 94, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        94, 5, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(5, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (4): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        5, 190, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        190, 190, kernel_size=(5, 5), stride=(1, 1), groups=190, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        190, 5, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        5, 190, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        190, 5, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(5, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (5): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        5, 190, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        190, 190, kernel_size=(3, 3), stride=[2, 2], groups=190, bias=False
        (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        190, 5, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        5, 190, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        190, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (6): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        430, 430, kernel_size=(3, 3), stride=(1, 1), groups=430, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        430, 10, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        10, 430, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        430, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (7): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        430, 430, kernel_size=(3, 3), stride=(1, 1), groups=430, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        430, 10, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        10, 430, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        430, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (8): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        430, 430, kernel_size=(5, 5), stride=[1, 1], groups=430, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        430, 10, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        10, 430, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        430, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (9): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        622, 622, kernel_size=(5, 5), stride=(1, 1), groups=622, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        622, 18, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        18, 622, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        622, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (10): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        622, 622, kernel_size=(5, 5), stride=(1, 1), groups=622, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        622, 18, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        18, 622, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        622, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (11): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        622, 622, kernel_size=(5, 5), stride=[2, 2], groups=622, bias=False
        (static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        622, 18, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        18, 622, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        622, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (12): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        1102, 38, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        38, 1102, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (13): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        1102, 38, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        38, 1102, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (14): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
        (static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
      )
      (_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        1102, 38, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        38, 1102, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
    (15): MBConvBlock(
      (_expand_conv): Conv2dStaticSamePadding(
        112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_depthwise_conv): Conv2dStaticSamePadding(
        1102, 1102, kernel_size=(3, 3), stride=[1, 1], groups=1102, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        1102, 38, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        38, 1102, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_project_conv): Conv2dStaticSamePadding(
        1102, 300, kernel_size=(1, 1), stride=(1, 1), bias=False
        (static_padding): Identity()
      )
      (_bn2): BatchNorm2d(300, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_swish): MemoryEfficientSwish()
    )
  )
  (_conv_head): Conv2dStaticSamePadding(
    300, 1260, kernel_size=(1, 1), stride=(1, 1), bias=False
    (static_padding): Identity()
  )
  (_bn1): BatchNorm2d(1260, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
  (_avg_pooling): AdaptiveAvgPool2d(output_size=1)
  (_dropout): Dropout(p=0.2, inplace=False)
  (_fc): Linear(in_features=1260, out_features=1000, bias=True)
  (_swish): MemoryEfficientSwish()
)
  Params: 5288548 => 3784884
  Output:  torch.Size([1, 1000])
------------------------------------------------------

[ooodragon94]

it's working perfectly! one question if I may... can I specify percentage in which I want to prune? hm.. I tried validating on the pruned model (which is loaded as pretrained) and its giving me 0.1% accuracy is this normal before finetuning?

Oh I see.. so it is pruning random indexes and giving back a newly made modules hm.. is there option on pruning a pretrained model?

[VainF]

Finetuning is necessary after pruning. The exaple code is only for testing.

[ooodragon94]

ok great! thanks a lot for your help!

[ooodragon94]

hello sorry to interrupt again I'm trying to prune another efficientnet b0 model from here

from models import *
import torch
import torch.nn as nn
import random
import torch_pruning as tp

model = EfficientNetB0()
model.eval()
example_inputs = torch.randn(1,3,224,224)
prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d )
prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ]
DG = tp.DependencyGraph().build_dependency(model, example_inputs=example_inputs)
for layer_to_prune in prunable_modules:
    # select a layer
    if isinstance( layer_to_prune, nn.Conv2d ):
        prune_fn = tp.prune_conv
    elif isinstance(layer_to_prune, nn.BatchNorm2d):
        prune_fn = tp.prune_batchnorm

    ch = tp.utils.count_prunable_channels( layer_to_prune )

    rand_idx = random.sample(list(range(ch)), min(10, ch//2))
    plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx)
    plan.exec()

and such error shows up

File "/home/user/anaconda3/lib/python3.7/site-packages/torch_pruning/dependency.py", line 328, in get_pruning_plan
    root_node = self.module_to_node[module]
KeyError: Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)

I'm looking at self.module_to_node dictionary at it seems like the model above is not added for some reason and I can't see why it fails. May I get some help on this?

thank you

[KIM7AZEN]

hello sorry to interrupt again I'm trying to prune another efficientnet b0 model from here

from models import *
import torch
import torch.nn as nn
import random
import torch_pruning as tp

model = EfficientNetB0()
model.eval()
example_inputs = torch.randn(1,3,224,224)
prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d )
prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ]
DG = tp.DependencyGraph().build_dependency(model, example_inputs=example_inputs)
for layer_to_prune in prunable_modules:
    # select a layer
    if isinstance( layer_to_prune, nn.Conv2d ):
        prune_fn = tp.prune_conv
    elif isinstance(layer_to_prune, nn.BatchNorm2d):
        prune_fn = tp.prune_batchnorm

    ch = tp.utils.count_prunable_channels( layer_to_prune )

    rand_idx = random.sample(list(range(ch)), min(10, ch//2))
    plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx)
    plan.exec()

and such error shows up

File "/home/user/anaconda3/lib/python3.7/site-packages/torch_pruning/dependency.py", line 328, in get_pruning_plan
    root_node = self.module_to_node[module]
KeyError: Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)

I'm looking at self.module_to_node dictionary at it seems like the model above is not added for some reason and I can't see why it fails. May I get some help on this?

thank you

Has the problem been solved? I have the same issue. get error while pruning a bottleneck layer

[ooodragon94]

@KIM7AZEN no I think it worked quite well maybe check your code's version??