batchnorm_prune enable the modularity of core API

Need to remove dependency on explicitly computing the pruning weights. (Current is hard-coding)
Need to properly handle input.op.name in compressed mode. (Current is hard-coding)

Oct 17 '18 05:10 bobye

hi， i want to follow your work， shall you show me the detail about how changing the code？ Very thanks!!!

Sep 01 '19 10:09 shz1314

@shz1314 what specific changes you want to make? I recommend you try to get the code running first.

Sep 01 '19 15:09 bobye

Hi, my error is because of python3.5. i want to use your method to accelerate my RCNN, but i need Deconvolution layer, i see your paper have similar work， so shall you show the code about 5.3 SEGMENTATION EXPERIMENT in your paper. So thanks. Sun Jose

Sep 03 '19 09:09 shz1314

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

Sep 03 '19 14:09 bobye

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Sep 04 '19 01:09 shz1314

您的邮箱是多少啊我想把我的网络结构发给您基于tensorflow的你可以看一下这种结构是否可以使用这种方法！有您的帮助我应该更容易在检测阶段复现出您的方法!

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Sep 04 '19 02:09 shz1314

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

完成layer本身的功能（定义变量，设置operator等）
构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
（如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）
定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning）实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

On Tue, Sep 3, 2019 at 6:54 PM shz1314 [email protected] wrote:

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

Sep 04 '19 05:09 bobye

非常感谢您的回复，迷糊灌顶，我先来搞一搞。还有两点建议： 1其实这块原理我还是刚入门，但是你提到权重大小，还是输出大小决定怎么进行剪枝，我之前使用过权重归一化（weight normalization）的方法，感觉可以用其来衡量权重大小从而剪枝？ 2 如果使用densenet的block结构，是不是每一个block里都可以做到剪枝呢。

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月4日(星期三) 中午1:34 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

完成layer本身的功能（定义变量，设置operator等）
构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
（如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）
定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning）实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

On Tue, Sep 3, 2019 at 6:54 PM shz1314 [email protected] wrote:

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Sep 04 '19 06:09 shz1314

您好，我观察了下剪枝的结果，在一个残差网络块结构中输入输出的通道数为（64，64）（64，64）修剪形如（64，32）（32，64），但是我之前的工作构建残差网络的时候就考虑过这点，从而我的初始通道结构就为（64，32）（64，32），如果接着使用这种方法，是微调我的结果，还是？而且我之前试过手调通道数，将原来的深层通道数为256，全部设置为128，此时性能不减！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月4日(星期三) 中午1:34 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

完成layer本身的功能（定义变量，设置operator等）
构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
（如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）
定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning）实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

On Tue, Sep 3, 2019 at 6:54 PM shz1314 [email protected] wrote:

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"[email protected]; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"[email protected]; 抄送: "Sun Jose"[email protected];"Mention"[email protected]; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 [email protected] wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Sep 04 '19 07:09 shz1314

batchnorm_prune batchnorm_prune copied to clipboard

enable the modularity of core API

batchnorm_prune
batchnorm_prune copied to clipboard