ProxylessNAS-Pytorch icon indicating copy to clipboard operation
ProxylessNAS-Pytorch copied to clipboard

Published 20 hours ago • verified publisher icon kairos03

Implement Latency Loss

Open xieydd opened this issue 5 years ago • 12 comments

xieydd avatar Aug 23 '19 05:08 xieydd

@kairos03 Now I have full time on it , and i notice the LatencyLoss Class already implement a part of Latencyloss. I have some point confused?

  • how to get the latency of every op, I need to implement it, right?
  • I need define λ2 hyper-parameters in LatencyLoss Class, right?
  • Finally, I need make the Class Differentiable.

xieydd avatar Sep 04 '19 02:09 xieydd

  1. latency of every op. use https://github.com/kairos03/ProxylessNAS-Pytorch/blob/master/proxylessnas/latency.csv file. The file is official latency result in https://hanlab.mit.edu/files/proxylessNAS/LatencyTools/.
  2. no, I think we add new loss and λ in below code in train_search.py. https://github.com/kairos03/ProxylessNAS-Pytorch/blob/c87b233aaffb9e38329cbb7d4fc5f5398b1312a8/proxylessnas/train_search.py#L84
  3. no, You dont need that, just calculate loss. thx.

kairos03 avatar Sep 04 '19 06:09 kairos03

Thank you for your prompt reply @kairos03 From your answer, i got some point:

  1. The latency of every op is fixed.
  2. Add the weight decay term and LatencyLoss to CrossEntropyLoss
  3. I mean the Latency loss can be differentiable, we will not make it differentiable manual.

Also Confused:

  1. This Latency implement is not complete? https://github.com/kairos03/ProxylessNAS-Pytorch/blob/master/proxylessnas/latencyloss.py#L70

xieydd avatar Sep 04 '19 08:09 xieydd

Yes, not completed. You can change whole class.

kairos03 avatar Sep 04 '19 08:09 kairos03

Thx

xieydd avatar Sep 04 '19 08:09 xieydd

what is the difference of pixel _trim and mobile_trim? @kairos03

xieydd avatar Sep 05 '19 02:09 xieydd

I dont know that.. So, I Just choose mobile_trim. @xieydd

kairos03 avatar Sep 06 '19 01:09 kairos03

@kairos03 I have writed one version of latency loss, and i am training(WIP) the model. But it is very slow, it is normal?

xieydd avatar Sep 07 '19 13:09 xieydd

yes training is very slow

kairos03 avatar Sep 08 '19 03:09 kairos03

@kairos03 One Question: Darts need Augment, and the Cell`s node number is fixed, and the Graph is stacked by the Cell. And ProxylessNAS no need Augment, What are your thoughts on this？

xieydd avatar Sep 09 '19 03:09 xieydd

I didn't under stand your question. you mean why ProxylessNAS are not using stacking the Cells?

kairos03 avatar Sep 30 '19 01:09 kairos03

@kairos03 Now, i am clear; Now the problem is accuracy is very low and train very slowly.

xieydd avatar Sep 30 '19 03:09 xieydd