Some confusions

[Jihao-Li]

Thank you for your code. I still have some confusions:

1. Your code is different from DARTS. You first update weight parameters and then update architecture parameters. But DARTS is contrary to you. Could you elaborate on it? When you update architecture parameters, you calculate loss again on train set. This may occupy more GPU memory. I think after you updated weight parameters, you can directly use this loss.

# model weight update
optimizer.zero_grad()
logits = model(input)
loss = criterion(logits, target)

loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip)
optimizer.step()

# architect parameter alpha update
architect.step(input, target, input_search, target_search, lr, optimizer, unrolled=args.unrolled)``

2. It is mentioned in the original paper that architecture parameters are resacled in section 3.2. Do you implement this function?
3. There's one thing I'm not sure about. When update architecture parameters, the author of the origin paper masks architecture parameters and only two paths are selected. When update weight parameters, how many paths are selected? Two paths or all paths? Thank you very much!

[xieydd]

@JihaoLee

1. Darts and ProxylessNAS are two step search, first update the weight parameters and then update the architecture parameters. And i think two step user same GPU memory in update weight first or update architecture parameters first.
2. Not implement the function, i implement it yet.
3. Still Confused, i am asking the author of papers, I prefer the latter。

[Jihao-Li]

@xieydd Thank you! Could you give me a link to your implementation about ProxylessNAS?

[xieydd]

Sorry, Still WIP.