I can't reproduce the architecture search process following the configuration in paper.

[ghost]

Hi Hanxiao @quark0 ,

Recently, I try to reproduce your work. I use the code https://github.com/quark0/darts/ and reproduce the test process. It's great! But I cannot reproduce the architecture search process to obtain the same or similar as the DART_V2 architecture. I run "train_search.py" following your paper. There was an overfitting during the training and the final valid loss was 6.56.

Is this result normal? If not, can you give me some advice?

[ghost]

Or can others reproduce the architecture search? We discuss it together.

[brycexu]

I cannot reproduce the architecture search process neither. I did not change any code, but I obtained a different architecture claimed in the paper. It is strange.

[ray-lee-94]

yes, the structure is different. The paper said we need to run many times.

[ShihuaHuang95]

@VCBE123 So have you reproduced the results when you run many times?

[ghost]

We need to change seed and run many times, but until now I can not get a similar result.

[pingguokiller]

Have any other reproduce the result successfully? I got the best valid acc 88% in seaching process with the random seed setted as 2(without cutout data precessing).

[HaFred]

Have any other reproduce the result successfully? I got the best valid acc 88% in seaching process with the random seed setted as 2(without cutout data precessing).

May I ask you which version of Pytorch are you using? Thanks

[pingguokiller]

Have any other reproduce the result successfully? I got the best valid acc 88% in seaching process with the random seed setted as 2(without cutout data precessing).

May I ask you which version of Pytorch are you using? Thanks

I use torch 1.2.

[Debrove]

Have any other reproduce the result successfully? I got the best valid acc 88% in seaching process with the random seed setted as 2(without cutout data precessing).

Hi @pingguokiller , I am also trying to reproduce the results. Do you change the seed to search architectures? It's strange that once I changed the seed the searching time greatly increase.

[pingguokiller]

Have any other reproduce the result successfully? I got the best valid acc 88% in seaching process with the random seed setted as 2(without cutout data precessing).

Hi @pingguokiller , I am also trying to reproduce the results. Do you change the seed to search architectures? It's strange that once I changed the seed the searching time greatly increase.

I have reproduced the results successfully(First order). I overlooked some important hyper-parameters. Other researchers told the results were very random with the change of seed.

[Debrove]

@pingguokiller Do you reproduce the results(First order) with the process of searching four times, and then choosing the best valid accuracy of the arch. Finally training it three times and get the average accuracy?

Moreover, would you mind pointing out what important hyperparameters is you mentioned above.

Thank you very much.

[d12306]

@pingguokiller , same question here, could you help us getting the right shot?

[pingguokiller]

@pingguokiller , same question here, could you help us getting the right shot?

The search progress is very random. Maybe it's difficult to get the same architecture as presented in paper. You can reproduce the valid process with the code provided by the author on cifar10. However, I still can't reproduce the valid process on imagenet until now.