Can not reproduce the search result on PTB dataset (Fig3)?

[fortunechen]

trafficstars

Hi, everyone.

I try to reproduce the model search performance in Fig 3 on PTB dataset. Becauce it tells us as searching time growing, the searched model will be improved.

I run the train_search.py code 4 times with different random seed. Then I get the result below. The x-axis denotes how many gpu hours I trained to get the model and y-axis denotes the performance of searched model after training for 300 epoch. Except for neccssacy changes to run the code successfully on pytorch1.7 +cuda 11.. I didn't change any code in original code.

The result shows below, we can see that as searching time grows, the searched model will NOT be improved.

[fortunechen]

Here is the code I used darts-rnn with pytorch1.7+cuda11.0

[fortunechen]

I will try reproduce the result on pytorch 0.3.1 again.

[chaoji90]

Empirically, darts search time and performance are often inversely proportional and the early stop is usually inevitable.

[fortunechen]

Empirically, darts search time and performance are often inversely proportional and the early stop is usually inevitable.

I agree so. However the total search time is the same as the paper(50 epochs). and the preformace is much lower than the author's.

[fortunechen]

I will try reproduce the result on pytorch 0.3.1 again.

Here is the result without any modification of original code.