Clarification of Paragraph in Paper

[theevann]

Section 4.2 p7, I read:

NAO seems to constantly outperform random search in the reduced space. Nevertheless, the final architecture chosen by NAO is always one of the architectures from the initial pool, which were sampled uniformly randomly. This indicates that the ranking of NAO is not correctly updated throughout the search and that, in practice, in a reduced space, NAO is similar to random search.

I do not understand this paragraph, in particular the second sentence and why you claim NAO is similar to random while seemingly outperforming it. Could you help clarify it ?

[kcyu2014]

Sorry for my late reply. If you check the standard deviation, you could see that the confidence interval heavily overlaps. So from pure statistical point of view, the performance of random search and NAO is not really different.