Hyper-parameter selection ?

[debasmitdas]

It seems that the reverse validation (rv) approach (as mentioned in the paper) to hyper-parameter selection has not been carried out in the repository. So I wanted to ask some of the gory implementation details that you used for reporting in the paper -

1. For the rv procedure you retrain the neural network multiple times using different parameters and test on the validation splits and choose the hyperparameter with the best result. Once you find the best hyper-parameter, you retrain the neural network using all the data and report accuracy on test data. Correct me if you did not follow such a policy.
2. Also, reverse validating a neural network over a range of hyparameters followed by retraining using the best hyperparameters, takes time . So, did you use a parallel procedure to report such results.

[pgermain]

Hi @debasmitdas. First of all, sorry for the late answer. We should publish code here soon to compute the RV risk. Meanwhile, here are my answers to your questions.

1. Our hyper-parameter search procedure is not exactly as you described. We split our source (labeled) sample once into 90%/10% source training/validation sets, and our target (unlabeled) sample into a 90%/10% target training/validation sets. For each hyper-parameter, we learn a single predictor F, and compute its reverse validation (rv) score, using the procedure described in the paper (which implies learning a reverse predictor on the self-labeled target set). Thus, every predictor F is associated with a rv score. We keep the predictor F with the lowest rv score. There is no need to retrain the predictor, which reduces the computational burden.

2. Even then, this takes time. We performed these experiments on a computer grid. The parallelization here was nothing fancy: we executed a separated process for every hyper-parameter and compare the outputs of all processes afterwards.

I hope this helps! Pascal.