Calculation of accuracy and other queries

[Sammed98]

I have a few more queries:

1. When I am running the evaluation script it gives me a warning "WARNING: right now 'eval' gives correct 'eval' test set". What does this mean?
2. When printing the accuracy of the model at the end, do you only check if the generated code is compilable or do you check if for all inputs, of the corresponding instance, the output by the generated code is correct?
3. When I ran the code for main_supervised_algolisp.py --pretrain, it ran the pretraining code but automatically started the training process. Is this supposed to happen? Now that the training process has started do I have to again run the main_supervised_algolisp.py without the pretrain flag?
4. If I want to save the code generated for the test dataset, how should I proceed?
5. For the calculation of accuracy have you matched the generated code to the ground-truth value without compiling the code?
6. Based on our research we found that the theoretical maximum accuracy should be 89.9% in the normal dataset and you have reported a 90% accuracy for the same. In that case, your accuracy for the filtered dataset should be 100% since all the non-compilable code will be removed via filtering.

Can you clear these doubts?

[dotieuthien]

Hi guy, can you run the command python train/deepcoder_train_dc_model.py. I cannot run from grammar import Grammar, NoCandidates. Do yo know the package grammar used in this repo? Thank you

[Sammed98]

You will find it in "https://github.com/lcary/ec-backup" repository.