Query about evaluation script

[smith-co]

Readme mentions to run plur_evaluator.py.

python3 plur_evaluator.py
   --dataset_name=manysstubs4j_dataset 
   --target_file_pattern=/tmp/manysstubs4j_dataset/targets.txt 
   --prediction_file_pattern=/tmp/manysstubs4j_dataset/predictions.txt

Here the ground truth is targets.txt (`-target_file_pattern=/tmp/manysstubs4j_dataset/targets.txt).

I have two queries:

• What's the format of this file?
• Should not be this targets.txt file created after running the Data loader script (plur_data_generation.py)?

But after running plur_data_generation.py, I don't find the targets.txt in the /tmp/manysstubs4j_dataset/ folder.

@smoitra87 @VHellendoorn @dan-zheng can you please help me with this query?

[dan-zheng]

Hi there! I haven't tried running PLUR personally, so I'm not sure how much I can help.

But after running plur_data_generation.py, I don't find the targets.txt in the /tmp/manysstubs4j_dataset/ folder.

Did you try specifying the --stage_1_dir and --stage_2_dir flags? They seem to control where data is generated.

https://github.com/google-research/plur/blob/6181909a0d09cc6492a5819f68ad0ca9cdc082e8/plur/plur_data_generation.py#L75-L78

[smoitra87]

Hi @smith-co and thanks @dan-zheng for jumping in. The format of targets.txt should be here - https://github.com/google-research/plur/blob/main/plur/eval/README.md

Hope this helps. If you're still having trouble generating the dataset please post here again with a full set of commands you ran and the outputs you obtained.

[nashid]

@smoitra87 where is the equivalent file for /tmp/code2seq_dataset/targets.txt with the manysstubs4j_dataset? I dont see targets.txt file for the manysstubs4j_dataset.

@smith-co have you experienced the same?