How to reproduce the results in Table 5 of the paper?

[skye95git]

The table5 in the paper shows:

With following the steps in CoCLR on Code Search section, I have applied CoCLR on the task of code search and got the result:

1.How can I get the results for rows 2 and 3 in Table 5? Is it to follow the steps in Vanilla Model? The step2 of Vanilla Model descripts To train a search model without CoCLR. What does the model without CoCLR refer to? Is it CodeBert?

2.The paper descripts Hence, we simply choose RoBERTa-base and CodeBERT as the baseline methods. So the result of row 2 in the table 5 can be get by https://github.com/microsoft/CodeBERT/tree/master/CodeBERT/codesearch, right? Codebert is then evaluated on the data cosqa-retrieval-test-500.json with follow command, right?

3.The row 3 in table5 shows the data is CSN + CoSQA. Can CodeSearchNet Python corpus and CoSQA be used for training at the same time? They have different data formats. The CodeSearchNet data format is:

CoSQA data format is: Can CoSQA be directly used to train codebert?

[Jun-jie-Huang]

For question 1: Yes! You can refer to this part of README to reproduce the results. And the model without CoCLR means training with original data to some extend.

For question 2: No. Our model is different from vanilla CodeBERT in architecture. On one hand, we add one more layer above CodeBERT for better semantic fusion. On the other hand, we use a siamese network architecture while CodeBERT uses a cross-encoder architecture. Therefore, you can not reproduce the results only with the evaluation command. Instead, you can process the CodeSearchNet python corpus into the same format as the code search training data. Then training the model to reproduce row 2 and row 1 in table 5.

For question 3: You can try to train with CodeSearchNet Python corpus and CoSQA at the same time. But it ill-performs than training with CodeSearchNet first then with CoSQA. We guess the former adds noise while the latter enables the model to learn discriminant ability. You can also try to train codebert with CoSQA directly. Also the results will be poorer.

[skye95git]

For question 1: Yes! You can refer to this part of README to reproduce the results. And the model without CoCLR means training with original data to some extend.

For question 2: No. Our model is different from vanilla CodeBERT in architecture. On one hand, we add one more layer above CodeBERT for better semantic fusion. On the other hand, we use a siamese network architecture while CodeBERT uses a cross-encoder architecture. Therefore, you can not reproduce the results only with the evaluation command. Instead, you can process the CodeSearchNet python corpus into the same format as the code search training data. Then training the model to reproduce row 2 and row 1 in table 5.

For question 3: You can try to train with CodeSearchNet Python corpus and CoSQA at the same time. But it ill-performs than training with CodeSearchNet first then with CoSQA. We guess the former adds noise while the latter enables the model to learn discriminant ability. You can also try to train codebert with CoSQA directly. Also the results will be poorer.

Thanks for your reply! For answer 1: I run the vanilla model on code search and get the result as follow:

I don't know which model in the table this result belongs to. Is this the result of row 3 Model CodeBERT, Data CSN+CoSQA in table 5?

I'm a little confused. Why is the name of the third line model CodeBert? Vanilla Model was probably your method. Did I understand something wrong? And Vanilla Model's results (Final test MRR 0.6502) were not much different from CoCLR on Code Search's (Final test MRR 0.6526). It turns out a little bit strange.

For answer 2: You said Then training the model to reproduce row 2 and row 1 in table 5. Do the models in row 1 and row 2 refer to vanilla RoBERTa and vanilla CodeBERT? you said you can process the CodeSearchNet python corpus into the same format as the code search training data. Does the code search training data refer to the training data of vanilla RoBERTa and vanilla CodeBERT or CoSQA?

[Jun-jie-Huang]

As for question 1, we checked the code and found there's a bug, which lead to the mis-use of the model. Now we fix it, you can use the new code to reproduce the results in line 3 with the same command as before.

As for question 2, yes! Row 1&2 refer to vanilla RoBERTa and CodeBERT trained only on CodeSearchNet. And the code search training data in your question refer to the training data of vanilla RoBERTa and vanilla CodeBERT.

[skye95git]

As for question 1, we checked the code and found there's a bug, which lead to the mis-use of the model. Now we fix it, you can use the new code to reproduce the results in line 3 with the same command as before.

As for question 2, yes! Row 1&2 refer to vanilla RoBERTa and CodeBERT trained only on CodeSearchNet. And the code search training data in your question refer to the training data of vanilla RoBERTa and vanilla CodeBERT.

Thanks for your reply! I try the new code to reproduce the results in line 3. There is an error:

What should I do? I don't meet the error when I use the old code.

I have another question: Can the models in row 1 and 2 also be evaluated with the testset cosqa-retrieval-test-500.json in CoSQA? The data format in CoSQA is different from the test set data format in CodeSearchNet. In cosqa-retrieval-test-500.json:

In CodeSearchNet:

[ybagdi]

Hi @skye95git ,

Did you try training with CodeSearchNet python corpus. If yes, could you share the scripts for converting the CodeSearchNet data into CoSQA data format. I am also trying to replicate for the CodeBERT + CodeSearchNet result.

[wusuhuang]

For question 1: Yes! You can refer to this part of README to reproduce the results. And the model without CoCLR means training with original data to some extend.

For question 2: No. Our model is different from vanilla CodeBERT in architecture. On one hand, we add one more layer above CodeBERT for better semantic fusion. On the other hand, we use a siamese network architecture while CodeBERT uses a cross-encoder architecture. Therefore, you can not reproduce the results only with the evaluation command. Instead, you can process the CodeSearchNet python corpus into the same format as the code search training data. Then training the model to reproduce row 2 and row 1 in table 5.

For question 3: You can try to train with CodeSearchNet Python corpus and CoSQA at the same time. But it ill-performs than training with CodeSearchNet first then with CoSQA. We guess the former adds noise while the latter enables the model to learn discriminant ability. You can also try to train codebert with CoSQA directly. Also the results will be poorer.

As for question 1, we checked the code and found there's a bug, which lead to the mis-use of the model. Now we fix it, you can use the new code to reproduce the results in line 3 with the same command as before. As for question 2, yes! Row 1&2 refer to vanilla RoBERTa and CodeBERT trained only on CodeSearchNet. And the code search training data in your question refer to the training data of vanilla RoBERTa and vanilla CodeBERT.

Thanks for your reply! I try the new code to reproduce the results in line 3. There is an error:

What should I do? I don't meet the error when I use the old code.

I have another question: Can the models in row 1 and 2 also be evaluated with the testset cosqa-retrieval-test-500.json in CoSQA? The data format in CoSQA is different from the test set data format in CodeSearchNet. In cosqa-retrieval-test-500.json:

In CodeSearchNet:

hello this issue author, I have the same question. I just want to know how to get row 1 and row 2 results. Can the models in row 1 and 2 also be evaluated with the testset cosqa-retrieval-test-500.json in CoSQA? Are RoBERTa and CodeBERT the same model archetecture?