deep-code-search Dataset

Hi guys, awesome project. Would you mind releasing the original training and testing dataset, without any pickled or preprocessed files?

Aug 06 '18 12:08 villmow

@villmow Check out the files can be downloaded from here on Gdrive or Baidu url.

Dec 04 '18 00:12 0bserver07

@0bserver07 @guxd could you share the raw method names\descriptions?

Dec 04 '18 07:12 matanpugach

@matanpugach That's what I meant!

I already downloaded the files from Gdrive, but they are preprocessed. Everyone using your dataset is limited to your features (token, api sequence, name tokens) and a vocabulary limit of 10,000. One cannot restore the original "RawCode" - "Documentation" mapping from your dataset, to - for example - try new features.

Dec 04 '18 08:12 villmow

Hi @villmow, I have the same requirement, have you got the raw datasets without preprocessing from @guxd?

Dec 17 '18 18:12 wanyao1992

i have python raw data how to preprocess it as same u did for java. I would to build code search for python code. How to do preprocessing??

Apr 02 '19 12:04 gauravkoradiya

@gauravkoradiya You should use python code parser. Python provides an ast module which supports the parsing.
Here is a sample project which parses python code into ASTs. https://github.com/fyrestone/pycode_similar You may find more from the GitHub. After that, you need to convert ASTs into call sequences.

Apr 02 '19 12:04 guxd

@gauravkoradiya You should use python code parser. Python provides an ast module which supports the parsing. Here is a sample project which parses python code into ASTs. https://github.com/fyrestone/pycode_similar You may find more from the GitHub. After that, you need to convert ASTs into call sequences.

Apr 19 '19 11:04 gauravkoradiya

Awesome..thank you....I got it.

Apr 19 '19 11:04 gauravkoradiya

could you share the original datasets without any pickled or preprocessed files?