train data json explanation

[Educational-Project]

Hi! Thank you so much for this shallow discourse parser I would like to adapt it to the Ukrainian language. It would be very helpful if you give some step-by-step recommendations on how to create a JSON file, how to conduct the BERT training, and so on. It will be very helpful! Thank you in advance!

[rknaebel]

sorry, I am not very familiar with the Ukrainian language. The json format itself follow mostly the description here: https://nbviewer.ipython.org/github/attapol/conll16st/blob/master/tutorial/tutorial.ipynb

I myself use the tokenization cli: discopy-tokenize -i TEXT_DOC which generates the corresponding format. It also breaks texts into separate documents if it finds two blank lines. You probably have to adapt the internal parser which is currently fixed to english... (https://github.com/rknaebel/discopy-data/blob/8edbccc7f77c54cb063c20fa53136fdf4c68f291/discopy_data/data/loaders/raw.py#L17)