Open-korean-corpora

Open-korean-corpora copied to clipboard

Open Korean Corpora: A Living Document for Korean NLP Dataset Curation

Overview

• Korean, a language with 80M users is often overlooked in NLP research
• The availability of public datasets and tasks has hindered investigation
• Even the publicly available datasets are not always accompanied by English documentation and have poor discoverability
• Our work attempts to tackle this problem by curating a living document of open resources for the Korean language

NLP-OSS @ EMNLP 2020

We will be in the live session and monitoring the slide chat during EMNLP 2020. If you have any questions or would simply want to drop by to say hello, please drop by!

Public Institutions

Multiple government-funded institutions create datasets for the Korean language

• National Institute of Korean Language (NIKL)
• Electronics and Telecommunications Research Institute (ETRI)
• NIA AI HUB

Generally, government funded datasets tend to be very restrictive at allowing access to non-Korean citizens

Thus, Open Corpora here denotes a freely accessible and downloadable (at least only with a simple sign-in) dataset

Open Dataset for Korean NLP

Our work focuses on curating open Korean corpora under the following criteria:

• Documentation status
• License for use and distribution

Documentation and License

For documentation status Docu. the following holds.

• doc - Does the corpus have any documentation on the usage?
• art - Does the corpus have a related article?
• inter - Does the corpus have a internationally available publication?

License

For License, we check the followings:

• Commercially available (com), academic use only (acad), unknown (unk)
• Redistribution is available with/without modification (rd and rd/mod-x), neither (no), unknown (unk)

Other Attributes

• In Providers, we note if the dataset is provided by universities or institutes (Academia), companies or the research group thereof (Industry), or something combined, as Competition purpose.
• In Volume, (w) denotes words, (s) denotes sentences, (p) denotes pairs (either document or sentence pairs), (d) denotes dialogues, (h) denotes hours, and (u) denotes speech utterances.
• In Goal, Eval is noted if the purpose is suggested as an evaluation.

View at a Glance

The table below describes the open Korean corpora investigated so far. To be updated along with our survey or PR. You can visit Here for the Korean description, and more information regarding government-driven database.

Citing

To cite our work, please use the following: (Also available as cho-etal-2020-open in anthology.bib)

@inproceedings{cho-etal-2020-open,
    title = "Open {K}orean Corpora: A Practical Report",
    author = "Cho, Won Ik  and
      Moon, Sangwhan  and
      Song, Youngsook",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.12",
    pages = "85--93",
    abstract = "Korean is often referred to as a low-resource language in the research community. While this claim is partially true, it is also because the availability of resources is inadequately advertised and curated. This work curates and reviews a list of Korean corpora, first describing institution-level resource development, then further iterate through a list of current open datasets for different types of tasks. We then propose a direction on how open-source dataset construction and releases should be done for less-resourced languages to promote research.",
}

Contributing

Please read the contributor guidelines before sending a pull request.