NLU datasets with task-oriented dialogue

Datasets of natural language understanding and dialogue state tracking for task-oriented dialogue, which can be used in research. There are some other survey of datasets in respective of diaogue system, like AtmaHou's Task-Oriented-Dialogue-Dataset-Survey (I am one of the contributors). But we focus on how to build a semantic parser for spoken dialogue system.

If you want to know more about NLU of task-oriented dialogue, please see recommended papers.

There is an implementation of joint training of slot filling and intent detection for SLU, which is evaluated on ATIS and SNIPS datasets.

Table of Contents

• Introduction

• Datasets with single turn

• Datasets with multiple turns

• Details

Introduction

• Intent Detection or intent classification: sentence classification task
• Slot Tagging: sequence labelling task
• Slot Filling: It equals to slot tagging if all values of slots can be aligned into input sentence. Otherwise, the value of slot should be predicted in a classification or generation way.

Datasets with single turn (not a dialogue)

Datasets with multiple turns (dialogue with context)

Details

More information about each dataset.

ATIS

• single turn;
• input sentences: natural language;
• data size (single domain of "flight information searching"):
  • training set: 4978 utterances;
  • test set: 893 utterances;
• semantic annotation: intent (sentence class), slot (sequence labelling)
  • intent number: 18
  • slot number: 83
• Download: https://github.com/yvchen/JointSLU

MIT corpus

• single turn;
• input sentences: natural language;
• data size:
  • MIT_Restaurant domain:
    • training set: 7660 utterances;
    • test set: 1521 utterances;
  • MIT_Movie domain (simple query):
    • training set: 9775 utterances;
    • test set: 2443 utterances;
  • MIT_Movie domain (complex query):
    • training set: 7816 utterances;
    • test set: 1953 utterances;
• semantic annotation: slot (sequence labelling)
• Download: https://groups.csail.mit.edu/sls/downloads

SNIPS

• single turn;
• input sentences: natural language;
• data size:
  • 7 intents: each has more than 2000 queries.
• semantic annotation: intent (sentence class), slot (sequence labelling)
• Download: https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines

TOP semantic parsing

• single turn;
• input sentences: natural language;
• data size:
  • training set: 35741 queries
  • test set: 9042 queries
• semantic annotation: hierarchical intents, slot (it is a tree)
  • intent number: 25
  • slot number: 36
• Download: http://fb.me/semanticparsingdialog

SMP2017-ECDT (in Chinese)

• single turn;
• input sentences: natural language;
• data size:
  • http://ir.hit.edu.cn/SMP2017-ECDT
  • training set: 2299 queries
  • development set: 770 queries
  • test set: 666 queries
• semantic annotation: intent
  • intent number: 31
• Download: https://github.com/HITlilingzhi/SMP2017ECDT-DATA

DSTC 2&3

• multiple turns: human-machine dialogues;
• input sentences:
  • transcription by human;
  • ASR output: n-best, word confusion network;
• data size:
  • DSTC 2 (Restaurant Information Domain): source domain
    • training set: about 2k dialogues;
    • test set: about 1k dialogues;
  • DSTC 3 (Tourist Information Domain): extented domain
    • seed data: about 10 dialogues;
    • test set: about 2k dialogues;
• semantic annotation: dialogue act
  • DSTC 2: 8 slots;
  • DSTC 3: 13 slots;
• Download: https://github.com/matthen/dstc

DSTC 4

• multiple turns: human-human dialogues;
• input sentences: natural language, transcription by human;
• data size:
  • This data is about touristic information for Singapore collected from Skype calls.
  • 35 dialogs sum up to 31,034 utterances and 273,580 words
• semantic annotation: speech action, slot, dialogue state (slot-value pairs) in sub-dialogue level
• Download: challenge participants only, http://www.colips.org/workshop/dstc4/

google Sim-R/Sim-M/Sim-gen

• multiple turns: conversations between an agent and a simulated user;
• input sentences: natural language;
• data size:

• semantic annotation: slot
• Download: https://github.com/google-research-datasets/simulated-dialogue

cam MultiWOZ 2.0/2.1

• multiple turns: human-human dialogues collected in the way of WOZ (Wizard-of-Oz);
• input sentences: natural language;
• data size: There are 3,406 single-domain dialogues that include booking if the domain allows for that and 7,032 multi-domain dialogues consisting of at least 2 up to 5 domains.
• semantic annotation: dialogue state (slot-value pairs)
• Download: http://dialogue.mi.eng.cam.ac.uk/index.php/corpus/

maluuba Frames

• multiple turns: human-human dialogues collected in the way of WOZ (Wizard-of-Oz);
• input sentences: natural language;
• data size:
  • It is about travel.
  • 1369 dialogues, 19986 turns;
  • http://www.aclweb.org/anthology/W17-5526
• semantic annotation: intent, dialogue act
• tasks: NLU (intent classification, slot tagging), DST (slot-value pairs)
• Download: https://datasets.maluuba.com/Frames/dl

Microsoft Dialogue Challenge

• multiple turns:
  • human-human dialogues collected via Amazon Mechanical Turk;
  • Built-in user simulators are provided;
• input sentences: natural language;
• data size:

• semantic annotation: dialogue act
• tasks: NLU (slot tagging)
• Download: https://github.com/xiul-msr/e2e_dialog_challenge