Qi Zhu

> the spacy tokenizer in the NLU module (specifically, jointBERT) is downloaded every time the script is launched? No, I think only the first time. Once downloaded, `self.nlp = spacy.load("en_core_web_sm")`...

`evaluate.py` uses the unified interface inherited from `NLU` such as [`class BERTNLU(NLU)`](https://github.com/thu-coai/ConvLab-2/blob/master/convlab2/nlu/jointBERT/multiwoz/nlu.py#L17). Each NLU model should provide such a class so that we can compare different models given the same...

Yes, `evaluate.py` is used to compare the performance of different NLU. it will be slower than `test.py` since it uses batch_size=1. If you want to test only BERTNLU (e.g., to...

Yes, they are. TP, FP, FN are accumulated through the test set

我们后来修改了 `evaluate.py` (https://github.com/thu-coai/ConvLab-2/pull/85/files) 但还没有更新结果。多谢反馈！

Thanks! We've noticed MultiWOZ 2.2. We will add it if it is of high quality

We are planning to add many datasets (SchemaGuided, Taskmaster, etc.) using a unified format.

Actually, we have processed SGD, Taskmaster, and other datasets. We will update them with MultiWOZ 2.2 & 2.3 in few days. Thanks!

@tomolopolis SGD and Taskmaster are available in unified format #180.

@tomolopolis we will modify the [unified data process](https://github.com/thu-coai/ConvLab-2/tree/master/convlab2/util/dataloader) and support some of the useful models. However, some models have a lot of dataset-specific processes which can not be well unified.