multiwoz icon indicating copy to clipboard operation
multiwoz copied to clipboard

Published 20 hours ago • verified publisher icon budzianowski

Normalization during test

Open gusalsdmlwlq opened this issue 3 years ago • 2 comments

Hi. https://github.com/budzianowski/multiwoz/blob/a24d299fafa00371d03880bce34cb3b0923518fa/evaluate.py#L133 https://github.com/budzianowski/multiwoz/blob/a24d299fafa00371d03880bce34cb3b0923518fa/utils/nlp.py#L73 In this line, the values of belief state are normalized before searching DB.

But, the time values of train(leaveAt, arrvieBy) are also normalized to [value_time].

In this case, I think we cannot check whether the values meet the user's goal, right?

Is it intended?

gusalsdmlwlq avatar Oct 19 '20 01:10 gusalsdmlwlq

Hi, thanks for catching this! It's a bug! However, due to backwards compatibility with previous results, it won't be easy to fix that straight away. I will try to come up with a good solution.

budzianowski avatar Nov 09 '20 17:11 budzianowski

Thanks for relpy! Well, I'll leave this issue open. If you want or resolve, close this issue please.

gusalsdmlwlq avatar Nov 10 '20 05:11 gusalsdmlwlq