SpokenWOZ is a large-scale multi-domain speech-text dataset for spoken task-oriented dialogue modeling, which consists of 203k turns, 5.7k dialogues and 249 hours audios from realistic human-to-human spoken conversations.

The majority of existing TOD datasets are constructed via writing or paraphrasing from annotators rather than being collected from realistic spoken conversations. The written TDO datasets may not be representative of the way people naturally speak in real-world conversations, and make it difficult to train and evaluate models that are specifically designed for spoken TOD. Additionally, the robustness issue, such as ASR noise, also can not be fully explored using these written TOD datasets. Different exsiting spoken TOD datasets, we introduce common spoken characteristics in SpokenWOZ, such like word-by-word processing and commonsense in spoken language. SpokenWOZ also includes cross-turn detection and reasoning slot detection as new challenges to better handle these spoken characteristics. SpoeknWOZ Paper

The data is split into training, dev, and test sets. Download the dataset here (distributed under the CC BY-NC 4.0 license):

Details of baseline models and evaluation script can be found on the following GitHub site: SpokenWOZ Github Page

We will update the models and results on the leaderboard based on the publicly available papers. Feel free to contact Shuzheng Si if you want to submit your results.

Data collection includes (1) Collection of dialogue audio and (2) Annotation of dialogue. Strict quality control is performed at each collection stage. More details can be found in our paper.

Feel free to contact Shuzheng Si, Wentao Ma, and Haoyu Gao. We would greatly appreciate it if you could provide us your helpful suggestions for this project.

We would like to thank Prof. Milica Gasic for her appreciation and advice on our idea at the beginning of the project. We also thank Dr. Bowen Yu for his constructive comments on our writing. Finally, we would like to thank all the annotators for their efforts. We look forward to our dataset advancing the research on spoken TOD.

@inproceedings{NEURIPS2023_7b16688a,
author = {Si, Shuzheng and Ma, Wentao and Gao, Haoyu and Wu, Yuchuan and Lin, Ting-En and Dai, Yinpei and Li, Hangyu and Yan, Rui and Huang, Fei and Li, Yongbin},
booktitle = {Advances in Neural Information Processing Systems},
editor = {A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine},
pages = {39088--39118},
publisher = {Curran Associates, Inc.},
title = {SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents},
url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/ 7b16688a2b053a1b01474ab5c78ce662-Paper-Datasets_and_Benchmarks.pdf},
volume = {36},
year={2023}}

We use the joint goal accuracy (JGA) to evaluate DST task, which measures the ratio of dialogue turns for which the value of each slot is correctly predicted. Challenges of DST in spoken dialogue include robustness against noisy text, cross-turn slot and reasoning slot.

The next challenges becomes the response generation component, including policy optimization and end-to-end modeling. We use INFORM, SUCCESS, BLEU and Combined Score to evaluate generated response.

In end-to-end modeling task, given the utterance from the user, the dialogue system is expected to give the right response.

In policy optimization task, given the utterances and the dialogue state, the dialogue system is expected to optimize conversation policy and give high quality responses.