Book ; Online: PoE
a Panel of Experts for Generalized Automatic Dialogue Assessment
2022
Abstract: Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat ... To tackle the multi-domain dialogue evaluation task, we propose a Panel of Experts (PoE), a multitask ... domain dialogue dataset leveraging data augmentation and pseudo-labeling. The PoE network is ...
| Abstract | Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat, exchange of information, and grounding in emotional situations. To effectively measure the quality of such conversational agents, a model-based automatic dialogue evaluation metric (ADEM) is expected to perform well across multiple domains. Despite significant progress, an ADEM that works well in one domain does not necessarily generalize to another. This calls for a dedicated network architecture for domain generalization. To tackle the multi-domain dialogue evaluation task, we propose a Panel of Experts (PoE), a multitask network that consists of a shared transformer encoder and a collection of lightweight adapters. The shared encoder captures the general knowledge of dialogues across domains, while each adapter specializes in one specific domain and serves as a domain expert. To validate the idea, we construct a high-quality multi-domain dialogue dataset leveraging data augmentation and pseudo-labeling. The PoE network is comprehensively assessed on 16 dialogue evaluation datasets spanning a wide range of dialogue domains. It achieves state-of-the-art performance in terms of mean Spearman correlation over all the evaluation datasets. It exhibits better zero-shot generalization than existing state-of-the-art ADEMs and the ability to easily adapt to new domains with few-shot transfer learning. Comment: Currently under review at TASLP, upload to arxiv for easy cross-reference |
|---|---|
| Keywords | Computer Science - Computation and Language |
| Subject code | 006 |
| Publishing date | 2022-12-17 |
| Publishing country | us |
| Document type | Book ; Online |
| Database | BASE - Bielefeld Academic Search Engine (life sciences selection) |
Full text online
More links
Kategorien
Inter-library loan at ZB MED
Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.