学习角色表征可以增强对话理解

在这项研究中，我们提出了一种通过利用角色表示学习来增强对话生成的新颖方法。我们的模型是在中国小说的大规模数据集上进行训练的，并明确地模拟了故事中人物之间的关系。这使得我们的模型能够通过考虑不同角色的动机和互动来生成更加连贯和合理的对话。我们在两个对话生成任务（蒙面对话生成和对话说话者识别）上评估我们的模型，并证明其相对于强大基线的优越性。我们的工作强调了角色表征学习对于对话生成的重要性，并为未来的研究开辟了新的途径。

Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清华大学 CoAI 课题组，中国北京清华大学计算机科学与技术系，北京信息科学技术国家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清华大学 CoAI 组，中国北京清华大学计算机科学与技术系，中国北京北京信息科学技术国家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 关健，清华大学计算机科学与技术系，北京，中国北京信息科学技术国家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Related Works DIALSTORY Dataset Proposed Tasks Methodology Experiments Discussion Future Work Conclusion Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Fluency: Grammatical correctness and intra-sentence linguistic quality.

(4) 黄敏烈，CoAI 小组，清华大学，北京，中国 清华大学计算机科学与技术系，北京，中国 北京信息科学与技术国家研究中心。链接表 摘要和介绍 相关工作 DIALSTORY 数据集 建议的任务 方法论 实验讨论 未来工作 结论 局限性和参考 6 实验 6.1 蒙面对话生成实现细节 为了对蒙面对话生成任务进行实验，我们根据以下性能决定超参数：验证集。我们训练 Shao 等人。 (2021) 的 BART 模型使用 1 天的 1e-4 学习率进行 4.6 个周期的训练，对于我们的模型，我们使用 1 天的 1e-4 学习率对其进行 5.6 个周期的训练。所有基线和我们的模型都是使用 Adam 优化器进行训练的。在我们的方法的训练过程中，我们计算了单个故事中角色的选择覆盖率。结果表明，在每 1000 个训练步骤中，不同角色的覆盖率在 98.64% 到 99.00% 之间，这意味着在训练过程中几乎选择了所有角色，并且所有角色都对生成的对话做出了贡献。进一步证明了方程中的argmax。 3 操作在训练该任务时不会破坏梯度进度。自动评估继之前的工作之后，我们使用了几种标准的、广泛使用的自动评估指标。我们使用 BLEUn (Papineni et al., 2002) 来测量每个生成对话轮和真实对话轮之间的平均单词重叠 (n=1,2)，并使用 Distinct-n (Li et al., 2015) 来评估 n-gram 多样性生成的对话轮数 (n=2,3,4)。更具体地说，对于连贯性分类器，我们通过随机打乱对话顺序并保持其他内容的正确顺序来构建训练集和验证集。我们把扰动的故事看成反面例子，把原来的故事看成正面例子。我们从关等人的小说中抽取了另外 195,000 个故事（《日记故事》中的故事除外）。 (2022) 构建训练集（190k 个示例）和验证集（5k 个示例）。我们使用 Adam 优化器以 2e-5 的学习率和 16 批次大小对模型进行 4 个 epoch 的训练。在评估过程中，当分类器预测的相干概率大于 0.5 时，我们认为示例是相干的。我们使用被分类器分类为相干的输出（以及输入）与所有生成的输出的比率作为相干性分数。自动评估的结果如表3所示。根据该表，与BART基线相比，我们的模型始终生成更多与真实值重叠的单词，并在字符表示的指导下实现更好的多样性，这意味着我们的模型可以生成更加多样化但不常见的反应。图 3 绘制了随屏蔽对话轮数变化的连贯性得分。结果表明，当需要在一个故事中生成七轮以上对话时，我们的模型比 BART 获得更高的连贯性分数。手动评估 我们对我们的模型和 BART 基线进行了成对比较。我们从测试集中随机选择 100 个示例。对于每对输出和输入，我们要求三位注释者在流畅性、连贯性和信息性方面给出偏好（获胜、失败和平局）。所有注释都是以中文为母语的。我们采用多数投票的方式在注释者中做出最终决定。人工评价的三个方面如下： 流畅性：语法正确性和句内语言质量。

Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

连贯性：句子间的相关性、因果关系和时间依赖性。我们按照表5中的标准来判断故事和对话回合之间的连贯性。我们将故事中所有生成的对话回合的分数相加，得到故事的整体连贯性分数，然后将其与每个对话回合进行比较其他。

Informativeness: Interesting, diverse and rich details. As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. This paper is available on arxiv under CC 4.0 DEED license. [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清华大学 CoAI 课题组，中国北京清华大学计算机科学与技术系，北京信息科学技术国家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清华大学 CoAI 组，中国北京清华大学计算机科学与技术系，中国北京北京信息科学技术国家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 关健，清华大学计算机科学与技术系，北京，中国北京信息科学技术国家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Authors:

作者：

Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清华大学 CoAI 课题组，中国北京清华大学计算机科学与技术系，北京信息科学技术国家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清华大学 CoAI 组，中国北京清华大学计算机科学与技术系，中国北京北京信息科学技术国家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 关健，清华大学计算机科学与技术系，北京，中国北京信息科学技术国家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Abstract and Intro Related Works Related Works DIALSTORY Dataset DIALSTORY Dataset Proposed Tasks Proposed Tasks Methodology Methodology Experiments Experiments Discussion Discussion Future Work Future Work Conclusion Conclusion Limitations and References Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. Implementation Details During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). Automatic Evaluation To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Manual Evaluation Fluency: Grammatical correctness and intra-sentence linguistic quality. Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

连贯性：句子间的相关性、因果关系和时间依赖性。我们按照表5中的标准来判断故事和对话回合之间的连贯性。我们将故事中所有生成的对话回合的分数相加，得到故事的整体连贯性分数，然后将其与每个对话回合进行比较其他。

Informativeness: Interesting, diverse and rich details. Fluency: Grammatical correctness and intra-sentence linguistic quality. Fluency : Grammatical correctness and intra-sentence linguistic quality. Fluency Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

连贯性：句子间的相关性、因果关系和时间依赖性。我们按照表5中的标准来判断故事和对话回合之间的连贯性。我们将故事中所有生成的对话回合的分数相加，得到故事的整体连贯性分数，然后将其与每个对话回合进行比较其他。

Coherence : Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other. Coherence Informativeness: Interesting, diverse and rich details. Informativeness : Interesting, diverse and rich details. Informativeness As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. Case Study We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Metrics Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. Results This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base