學習角色表徵可以增強對話理解

在這項研究中，我們提出了一種透過利用角色表示學習來增強對話生成的新穎方法。我們的模型是在中國小說的大規模資料集上進行訓練的，並明確地模擬了故事中人物之間的關係。這使得我們的模型能夠透過考慮不同角色的動機和互動來產生更連貫和合理的對話。我們在兩個對話生成任務（蒙面對話生成和對話說話者識別）上評估我們的模型，並證明其相對於強大基線的優越性。我們的工作強調了角色表徵學習對於對話生成的重要性，並為未來的研究開闢了新的途徑。

Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清華大學 CoAI 課題組，中國北京清華大學計算機科學與技術系，北京資訊科學技術國家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清華大學 CoAI 組，中國北京清華大學計算機科學與技術系，中國北京北京資訊科學技術國家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 關健，清華大學電腦科學與技術系，北京，中國北京資訊科學技術國家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Related Works DIALSTORY Dataset Proposed Tasks Methodology Experiments Discussion Future Work Conclusion Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Fluency: Grammatical correctness and intra-sentence linguistic quality.

(4) 黃敏烈，CoAI 小組，清華大學，北京，中國 清華大學計算機科學與技術系，北京，中國 北京資訊科學與技術國家研究中心。連結表 摘要與介紹 相關工作 DIALSTORY 資料集 建議的任務 方法論 實驗討論 未來工作 結論 限制與參考 6 實驗 6.1 蒙面對話產生實作細節 為了對蒙面對話產生任務進行實驗，我們根據以下表現決定超參數：驗證集。我們訓練 Shao 等人。 (2021) 的 BART 模型使用 1 天的 1e-4 學習率進行 4.6 個週期的訓練，對於我們的模型，我們使用 1 天的 1e-4 學習率對其進行 5.6 個週期的訓練。所有基線和我們的模型都是使用 Adam 優化器進行訓練的。在我們的方法的訓練過程中，我們計算了單一故事中角色的選擇覆蓋率。結果表明，在每1000 個訓練步驟中，不同角色的覆蓋率在98.64% 到99.00% 之間，這意味著在訓練過程中幾乎選擇了所有角色，並且所有角色都對生成的對話做出了貢獻。進一步證明了方程式中的argmax。 3 操作在訓練該任務時不會破壞梯度進度。自動評估繼先前的工作之後，我們使用了幾種標準的、廣泛使用的自動評估指標。我們使用BLEUn (Papineni et al., 2002) 來測量每個生成對話輪和真實對話輪之間的平均單字重疊(n=1,2)，並使用Distinct-n (Li et al., 2015) 來評估n-gram 多樣性產生的對話輪數 (n=2,3,4)。更具體地說，對於連貫性分類器，我們透過隨機打亂對話順序並保持其他內容的正確順序來建立訓練集和驗證集。我們把擾動的故事看成反面例子，把原來的故事看成正面例子。我們從關等人的小說中抽取了另外 195,000 個故事（《日記故事》中的故事除外）。 (2022) 建立訓練集（190k 個範例）和驗證集（5k 個範例）。我們使用 Adam 優化器以 2e-5 的學習率和 16 個批次大小對模型進行 4 個 epoch 的訓練。在評估過程中，當分類器預測的相干機率大於 0.5 時，我們認為範例是相干的。我們使用被分類器分類為相干的輸出（以及輸入）與所有產生的輸出的比率作為相干性分數。自動評估的結果如表3所示。著我們的模型可以產生更多樣化但不常見的反應。圖 3 繪製了隨屏蔽對話輪數變化的連貫性得分。結果表明，當需要在一個故事中產生七輪以上對話時，我們的模型比 BART 獲得更高的連貫性分數。手動評估 我們對我們的模型和 BART 基線進行了成對比較。我們從測試集中隨機選擇 100 個範例。對於每對輸出和輸入，我們要求三位註釋者在流暢性、連貫性和資訊性方面給出偏好（獲勝、失敗和平局）。所有註釋都是以中文為母語的。我們採用多數投票的方式在註釋者中做出最終決定。人工評價的三個面向如下： 流暢性：語法正確性和句內語言品質。

Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

連貫性：句子間的相關性、因果關係與時間依賴性。我們依照表5的標準來判斷故事與對話回合之間的連貫性。比較其他。

Informativeness: Interesting, diverse and rich details. As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. This paper is available on arxiv under CC 4.0 DEED license. [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清華大學 CoAI 課題組，中國北京清華大學計算機科學與技術系，北京資訊科學技術國家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清華大學 CoAI 組，中國北京清華大學計算機科學與技術系，中國北京北京資訊科學技術國家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 關健，清華大學電腦科學與技術系，北京，中國北京資訊科學技術國家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Authors:

作者：

Authors:

作者：

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) 姚建柱，清華大學 CoAI 課題組，中國北京清華大學計算機科學與技術系，北京資訊科學技術國家研究中心；

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu，清華大學 CoAI 組，中國北京清華大學計算機科學與技術系，中國北京北京資訊科學技術國家研究中心；

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) 關健，清華大學電腦科學與技術系，北京，中國北京資訊科學技術國家研究中心；

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Abstract and Intro Related Works Related Works DIALSTORY Dataset DIALSTORY Dataset Proposed Tasks Proposed Tasks Methodology Methodology Experiments Experiments Discussion Discussion Future Work Future Work Conclusion Conclusion Limitations and References Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. Implementation Details During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). Automatic Evaluation To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Manual Evaluation Fluency: Grammatical correctness and intra-sentence linguistic quality. Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

連貫性：句子間的相關性、因果關係與時間依賴性。我們依照表5的標準來判斷故事與對話回合之間的連貫性。比較其他。

Informativeness: Interesting, diverse and rich details. Fluency: Grammatical correctness and intra-sentence linguistic quality. Fluency : Grammatical correctness and intra-sentence linguistic quality. Fluency Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

連貫性：句子間的相關性、因果關係與時間依賴性。我們依照表5的標準來判斷故事與對話回合之間的連貫性。比較其他。

Coherence : Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other. Coherence Informativeness: Interesting, diverse and rich details. Informativeness : Interesting, diverse and rich details. Informativeness As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. Case Study We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Metrics Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. Results This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base