キャラクターの表現を学ぶことで会話の理解が深まります

この研究では、キャラクター表現学習を活用して対話生成を強化する新しいアプローチを紹介します。私たちのモデルは中国の小説の大規模なデータセットでトレーニングされ、物語内の登場人物間の関係を明示的にモデル化します。これにより、さまざまなキャラクターの動機や相互作用を考慮して、モデルがより一貫性のある合理的な対話を生成できるようになります。マスクされた対話生成と対話話者認識という 2 つの対話生成タスクでモデルを評価し、強力なベースラインよりも優れていることを示します。私たちの研究は、対話生成のためのキャラクター表現学習の重要性を強調し、将来の研究に新たな道を切り開きます。

Authors:

著者:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Related Works DIALSTORY Dataset Proposed Tasks Methodology Experiments Discussion Future Work Conclusion Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Fluency: Grammatical correctness and intra-sentence linguistic quality.

(4) Minlie Huang、CoAI グループ、清華大学、北京、中国、清華大学、北京、北京国立情報科学技術研究センター。リンク表 要約とイントロ 関連作品 DIALSTORY データセット 提案されたタスク 方法論 実験 ディスカッション 今後の作業 結論 制限事項と参考資料 6 実験 6.1 マスクされた対話生成の実装の詳細 マスクされた対話生成タスクの実験を行うために、次のパフォーマンスに基づいてハイパーパラメーターを決定します。検証セット。シャオらを訓練します。 (2021) の BART モデルは、1 日あたり 1e-4 の学習率で 4.6 エポック、私たちのモデルでは、1 日あたり 1e-4 の学習率で 5.6 エポックでトレーニングします。すべてのベースラインとモデルは、Adam オプティマイザーを使用してトレーニングされます。私たちのメソッドのトレーニング プロセス中に、単一のストーリー内のキャラクターの選択カバレッジを計算しました。また、1,000 トレーニング ステップごとに、さまざまなキャラクターのカバレッジが 98.64% から 99.00% の範囲であることがわかりました。これは、ほぼすべてのキャラクターがトレーニング中に選択され、すべてのキャラクターが生成されたダイアログに貢献していることを意味します。さらに、式の argmax が次のことを証明しました。 3 の操作は、このタスクのトレーニング時に勾配の進行を中断しません。自動評価 以前の作業に続き、広く使用されているいくつかの標準的な自動評価メトリクスを使用します。 BLEUn (Papineni et al., 2002) を使用して、生成された各対話ターンとグラウンドトゥルースの対話ターン (n=1,2) の間の平均単語の重複を測定し、Distinct-n (Li et al., 2015) を使用して n グラムの多様性を評価します。生成された対話ターンの数 (n=2,3,4)。より具体的には、一貫性分類器の場合、会話の順番をランダムにシャッフルし、他のコンテンツを正しい順序に保つことによってトレーニング セットと検証セットを構築します。混乱したストーリーをネガティブな例、元のストーリーをポジティブな例とみなします。 Guan らの小説からさらに 195,000 の物語 (DIALSTORY を除く) をサンプリングします。 (2022) トレーニング セット (190,000 例) と検証セット (5,000 例) を構築します。 Adam オプティマイザーを使用して、2e-5 学習率と 16 バッチ サイズで 4 エポックのモデルをトレーニングします。評価中、分類器によって予測されたコヒーレントである確率が 0.5 より大きい場合、サンプルはコヒーレントであると見なされます。生成されたすべての出力に対する、分類器によって一貫性があると分類された出力 (入力とともに) の比率を一貫性スコアとして使用します。自動評価の結果を表 3 に示します。表によると、BART ベースラインと比較して、私たちのモデルは一貫してグランド トゥルースとのより多くの単語の重複を生成し、文字表現の指導の下でより優れた多様性を実現しています。つまり、私たちのモデルは、より多様ではありますが、一般的な反応ではありません。図 3 は、マスクされた対話のターン数に応じて変化する一貫性スコアをプロットしています。結果は、1 つのストーリーで 7 ターンを超える会話を生成する必要がある場合、私たちのモデルが BART よりも高い一貫性スコアを獲得することを示しています。手動評価 モデルと BART ベースラインの間でペアごとの比較を実行します。テスト セットから 100 個の例をランダムに選択します。入力と出力のペアごとに、3 人のアノテーターに、流暢さ、一貫性、有益性の観点から優先順位 (勝ち、負け、引き分け) を付けるように依頼します。注釈はすべて中国語のネイティブスピーカーです。アノテーター間で最終決定を下すために多数決を採用します。手動評価の 3 つの側面は次のとおりです。 流暢さ: 文法の正しさと文内の言語品質。

Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

一貫性: 文間の関連性、因果関係および時間的依存関係。表 5 の基準に従って、ストーリーと会話ターンの間の一貫性を判断します。ストーリー内で生成されたすべての会話ターンのスコアを加算して、ストーリー全体の一貫性スコアを取得し、それを使用してそれぞれの会話ターンを比較します。他の。

Informativeness: Interesting, diverse and rich details. As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. This paper is available on arxiv under CC 4.0 DEED license. [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base Authors:

著者:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Authors:

著者:

Authors:

著者:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan、CoAI グループ、清華大学、北京、中国、清華大学コンピュータ科学技術学部、北京国立情報科学技術研究センター。

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Abstract and Intro Related Works Related Works DIALSTORY Dataset DIALSTORY Dataset Proposed Tasks Proposed Tasks Methodology Methodology Experiments Experiments Discussion Discussion Future Work Future Work Conclusion Conclusion Limitations and References Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. Implementation Details During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). Automatic Evaluation To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Manual Evaluation Fluency: Grammatical correctness and intra-sentence linguistic quality. Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

一貫性: 文間の関連性、因果関係および時間的依存関係。表 5 の基準に従って、ストーリーと会話ターンの間の一貫性を判断します。ストーリー内で生成されたすべての会話ターンのスコアを加算して、ストーリー全体の一貫性スコアを取得し、それを使用してそれぞれの会話ターンを比較します。他の。

Informativeness: Interesting, diverse and rich details. Fluency: Grammatical correctness and intra-sentence linguistic quality. Fluency : Grammatical correctness and intra-sentence linguistic quality. Fluency Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

一貫性: 文間の関連性、因果関係および時間的依存関係。表 5 の基準に従って、ストーリーと会話ターンの間の一貫性を判断します。ストーリー内で生成されたすべての会話ターンのスコアを加算して、ストーリー全体の一貫性スコアを取得し、それを使用してそれぞれの会話ターンを比較します。他の。

Coherence : Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other. Coherence Informativeness: Interesting, diverse and rich details. Informativeness : Interesting, diverse and rich details. Informativeness As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. Case Study We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Metrics Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. Results This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base