캐릭터 표현을 학습하면 대화 이해도가 향상됩니다.

본 연구에서는 캐릭터 표현 학습을 활용하여 대화 생성을 향상시키는 새로운 접근 방식을 제시합니다. 우리 모델은 중국 소설의 대규모 데이터세트로 훈련되었으며 이야기 내 등장인물 간의 관계를 명시적으로 모델링합니다. 이를 통해 우리 모델은 다양한 캐릭터의 동기와 상호 작용을 고려하여 보다 일관되고 합리적인 대화를 생성할 수 있습니다. 우리는 마스크된 대화 생성과 대화 화자 인식이라는 두 가지 대화 생성 작업에 대한 모델을 평가하고 강력한 기준보다 우월함을 보여줍니다. 우리의 작업은 대화 생성을 위한 문자 표현 학습의 중요성을 강조하고 향후 연구를 위한 새로운 길을 열어줍니다.

Authors:

저자:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부 중국 베이징 국가 정보 과학 기술 연구 센터;

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부, 중국 베이징 국가 정보 과학 기술 연구 센터;

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan, 중국 베이징 칭화대학교 CoAI 그룹, 중국 베이징 칭화대학교 컴퓨터 과학 기술부 베이징 국가 정보 과학 기술 연구 센터;

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Related Works DIALSTORY Dataset Proposed Tasks Methodology Experiments Discussion Future Work Conclusion Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Fluency: Grammatical correctness and intra-sentence linguistic quality.

(4) Minlie Huang, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부, 중국 베이징 국가 정보 과학 기술 연구 센터. 링크 테이블 Abstract 및 Intro 관련 연구 DIALSTORY Dataset 제안된 작업 방법론 실험 토론 향후 작업 결론 제한 사항 및 참조 6 실험 6.1 Masked Dialogue Generation 구현 세부 사항 Masked Dialogue Generation 작업에 대한 실험을 수행하기 위해 성능을 기반으로 하이퍼 매개변수를 결정합니다. 검증 세트. 우리는 Shao 등을 훈련합니다. (2021)의 BART 모델은 1일 동안 1e-4 학습 속도로 4.6 epoch 동안 학습되었으며, 우리 모델의 경우 1일 동안 1e-4 학습 속도로 5.6 epoch 동안 훈련했습니다. 모든 기준선과 모델은 Adam 최적화 프로그램을 사용하여 학습되었습니다. 우리 방법의 훈련 과정에서 우리는 단일 스토리 내에서 캐릭터의 선택 범위를 계산했습니다. 그리고 1000개의 훈련 단계마다 다양한 캐릭터의 적용 범위가 98.64%에서 99.00%에 이르는 것으로 나타났습니다. 이는 훈련 중에 거의 모든 캐릭터가 선택되고 모든 캐릭터가 생성된 대화에 기여했음을 의미합니다. 또한 Eq.의 argmax가 입증되었습니다. 3 작업은 이 작업을 훈련할 때 그라데이션 진행을 중단하지 않습니다. 자동 평가 이전 연구에 이어 우리는 널리 사용되는 몇 가지 표준 자동 평가 지표를 사용합니다. 우리는 BLEUn(Papineni et al., 2002)을 사용하여 생성된 대화와 실제 대화 차례(n=1,2) 사이의 평균 단어 중첩을 측정하고 Distinct-n(Li et al., 2015)을 사용하여 n-gram 다양성을 평가합니다. 생성된 대화 턴 수(n=2,3,4). 좀 더 구체적으로 말하자면, 일관성 분류기의 경우 대화 순서를 무작위로 섞고 다른 콘텐츠를 올바른 순서로 유지하여 훈련 및 검증 세트를 구성합니다. 우리는 혼란스러운 이야기를 부정적인 사례로, 원작의 이야기를 긍정적인 사례로 간주합니다. 우리는 Guan et al.의 소설에서 또 다른 195,000개의 이야기(DIALSTORY의 이야기 제외)를 샘플링합니다. (2022) 훈련 세트(190,000개 예시)와 검증 세트(5,000개 예시)를 구성합니다. Adam 최적화 프로그램을 사용하여 2e-5 학습률과 16배치 크기로 4개 에포크 동안 모델을 훈련합니다. 평가하는 동안 분류기에 의해 예측된 일관성이 있을 확률이 0.5보다 큰 경우 일관성 있는 예시로 간주합니다. 생성된 모든 출력에 대한 분류기에 의해 일관성 있는 것으로 분류된 출력(입력과 함께)의 비율을 일관성 점수로 사용합니다. 자동 평가 결과는 표 3에 나와 있습니다. 표에 따르면, BART 기준과 비교하여 우리 모델은 지속적으로 지상 진실과 더 많은 단어 중복을 생성하고 문자 표현의 지침에 따라 더 나은 다양성을 달성합니다. 더 다양하지만 평범하지 않은 반응. 그림 3은 마스크된 대화 차례 수에 따라 달라지는 일관성 점수를 보여줍니다. 결과는 하나의 스토리에서 7턴 이상의 대화를 생성해야 할 때 우리 모델이 BART보다 더 높은 일관성 점수를 얻는다는 것을 보여줍니다. 수동 평가 우리는 모델과 BART 기준선을 쌍으로 비교합니다. 테스트 세트에서 무작위로 100개의 예를 선택합니다. 입력과 함께 출력의 각 쌍에 대해 세 명의 주석자에게 유창성, 일관성 및 정보성 측면에서 선호도(승리, 패배, 동점)를 부여하도록 요청합니다. 모든 주석은 중국어 원어민입니다. 우리는 주석자 사이에서 최종 결정을 내리기 위해 다수결 투표를 채택합니다. 수동 평가의 세 가지 측면은 다음과 같습니다. 유창성: 문법적 정확성 및 문장 내 언어 품질.

Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

일관성: 문장 간 관련성, 인과관계 및 시간적 종속성. 표 5의 기준에 따라 스토리와 대화 차례 사이의 일관성을 판단합니다. 스토리에서 생성된 모든 대화 차례의 점수를 합산하여 스토리의 전체 일관성 점수를 얻은 다음 이를 비교하는 데 사용됩니다. 다른.

Informativeness: Interesting, diverse and rich details. As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. This paper is available on arxiv under CC 4.0 DEED license. [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base Authors:

저자:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부 중국 베이징 국가 정보 과학 기술 연구 센터;

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부, 중국 베이징 국가 정보 과학 기술 연구 센터;

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan, 중국 베이징 칭화대학교 CoAI 그룹, 중국 베이징 칭화대학교 컴퓨터 과학 기술부 베이징 국가 정보 과학 기술 연구 센터;

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Authors:

저자:

Authors:

저자:

(1) Jianzhu Yao, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(1) Jianzhu Yao, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부 중국 베이징 국가 정보 과학 기술 연구 센터;

(2) Ziqi Liu, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(2) Ziqi Liu, CoAI 그룹, 중국 베이징 칭화 대학교 컴퓨터 과학 기술부, 중국 베이징 국가 정보 과학 기술 연구 센터;

(3) Jian Guan, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology;

(3) Jian Guan, 중국 베이징 칭화대학교 CoAI 그룹, 중국 베이징 칭화대학교 컴퓨터 과학 기술부 베이징 국가 정보 과학 기술 연구 센터;

(4) Minlie Huang, The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology. Table of Links Abstract and Intro Abstract and Intro Related Works Related Works DIALSTORY Dataset DIALSTORY Dataset Proposed Tasks Proposed Tasks Methodology Methodology Experiments Experiments Discussion Discussion Future Work Future Work Conclusion Conclusion Limitations and References Limitations and References 6 Experiments 6.1 Masked Dialogue Generation Implementation Details To conduct experiments on the masked dialogue generation task, we decide the hyper-parameters based on the performance of the validation set. We train Shao et al. (2021)’s BART model for 4.6 epochs with a 1e-4 learning rate for 1 day, and for our model, we train it for 5.6 epochs with a 1e-4 learning rate for 1 day. All baselines and our model are trained using the Adam optimizer. Implementation Details During the training process for our method, we computed the selection coverage of characters within a single story. And it showed that in every 1000 training steps, the coverage of different characters ranged from 98.64% to 99.00%, which meant nearly all the characters are selected during training, and all the characters contributed to the generated dialogue. It further proved that the argmax in Eq. 3 operation doesn’t break the gradient progress when training for this task. Automatic Evaluation Following previous works, we use several standard, widely used automatic evaluation metrics. We use BLEUn (Papineni et al., 2002) to measure the average word overlap between each generated and groundtruth dialogue turn (n=1,2), and Distinct-n (Li et al., 2015) to evaluate n-gram diversity of generated dialogue turns (n=2,3,4). Automatic Evaluation To be more specific, for the coherence classifier, we construct the training and validation sets by randomly shuffling the order of dialogue turns and keeping other content in the correct order. We regard the perturbed story as a negative example and the original story as a positive example. We sample another 195k stories (except those in DIALSTORY) from the novels of Guan et al. (2022) to construct the training set (190k examples) and the validation set (5k examples). We train the model for 4 epochs with a 2e-5 learning rate and a 16- batch size, using the Adam optimizer. During the evaluation, we consider an example coherent when the probability of being coherent predicted by the classifier is greater than 0.5. We use the ratio of outputs (along with the input) that are classified as coherent by the classifier to all generated outputs as the coherence score. The result of the automatic evaluation is presented in Table 3. According to the table, compared to the BART baseline, our model consistently generates more word overlaps with ground truth and achieves better diversity under the guidance of character representations, which means our model can generate more diverse but not commonplace responses. Figure 3 plots the coherence score varying with the number of masked dialogue turns. The result shows that our model gets a higher coherence score than BART when required to generate more than seven turns of dialogue in one story. Manual Evaluation We conduct a pairwise comparison between our model and the BART baseline. We randomly select 100 examples from the test set. For each pair of outputs along with the input, we ask three annotators to give a preference (win, lose and tie) in terms of fluency, coherence, and informativeness. All the annotations are native Chinese speakers. We adopt majority voting to make final decisions among the annotators. The three aspects of manual evaluation are as follows: Manual Evaluation Fluency: Grammatical correctness and intra-sentence linguistic quality. Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

일관성: 문장 간 관련성, 인과관계 및 시간적 종속성. 표 5의 기준에 따라 스토리와 대화 차례 사이의 일관성을 판단합니다. 스토리에서 생성된 모든 대화 차례의 점수를 합산하여 스토리의 전체 일관성 점수를 얻은 다음 이를 비교하는 데 사용됩니다. 다른.

Informativeness: Interesting, diverse and rich details. Fluency: Grammatical correctness and intra-sentence linguistic quality. Fluency : Grammatical correctness and intra-sentence linguistic quality. Fluency Coherence: Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other.

일관성: 문장 간 관련성, 인과관계 및 시간적 종속성. 표 5의 기준에 따라 스토리와 대화 차례 사이의 일관성을 판단합니다. 스토리에서 생성된 모든 대화 차례의 점수를 합산하여 스토리의 전체 일관성 점수를 얻은 다음 이를 비교하는 데 사용됩니다. 다른.

Coherence : Inter-sentence relatedness, causal and temporal dependencies. We judge the coherence between the story and a dialogue turn by following the criterion in Table 5. We add the scores of all the generated dialogue turns in a story to get the overall coherence score of the story, which is then used to compare with each other. Coherence Informativeness: Interesting, diverse and rich details. Informativeness : Interesting, diverse and rich details. Informativeness As shown in Table 4, all the results show moderate (κ > 0.4) agreement, which shows our model outperforms the BART baseline significantly in dialogue informativeness and coherence. Case Study Figure 5 showed two examples to investigate how learning character representations can help our model generate more coherent dialogue. We found that our model can better model the relationship between different characters and the direction of the storyline. For example, in the first case, we can see that the BART’s generation confuses different characters’ fathers, while our model captures the relationship between different characters, and generates proper responses for the corresponding characters, which also moves the plot forward. And in the second case, we can see that BART’s generation is commonplace and contradicts the plot development. In contrast, our model captures the intentions of the speaker and the development trend of the plot, generating an appropriate and coherent response. Since these two models use the same pretrained weight, we can infer that the character modeling module leverages the coherent and reasonable generation. Case Study We also summarize four error types of the generated dialogue turn for the DialGen task: (1) Intersentence Contradiction; (2) Inter-sentence Repetition; (3) Intra-sentence Contradiction; (4) Intrasentence Repetition. We show the typical corresponding cases in Figure 4. We conducted a quantitative analysis of those 4 error types on our model’s generation. We analyzed 20 stories with 103 dialog turns and the results are shown in Figure 6. We found that both our model and BART suffer from these errors, suggesting that there is still space for model improvement, especially in the inter-sentence repetition. 6.2 Dialogue Speaker Recognition Implementation Details To conduct experiments on the speaker recognition task, we decide the hyper-parameters based on the performance of the validation set. For the BART baseline and our approach, we insert a mask token before each dialogue needed to be predicted, and a person id token before and after each character name span. Then, we insert all the unique person id tokens before the input stories as different options, and make predictions based on the cos similarity of option tokens and mask tokens. We train Shao et al. (2021)’s BART model for 30 epochs with a 5e-5 learning rate for 3 days, For encoder-only baselines, we implemented BERT, RoBERTa, and MacBERT and trained them for 15 epochs wity a le-5 learning rate for 2 days. For our model, we train it for 22 epochs with a le-6 learning rate for 2 days. All baselines and our model are trained using the Adam optimizer. Metrics We evaluate the DialSpk task using two automatic metrics including dialogue-level accuracy (DAC) and story-level accuracy (SAC). DAC is calculated as the ratio of the correct predictions to the total number of specifies dialogue turns, while SAC is the ratio of the number of stories where all dialogue turns are correctly predicted to the number of all test examples. These two metrics provide the evaluation for dialogue understanding with different granularities. Metrics Results As shown in Table 7, our model outperforms all the baselines significantly (p< 0.01, Wilcoxon signed-rank test) on both DAS and SAC scores, suggesting the benefit of learning character representations. We tested the accuracy of automatic training set annotations, and the DAC/SAC scores are 86.78%/67.80%. Together with the model’s performance on the test set, we can see the automatic annotation for the training set is of good quality. We also conducted the human prediction experiment, and the DAC/SAC scores are 97.90%/90.70%, which are much higher than the best model. So there is much room for further improvement for machine-based approaches. Results This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv [2] https://huggingface.co/fnlp/ bart-base-chinese [3] https://huggingface.co/bert-base-chinese [4] https://huggingface.co/hfl/ chinese-roberta-wwm-ext [5] https://huggingface.co/hfl/ chinese-macbert-base