Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 22, No. 0, pp.1101-1115
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Print publication date 31 Jan 2022
Received 25 Aug 2022 Revised 20 Oct 2022 Accepted 30 Oct 2022
DOI: https://doi.org/10.15738/kjell.22..202210.1101

An Experimental Investigation of Discourse Expectations in Neural Language Models

Eunkyung Yi ; Hyowon Cho ; Sanghoun Song
(1st author) Assistant Professor, Dept. of English Education, Ewha Womans University, Tel: 02) 3277-4699 eyi@ewha.ac.kr
(co-author) Undergraduate Student, Dept. of Linguistics, Korea University, Tel: 02) 3290-2170 snhan9658@naver.com
(corresponding author) Associate Professor, Dept. of Linguistics, Korea University, Tel: 02) 3290-2177 sanghoun@korea.ac.kr


© 2022 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium provided the original work is properly cited.

Abstract

The present study reports on three language processing experiments with most up-to-date neural language models from a psycholinguistic perspective. We investigated whether and how discourse expectations demonstrated in the psycholinguistics literature are manifested in neural language models, using the language models whose architectures and assumptions are considered most appropriate for the given language processing tasks. We first attempted to perform a general assessment of a neural model’s discourse expectations about story continuity or coherence (Experiment 1), based on the next sentence prediction module of the bidirectional transformer-based model BERT (Devlin et al. 2019). We also studied language models’ expectations about reference continuity in discursive contexts in both comprehension (Experiment 2) and production (Experiment 3) settings, based on so-called Implicit Causality biases. We used the unidirectional (or left-to-right) RNN-based model LSTM (Hochreiter and Schmidhuber 1997) and the transformer-based generation model GPT-2 (Radford et al. 2019), respectively. The results of the three experiments showed, first, that neural language models are highly successful in distinguishing between reasonably expected and unexpected story continuations in human communication and also that they exhibit human-like bias patterns in reference expectations in both comprehension and production contexts. The results of the present study suggest language models can closely simulate the discourse processing features observed in psycholinguistic experiments with human speakers. The results also suggest language models can, beyond simply functioning as a technology for practical purposes, serve as a useful research tool and/or object for the study of human discourse processing.

Keywords:

discourse expectation, implicit causality bias, neural language model, BERT, GPT-2, LSTM, next sentence prediction, coreference resolution, surprisal

Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2020S1A5A2A03042760). We thank the anonymous reviewers for their valuable comments. We also would like to thank Unsub Shin for his feedback on an earlier draft. Any remaining errors are solely our responsibility.

References

  • Caramazza, A., E. Grober, C. Garvey and J. Yates. 1977. Comprehension of anaphoric pronouns. Journal of Verbal Learning and Verbal Behavior 16, 601-609. [https://doi.org/10.1016/S0022-5371(77)80022-4]
  • Davis, F. and M. van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, 396-407. November 19-20. Association for Computational Linguistics. [https://doi.org/10.18653/v1/2020.conll-1.32]
  • Garvey, C. and A. Caramazza. 1974. Implicit causality in verbs. Linguistic Inquiry 5(3), 459-464.
  • Goodfellow, I., Y. Bengio and A. Courville. (2016). Deep Learning. MIT press.
  • Hale, J. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, 1-8. Association for Computational Linguistics. [https://doi.org/10.3115/1073336.1073357]
  • Hobbs, J. R. 1979. Coherence and coreference. Cognitive Science 3, 67-90. [https://doi.org/10.1207/s15516709cog0301_4]
  • Hochreiter, S. and J. Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8), 1735-1780. [https://doi.org/10.1162/neco.1997.9.8.1735]
  • Jin, X., X. Wang, X. Luo, S. Huang and S. Gu. 2020. Inter-sentence and Implicit Causality extraction from Chinese Corpus. In H. Lauw, R. W. Wong, A. Ntoulas, E. P. Lim, S. K. Ng and S. Pan, eds., Advances in Knowledge Discovery and Data Mining, 739-751. Springer, Cham. [https://doi.org/10.1007/978-3-030-47426-3_57]
  • Jeretic, P., A. Warstadt, S. Bhooshan and A. Williams. 2020. Are natural language inference models impressive? Leanring implicature and presupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8690-8705. Association for Computational Linguistics. [https://doi.org/10.18653/v1/2020.acl-main.768]
  • Jurafsky, D. and J. H. Martin. To appear. Speech and Language Processing. 3rd ed. Prentice Hall.
  • Khetan, V., R. Ramnani, M. Anand, S. Sengupta and A. E. Fano. 2022. Causal-BERT: Language models for causality detection between events expressed in text. In K. Arai ed., Intelligent Computing, 965-980. Springer, Cham. [https://doi.org/10.1007/978-3-030-80119-9_64]
  • Kishimoto, Y., Y. Murawaki and S. Kurohashi. 2020. Adapting BERT to implicit discourse relation classification with a focus on discourse connectives. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 1152-1158.
  • Liang, S., Z. Wanli, Z. Shi, S. Wang, J. Wang and X. Zuo. 2022. A multi-level neural network for implicit causality detection in web texts. Neurocomputing 481, 121-132. [https://doi.org/10.1016/j.neucom.2022.01.076]
  • Linzen, T., E. Dupoux and Y. Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In Transactions of the Association for Computational Linguistics 4, 521-535. [https://doi.org/10.1162/tacl_a_00115]
  • Mann, W. C. and S. A. Thompson. 1987. Rhetorical structure theory: A theory of text organization. Technical Report RS-87-190, Information Sciences Institute.
  • Miltsakaki, E., R. Prasad, A. K. Joshi and B. L. Webber. 2004. The Penn Discourse Treebank. LREC.
  • Pickering, M. J. and C. Gambi. 2018. Predicting while comprehending language: a theory and review. Psychological Bulletin 144(10), 1002-1044. [https://doi.org/10.1037/bul0000158]
  • Prasad, R., N. Dinesh, A. Lee, E. Milt- sakaki, L. Robaldo, A. K. Joshi and B. L. Webber. 2008. The Penn Discourse TreeBank 2.0. LREC.
  • Prasad, R., B. L. Webber and A. Joshi. 2014. Reflections on the Penn Discourse Treebank, comparable corpora, and complementary annotation. Computational Linguistics 40(4), 921-950. [https://doi.org/10.1162/COLI_a_00204]
  • Radford, A., J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI tech report.
  • Radford, A., K. Narasimhan, T. Salimans and I. Sutskever. 2018. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/languageunsupervised/language understanding paper. pdf
  • Rohde, H. 2008. Coherence-driven Effects in Sentence and Discourse Processing. Unpublished doctoral dissertation, University of California, San Diego.
  • Rohde, H., R. Levy and A. Kehler. 2011. Anticipating explanations in relative clause processing. Cognition 118, 339-358. [https://doi.org/10.1016/j.cognition.2010.10.016]
  • Shi, W. and V. Demberg. 2019. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019, Poster), November 3-7, 2019, Hong Kong, China. [https://doi.org/10.18653/v1/D19-1586]
  • Stewart, A. J., M. J. Pickering and A. J. Sanford (1998). Implicit consequentiality. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society, 1031-1036. Mahwah, N: Lawrence Erlbaum Associates. [https://doi.org/10.4324/9781315782416-186]
  • Yi, E. and J.-P. Koenig. 2021. Grammar modulates discourse expectations: evidence from causal relations in English and Korean. Language and Cognition 13(1), 99-127. [https://doi.org/10.1017/langcog.2020.29]