Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 20, No. 1, pp.881-903
ISSN: 1598-1398 (Print)
Print publication date 31 Mar 2020
Received 01 Nov 2020 Revised 30 Nov 2020 Accepted 15 Dec 2020
DOI: https://doi.org/10.15738/kjell.20..202012.881

코퍼스와 딥러닝 언어 모델을 활용한 문장 처리의 예측성과 행동 반응 시간과의 관계 연구

서혜진 ; 신정아*
동국대학교
동국대학교
Exploring the relationship between the predictability and the behavioral reaction time in sentence processing using corpus and deep-learning language models
Hye-Jin Seo ; Jeong-Ah Shin*
Dongguk University
Dongguk University

제1저자: 서혜진; 교신저자: 신정아


Copyright 2020 KASELL
This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This study examined whether the predictability is associated with the behavioral reaction times in sentence processing. The information complexity measures have been proposed to quantify the predictability for word-by-word human sentence processing. The most traditional information complexity measure is known as surprisal, which calculates relative unexpectedness at each word in a sentence (Hale 2001, Levy 2005, 2008). The most traditional information complexity measure is known as surprisal, which calculates relative unexpectedness at each word in a sentence (Hale 2001, Levy 2005, 2008), and some studies suggested that surprisal and reading times are positively correlated (Monsalve, Frank and Vigliocco 2012, Smith and Levy 2013). In order to calculate surprisal, the previous studies used one of two ways: Corpus based language models and deep learning based language models. This study, however, used both of them to analyze human reading times, comparing surprisal calculated from corpus-based language models with that calculated from deep-learning-based language models. Many studies partially investigated either of them. In this study, human reading times were analyzed by comparing surprisal calculated from corpus-based language models with that calculated from deep-learning-based language models. The results showed that surprisal calculated from corpus-based language models is more suitable to explain the behavioral reaction time data. Although the deep learning technology performs very well in the field of natural language processing, it does not seem to be human-like processing. Nonetheless, this study can contribute to the development of deep learning technology as well as computational psycholinguistic research in that it tried to compare the outcomes of corpus and deep learning technology with human behavioral responses.

Keywords:

predictabililty, surprisal, corpus-based language model, deep-learning-based language model

References

  • 신정아(Shin, J.). 2019. 혼합효과모형(Mixed-Effects Model)을 이용한 실험언어학 데이터 분석 방법 고찰: 자기조절읽기 실험 데이터를 중심으로(How to analyze experimental linguistic data using a mixed-effects model in R: Focusing on data from a self-paced reading experiment). ≪언어학≫ (Korean Journal of Journal of English Language and Linguistics) 19(1), 76-94. [https://doi.org/10.15738/kjell.19.1.201903.76]
  • Altmann, G. T. and Y. Kamide. 1999. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73(3), 247-264. [https://doi.org/10.1016/S0010-0277(99)00059-1]
  • Baayen, R. H., D. J. Davidson and D. M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59(4), 390-412. [https://doi.org/10.1016/j.jml.2007.12.005]
  • Barrett, M., Ž. Agić and A. Søgaard. 2015, January. The dundee treebank. In The 14th International Workshop on Treebanks and Linguistic Theories (TLT 14).
  • Marcus, M., B. Santorini and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19, 313–330. [https://doi.org/10.21236/ADA273556]
  • Balota, D. A., A. Pollatsek and Rayner, K. 1985. The interaction of contextual constraints and parafoveal visual information in reading. Cognitive Psychology 17, 364-390. [https://doi.org/10.1016/0010-0285(85)90013-1]
  • Baayen, R. H. and P. Milin. 2010. Analyzing reaction times. International Journal of Psychological Research 3(2), 12-28. [https://doi.org/10.21500/20112084.807]
  • Bever, T. G. 1970. The cognitive basis for linguistic structures. Cognition and the Development of Language 279, 1-61.
  • Boston, M., J. Hale, R. Kliegl, U. Patil and S. Vasishth. 2008. Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research 2(1), 1-12.
  • Chomsky, N. 1956. Three models for the description of language. IRE Transactions on Information Theory 2(3), 113-124. [https://doi.org/10.1109/TIT.1956.1056813]
  • Chomsky, N. 1965. Aspects of the Theory of Syntax. Cambridge, MA: MITPress, [https://doi.org/10.21236/AD0616323]
  • Clark, A. 2013. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences 36(3), 181-204. [https://doi.org/10.1017/S0140525X12000477]
  • Clifton, C. Jr., F. Ferreira, J. M. Henderson, A. W. Inhoff, S. P. Liversedge, E. D. Reichle and E. R. Schotter. 2016. Eye movements in reading and information processing: Keith Rayner’s 40 year legacy. Journal of Memory and Language 86, 1-19. [https://doi.org/10.1016/j.jml.2015.07.004]
  • DeLong, K. A., T. P. Urbach and M. Kutas. 2005. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience 8(8), 1117-1121. [https://doi.org/10.1038/nn1504]
  • Devlin, J., M-W. Chang, K. Lee and K. Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
  • Federmeier, K. D. 2007. Thinking ahead: The role and roots of prediction in language comprehension. Psychophysiology 44(4), 491-505. [https://doi.org/10.1111/j.1469-8986.2007.00531.x]
  • Federmeier, K. D. and M. Kutas. 1999. A rose by any other name: Long-term memory structure and sentence processing. Journal of Memory and Language 41, 469-495. [https://doi.org/10.1006/jmla.1999.2660]
  • Frank, S. 2009. Surprisal-based comparison between a symbolic and a connectionist model of sentence processing. In Proceedings of the annual meeting of the Cognitive Science Society 31, 1139-1144.
  • Frank, S. L. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in Cognitive Science 5(3), 475-494. [https://doi.org/10.1111/tops.12025]
  • Frank, S. L., L. J. Otten, G. Galli and G. Vigliocco. 2015. The ERP response to the amount of information conveyed by words in sentences. Brain and Language 140, 1-11. [https://doi.org/10.1016/j.bandl.2014.10.006]
  • Futrell, R., E. Wilcox, T. Morita and R. Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint arXiv:1809. 01329.
  • Garnsey, S., N. Pearlmutter, E. Myers and M. Lotocky. 1997. The contributions of verb bias and plausibility to the comprehension of temporarily ambiguous sentences. Journal of Memory and Language 37(1), 58-93. [https://doi.org/10.1006/jmla.1997.2512]
  • Goodkind, A., and K. Bicknell. 2018, January. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th workshop on cognitive modeling and computational linguistics CMCL 2018, 10-18. [https://doi.org/10.18653/v1/W18-0102]
  • Gulordava, K., P. Bojanowski, E. Grave, T. Linzen and M. Baroni. 2018. Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138 [https://doi.org/10.18653/v1/N18-1108]
  • Hale, J. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second of the North American Chapter of the Association for Computational Linguistics on Language, 1-8, Stroudsburg, PA: Association for Computational Linguistics. [https://doi.org/10.3115/1073336.1073357]
  • Hale, J. 2006. Uncertainty about the rest of the sentence. Cognitive Science 30(4), 643-672. [https://doi.org/10.1207/s15516709cog0000_64]
  • Jaeger, F. 2009. Centering several variables. Retrieved from https://hlplab.wordpress.com/2009/04/27/centering-several-variables
  • Jelinek, F. and J. Lafferty. 1991. Computation of the probability of initial substring generation by stochastic context-free grammars. Computational Linguistics 17(3), 315-353.
  • Kennedy, A., R. Hill and J. Pynte. 2003. The Dundee Corpus. Paper presented at the 12th European Conference on Eye Movement, Dundee, Scotland.
  • Kennedy, A., J. Pynte, W. S. Murray and S.-A. Paul. 2013. Frequencyand predictability effects in the Dundee Corpus: An eye movement analysis. Quarterly Journal of Experimental Psychology 66, 601–618. [https://doi.org/10.1080/17470218.2012.676054]
  • Kim, E., M-K. Park and H-J. Seo. 2020. L2ers’ predictions of syntactic structure and reaction times during sentence processing. Linguistic Research 37, 189-218.
  • Kuperberg, G. R. and T. F. Jaeger. 2016. What do we mean by prediction in language comprehension? Language, Cognition and Neuroscience 31, 32-59. [https://doi.org/10.1080/23273798.2015.1102299]
  • Kutas, M. and S. A. Hillyard. 1984. Brain potentials during reading reflect word expectancy and semantic association. Nature 307, 161-163. [https://doi.org/10.1038/307161a0]
  • Lasnik H, Lidz J. 2017. The argument from the poverty of the stimulus. In I Roberts, ed. Oxford Handbook of Universal Grammar, 221-248. Oxford: Oxford University Press. [https://doi.org/10.1093/oxfordhb/9780199573776.013.10]
  • Levy, R. 2008. Expectation-based syntactic comprehension. Cognition 106(3), 1126-1177. [https://doi.org/10.1016/j.cognition.2007.05.006]
  • Linzen, T., & M. Baroni. 2020. Syntactic Structure from Deep Learning. arXiv preprint arXiv:2004.10827. [https://doi.org/10.1146/annurev-linguistics-032020-051035]
  • Linzen, T., E. Dupoux and Y. Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4, 521-535. [https://doi.org/10.1162/tacl_a_00115]
  • Linzen, T. and T. F. Jaeger. 2016. Uncertainty and expectation in sentence processing: Evidence from subcategorization distributions. Cognitive Science 40(6), 1382-1411. [https://doi.org/10.1111/cogs.12274]
  • Lowder, M. W., W-I. Choi, F. Ferreira and J. M. Henderson. 2018. Lexical predictability during natural reading: Effects of surprisal and entropy reduction. Cognitive science 42, 1166-1183. [https://doi.org/10.1111/cogs.12597]
  • Marcus, M. P., B. Santorini and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The PennTreebank. Computational Linguistics 19(2), 313–330. [https://doi.org/10.21236/ADA273556]
  • Marvin, R. and T. Linzen. 2018. Targeted syntactic evaluation of language models. arXiv preprint arXiv:1808.09031 [https://doi.org/10.18653/v1/D18-1151]
  • McCoy, R. T., R. Frank and T. Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. arXiv preprint arXiv:1802.09091
  • Monsalve, I. F., S. L. Frank and G. Vigliocco. 2012, April. Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics 398-408.
  • Radford, A., J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9.
  • Rayner, K. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124, 372-422. [https://doi.org/10.1037/0033-2909.124.3.372]
  • Smith, N. J. and R. Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition 128(3), 302-319. [https://doi.org/10.1016/j.cognition.2013.02.013]
  • Staub, A. 2015. The effect of lexical predictability on eye movements in reading: Critical review and theoretical interpretation. Language and Linguistics Compass 9, 311-327. [https://doi.org/10.1111/lnc3.12151]
  • Trueswell, J., M. Tanenhaus and C. Kello. 1993. Verb-specific constraints in sentence processing: Separating effects of lexical preference from garden-paths. Journal of Experimental Psychology: Learning, Memory, and Cognition 19(3), 528-553. [https://doi.org/10.1037/0278-7393.19.3.528]

서혜진(Seo, Hye-Jin), 대학원생(Graduate Student)동국대학교(Dongguk University)영어영문학부04620 서울특별시 중구 필동로 1길 30Tel: 02) 2260-8705E-mail: seohj0951@gmail.com

신정아(Shin, Jeong-Ah), 교수(Professor)동국대학교(Dongguk University)영어영문학부04620 서울특별시 중구 필동로 1길 30Tel: 02) 2260-3167E-mail: jashin@dongguk.edu