The Korean Association for the Study of English Language and Linguistics

Korean Journal of English Language and Linguistics - Vol. 21
[Paper List] [Goto Volume List]

[ Article ]
Korea Journal of English Language and Linguistics - Vol. 21, No. 0, pp. 487-509
Abbreviation: KASELL
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Received 25 Apr 2021 Revised 05 Jun 2021 Accepted 25 Jun 2021
DOI: https://doi.org/10.15738/kjell.21..202106.487

A Deep Learning-based Understanding of Nativelikeness: A Linguistic Perspective
Kwonsik Park ; Sanghoun Song
(1st author) Graduate Student, Dept. of Linguistics, Korea Univ., Tel: +82-2-3290-1648 (oneiric66@korea.ac.kr)
(corresponding author) Professor, Dept. of Linguistics, Korea Univ., Tel: +82-2-3290-2177 (sanghoun@korea.ac.kr)


© 2021 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Funding Information ▼
Abstract

Constructing deep learning models that identify nativelikeness in English sentences, this paper addresses two relevant research questions: is nativelikeness measurable, and is it determined by syntactic well-formedness and lexical associations? To address the first, our models are evaluated by judging every item in Test Suite I, which comprises learner and native sentences from four sources. The results show that the models predict nativelikeness reasonably well. Next, syntactic well-formedness is examined via Test Suite II, comprising correct–incorrect minimal pairs with two conditions. The results indicate that our models do not satisfactorily detect it. The learners’ results reveal their limited knowledge, suggesting that the models learn the inadequateness of lexical associations as a feature of non-nativelikeness because the learner training data comprises Korean English learner corpora. However, our models’ results also show poor performance. We conclude that deep learning is capable of measuring nativelikeness, and well-formedness and lexical associations are no more than necessary conditions for nativelikeness. This implies the need to consider other factors when defining and assessing nativelikeness.

Keywords: deep learning, nativelikeness, well-formedness, lexical association, learner corpora
Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2020S1A5A2A03042760).

Some earlier versions of the present study were partially presented at the 34th Pacific Asia Conference on Language, Information and Computation (online, Oct-24-2020) and the 2020 International Conference on English Linguistics (online, Oct-17-2020).

References
1. Abrahamsson, N. and K. Hyltenstam. 2009. Age of onset and nativelikeness in a second language: Listener perception versus linguistic scrutiny. Language Learning 59(2), 249–306.
2. Alain, G. and Y. Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.
3. Altenberg, B. 1991. Amplifier collocations in spoken English. In S. Johansson and A. Stenström, eds., English Computer Corpora: Selected Papers and Reearch Guide, 127–147. Berlin: Mouton de Gruyter.
4. Bengio, Y., I. Goodfellow and A. Courville. 2017. Deep Learning (Vol. 1). Massachusetts, USA: MIT press.
5. Birdsong, D. 2005. Nativelikeness and non-nativelikeness in L2A research. International Review of pplied Linguistics in Language Teaching 43(4), 319-328.
6. Bylund, E., N. Abrahamsson and K. Hyltenstam. 2012. Does first language maintenance hamper nativelikeness in a second language? A study of ultimate attainment in early bilinguals. Studies in Second Language Acquisition 34(2), 215-241.
7. Carlstrom, B. and N. Price. 2013. Gachon Learner Corpus. http://thegachonlearnercorpus.blogspot.kr
8. Clahsen, H. and C. Felser. 2006a. Continuity and shallow structures in language processing: A reply to our commentators. Applied Psycholinguistics 27(1), 107–126.
9. Clahsen, H. and C. Felser. 2006b. How native-like is non-native language processing? Trends in Cognitive Sciences 10(12), 564–570.
10. Davies, M. 2008. The Corpus of Contemporary American English (COCA): 560 million words, 1990–present. www.english-corpora.org/coca
11. Davies, M. 2013. Google Scholar and COCA-Academic: Two very different approaches to examining academic English. Journal of English for Academic Purposes 12(3), 155–165.
12. DeKeyser, R. M. 2000. The robustness of critical period effects in second language acquisition. Studies in Second Language Acquisition 22(4), 499–533.
13. Devlin, J., M W. Chang, K. Lee and K. Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
14. Ebeling, S. O. and H. Hasselgård. 2015. Learner corpora and phraseology. In S. Granger, G. Gilquin and F. Meunier, eds., The Cambridge Handbook of Learner Corpus Research, 207–230. Cambridge: Cambridge University Press.
15. Firth, J. R. 1961 [1957]. Papers in Linguistics: 1934–1951. London: Oxford University Press.
16. Gao, L. 2005. Latin Squares in Experimental Design. East Lansing: Michigan State University.
17. Goldberg, A. E. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford University Press on Demand.
18. Goldberg, Y. 2019. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287
19. Goodfellow, I., Y. Bengio, A. Courville and Y. Bengio. 2016. Deep Learning (Vol. 1, No. 2). Cambridge: MIT press.
20. Graff, D., J. Kong, K. Chen and K. Maeda. 2003. English Gigaword. Philadelphia: Linguistic Data Consortium.
21. Guerssel, M., K. Hale, M. Laughren, B. Levin and J. W. Eagle. 1985. A cross-linguistic study of transitivity alternations. Cls 21(2), 48-63.
22. Gulordava, K., P. Bojanowski, E. Grave, T. Linzen and M. Baroni. 2018. Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138
23. Herculano-Houzel, S. 2002. Do you know your brain? A survey on public neuroscience literacy at the closing of the decade of the brain. The Neuroscientist 8(2), 98–110.
24. Hochreiter, S. and J. Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8), 1735–1780
25. Ivanova, I., M. J. Pickering, J. F. McLean, A. Costa and H. P. Branigan. 2012. How do people produce ungrammatical utterances? Journal of Memory and Language 67(3), 355-370.
26. Johnson, J. S. and E. L. Newport. 1989. Critical period effects in second language learning: The influence of maturational state on the acquisition of English as a second language. Cognitive Psychology 21(1), 60–99.
27. Ko, H., T. Ionin and K. Wexler 2010. The role of presuppositionality in the second language acquisition of English articles. Linguistic Inquiry 41(2), 213–254.
28. Krohn, J., G. Beyleveld and A. Bassens. 2019. Deep Learning Illustrated: A Visual, Interactive Guide to Artificial Intelligence. Addison-Wesley Professional.
29. Lambrecht, K. 1986. Topic, Focus, and the Grammar of Spoken French. Doctoral dissertation, Berkeley: University of California.
30. Leacock, C., M. Chodorow, M. Gamon and J. Tetreault. 2010. Automated Grammatical Error Detection for Language Learners: Second Edition. Morgan & Claypool Publishers.
31. Linzen, T. 2019. What can linguistics and deep learning contribute to each other? Response to Pater. Language 95(1), e99–e108.
32. Linzen, T. and B. Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. arXiv preprint arXiv:1807.06882
33. Long, M. H. 1990. Maturational constraints on language development. Studies in Second Language Acquisition 12(3), 251-285.
34. Magnusson, J. E. and C. Stroud. 2012. High proficiency in markets of performance: A sociocultural approach to nativelikeness. Studies in Second Language Acquisition 34(2), 321–345.
35. Marvin, R. and T. Linzen. 2018. Targeted syntactic evaluation of language models. arXiv preprint arXiv:1808.09031
36. McCoy, R. T., E. Pavlick and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007
37. Myers, J. 2009. The design and analysis of small-scale syntactic judgment experiments. Lingua 119(3), 425-444.
38. Nesselhauf, N. 2003. The use of collocations by advanced learners of English and some implications for teaching. Applied Linguistics 24(2), 223–242.
39. Owens, M. T. and K. D. Tanner. 2017. Teaching as brain changing: Exploring connections between neuroscience and innovative teaching. CBE – Life Sciences Education 16(2), fe2.
40. Paszke, A., S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga and A. Lerer. 2017. Automatic differentiation in PyTorch. In NIPS workshop, 2017. California
41. Partington, A. 2004. “Utterly content in each other’s company”: Semantic prosody and semantic preference. International Journal of Corpus Linguistics 9(1), 131–156.
42. Pawley, A. and F. H. Syder. 1983. Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. In J. C. Richards and R. W. Schmidt, Language and Communication, 191–225. London: Longman.
43. Rhee, S. and C. Jung. 2012. Yonsei English Learner Corpus (YELC). In Proceedings of the First Yonsei English Corpus Symposium, 26–36. Seoul.
44. Rozovskaya, A. and D. Roth. 2010. Training paradigms for correcting errors in grammar and usage. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 154–162. Stroudsburg.
45. Rumelhart, D. E., G. E. Hinton and R. J. Williams. 1986. Learning representations by back-propagating errors. Nature 323(6088), 533–536.
46. Schmitt, N. 1998. Quantifying word association responses: What is native-like? System 26(3), 389–401.
47. Shei, C. C. and H. Pain. 2000. An ESL writer’s collocational aid. Computer Assisted Language Learning 13(2), 167–182.
48. Song, S. and E. Oh. 2017. In Defense of Comparisons between Formal and Informal Acceptability Judgments. Studies in Generative Grammar 27(4), 893-902.
49. Sorace, A. 2003. Near-nativelikeness. In C. J. Doughty and M. H. Long, eds., The Handbook of Second Language Acquisition, 130–151. New York: Blackwell.
50. Sorace, A. and F. Keller. 2005. Gradience in linguistic data. Lingua115(11), 1497-1524.
51. Sprouse, J., C. T. Schütze and D. Almeida. 2013. A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010. Lingua 134, 219-248.
52. Stubbs, M. 1995. Collocations and semantic profiles: On the cause of the trouble with quantitative studies. Functions of Language 2(1), 23–55.
53. Tanaka, Y. 2001. Compilation of a multilingual corpus. In Proceedings of the Pacific Association for Computational Linguistics (PACLING), 265–268. Kyushu.
54. Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
55. Warstadt, A., A. Singh and S. R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics 7, 625–641.
56. Warstadt, A., A. Parrish, H. Liu, A. Mohananey, W. Peng, S. Wang and S. R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics 8, 377–392.
57. Wilcox, E., R. Levy and R. Futrell. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. arXiv preprint arXiv:1906.04068
58. Wray, A. 2000. Formulaic sequences in second language teaching: Principle and practice. Applied Linguistics 21(4), 463–489.
59. Yang, Z., Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov and Q. V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237
60. Yoon, S., S. Park, J. Kim, H. Yoo and C. Jung. 2020. Incheon National University Multi-language Learner Corpus (INU-MULC): Its design and application. Abstract accepted at Asia Pacific Corpus Linguistics Conference 2020. Seoul, South Korea.