Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 25, No. 0, pp.1048-1065
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Print publication date 31 Jan 2025
Received 10 Jun 2025 Revised 23 Jul 2025 Accepted 05 Aug 2025
DOI: https://doi.org/10.15738/kjell.25..202508.1048

한국인 학습자 영어 말하기 자동 평가를 위한 문법 다양성 척도 연구

Min-Chang Sung ; Jin-Hwa Lee ; Heyoung Kim ; YunDeok Choi
(First author) Associate Professor, Department of English Education, Gyeongin National University of Education mcsung@ginue.ac.kr
(Corresponding author) Professor, Department of English Education, Chung-Ang University jinhlee@cau.ac.kr
(Co-author) Professor, Department of English Education, Chung-Ang University englishnet@cau.ac.kr
(Co-author) Assistant Professor, Department of English Education, Chungnam National University yundeokchoi@cnu.ac.kr
A study on grammatical diversity measures for automated English speaking assessment of Korean learners


© 2025 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This study investigates various measures of grammatical diversity for automated speaking assessment (ASA) with Korean learners of English. Data were extracted from the Korean monologue set in the International Corpus of Asian Learners of English and classified into two proficiency groups. Using two NLP toolkits, i.e., the Biber tagger and the argument structure construction annotator, we measured the grammatical diversity of the speaking data based on six features over three levels (i.e., word, phrase, and clause), and conducted correlation and binomial logistic regression analyses. The results reveal significant correlations among the features, particularly between those related to part-of-speech and the others. It is also found that only the clause-level feature of the subordination type frequency significantly predicts proficiency levels. These findings provide insights into the potential of grammatical diversity as a valuable metric in ASA systems for Korean learners of English.

Keywords:

automated speaking assessment, grammatical diversity, English proficiency, L2 corpus, NLP-based analyses

Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2023S1A5A2A03086569).

References

  • 교육부(Ministry of Education). 2022. 『2022 개정 영어과 교육과정』(2022 revised national curriculum for English). 교육부(Ministry of Education).
  • 남보라∙조규희∙박지현∙성민창∙황필아∙강정진∙이동환∙심창용∙최희경∙박선호∙김혜련(Nam, B., K. Jo, J. Park, M. Sung, P. Hwang, J. Kang, D. Lee, C. Sim, H. Choi, S. Park and H. Kim). 2024. 『인공지능 융합 영어교육의 이해』(Understanding of AI convergence English education). 경문사(Kyungmoonsa).
  • 문용(Moon, Y.). 2017. 『고급 영문법 해설』(Advanced English grammar explanation). 박영사(Parkyoungsa).
  • 신동광∙박용효∙박태준∙임수연(Shin, D, Y. Park, T. Park and S. Yim). 2015. 영어 말하기 자동채점의 현재와 미래(The present and future of an automated scoring program for speaking skills of English). ≪멀티미디어 언어교육≫(Multimedia-Assisted Language Learning) 18(1), 93-114.
  • 이진화∙최윤덕∙성민창∙김혜영(Lee, J.-H., Y. Choi, M. Sung and H. Kim). 2023. 자동채점 기반 영어 말하기 시험 현황 분석(Analysis of English automated speaking scoring tests). ≪영어교육≫(English Teaching) 78(2), 223-244.
  • ACTFL. 2012. ACTFL proficiency guidelines 2012. ACTFL.
  • ACTFL. 2024. ACTFL proficiency guidelines 2024. ACTFL.
  • Bhat, S. and S. Y. Yoon. 2015. Automatic assessment of syntactic complexity for spontaneous speech scoring. Speech Communication 67, 42-57. [https://doi.org/10.1016/j.specom.2014.09.005]
  • Bhat, S., H. Xue and S. Y. Yoon. 2014. Shallow analysis based assessment of syntactic complexity for automated speech scoring. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1305-1315. [https://doi.org/10.3115/v1/P14-1123]
  • Biber, D., B. Gray and K. Poonpon. 2011. Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly 45, 5-35. [https://doi.org/10.5054/tq.2011.244483]
  • Biber, D., S. Johansson, G. Leech, S. Conrad and E. Finegan. 1999. Longman grammar of spoken and written English. Longman.
  • Chen, L., K. Zechner, S. Yoon, K. Evanini, X. Wang, A., Loukina, … and B. Gyawali. 2018. Automated scoring of nonnative speech using the SpeechRater v. 5.0 engine. (ETS Research Report No. RR-18-10). Educational Testing Service. Available online at https://onlinelibrary.wiley.com/doi/full/10.1002/ets2.12198 [https://doi.org/10.1002/ets2.12198]
  • Choi, J. and M. Sung. 2020. Utterance-based measurement of L2 fluency in speaking interactions: A constructionist approach. English Teaching 75(S1), 105-126. [https://doi.org/10.15858/engtea.75.s1.202006.105]
  • Council of Europe. 2020. Common European framework of reference for languages: Learning, teaching, assessment – Companion volume. Council of Europe Publishing.
  • Cowan, R. 2008. The teacher’s grammar of English with answers: A course book and reference guide. Cambridge University Press.
  • De Marneffe, M. C., C. Manning, J. Nivre and D. Zeman. 2021. Universal dependencies. Computational Linguistics 47(2), 255-308. [https://doi.org/10.1162/coli_a_00402]
  • Dogar, M. F., T. Saleem, M. Aslam, and S. Khan. 2024. Exploring global linguistic nuances: Analyzing region-specific inflectional morpheme frequency in ICNALE. Asian-Pacific Journal of Second and Foreign Language Education 9(1), 65. [https://doi.org/10.1186/s40862-024-00291-z]
  • Dulay, H. C. and M. K. Burt. 1974. Natural sequences in child second language acquisition. Language Learning 24(1), 37-53. [https://doi.org/10.1111/j.1467-1770.1974.tb00234.x]
  • Ellis, N. C. and F. Ferreira-Junior. 2009. Construction learning as a function of frequency, frequency distribution, and function. The Modern Language Journal 93(3), 370-385. [https://doi.org/10.1111/j.1540-4781.2009.00896.x]
  • Goldberg, A. E. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press.
  • Ishikawa, S. 2019. The ICNALE spoken dialogue: A new dataset for the study of Asian learners’ performance in L2 English interviews. English Teaching 74(4), 153-177. [https://doi.org/10.15858/engtea.74.4.201912.153]
  • Khristoforov, S., V. Bochkarev and A. Shevlyakova. 2020. Recognition of parts of speech using the vector of bigram frequencies. In W. van der Aalst et al., eds., Analysis of Images, Social networks and Texts: AIST 2019, 137-147. Springer. [https://doi.org/10.1007/978-3-030-39575-9_13]
  • Kim, M. and X. Lu. 2024. L2 English speaking syntactic complexity: Data preprocessing issues, reliability of automated analysis, and the effects of proficiency, L1 background, and topic. The Modern Language Journal 108(1), 270-296. [https://doi.org/10.1111/modl.12907]
  • Krashen, S. and T. Terrell. 1983. The natural approach: Language acquisition in the classroom. Pergamon Press.
  • Kyle, K. 2016. Measuring Syntactic Development in L2 writing: Fine-grained Indices of Syntactic Complexity and Usage-based Indices of Syntactic Sophistication. Doctoral dissertation, Georgia State University.
  • Kyle, K. and S. Crossley. 2017. Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing 34(4), 513-535. [https://doi.org/10.1177/0265532217712554]
  • Kyle, K., H. Sung, H., D. Biber, R. Reppen, R. and J. Egbert. 2025. The development and evaluation of an open-source lexicogrammatical complexity analysis tool: The Lexicogrammatical Tagger. Paper presented at the Annual Conference of American Association of Applied Linguistics.
  • Lee, J.-H. and H. M. Kim. 2011. The L2 developmental sequence of English constructions and underlying factors. Korean Journal of English Language and Linguistics 11(3), 577-600. [https://doi.org/10.15738/kjell.11.3.201109.577]
  • McEnery, T. and A. Hardie. 2011. Corpus linguistics: Method, theory and practice. Cambridge University Press. [https://doi.org/10.1017/CBO9780511981395]
  • Nation, I. S. P. and D. Beglar. 2007. A vocabulary size test. The Language Teacher 31(7), 9-13.
  • Ortega, L. 2003. Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics 24(4), 492-518. [https://doi.org/10.1093/applin/24.4.492]
  • Park, J. and M. Sung. 2024. Expansion of verb-argument construction repertoires in L2 English writing. International Review of Applied Linguistics in Language Teaching 62(2), 903-925. [https://doi.org/10.1515/iral-2022-0145]
  • Putra, J. W. G., S. Teufel and T. Tokunaga. 2021. Multi-task and multi-corpora training strategies to enhance argumentative sentence linking performance. arXiv preprint arXiv:2109.13067, . [https://doi.org/10.18653/v1/2021.argmining-1.2]
  • Skehan, P. 1996. A framework for the implementation of task-based instruction. Applied linguistics 17, 38-62. [https://doi.org/10.1093/applin/17.1.38]
  • Sung, H. and K. Kyle. 2024. Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7302-7314. [https://doi.org/10.18653/v1/2024.emnlp-main.415]
  • Sung, M. 2022. Lexical verb forms in L1 and L2 spoken English: A corpus-based analysis. The Journal of Education 42(S), 53-66. [https://doi.org/10.25020/je.2022.42.s.53]
  • Sung, M. and H. Kim. 2022. Effects of verb–construction association on second language constructional generalizations in production and comprehension. Second Language Research 38(2), 233-257. [https://doi.org/10.1177/0267658320932625]
  • Vercellotti, M. L. 2019. Finding variation: Assessing the development of syntactic complexity in ESL Speech. International Journal of Applied Linguistics 29, 233-247. [https://doi.org/10.1111/ijal.12225]
  • Xu, J., E. Jones, V. Laxton and E. Galaczi. 2021. Assessing L2 English speaking using automated scoring technology: Examining automarker reliability. Assessment in Education: Principles, Policy & Practice 28(4), 411-436. [https://doi.org/10.1080/0969594X.2021.1979467]
  • Yoon, S. Y. and S. Bhat. 2012. Assessment of ESL learners’ syntactic competence based on similarity measures. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 600-608.
  • Zechner, K. and K. Evanini. 2019. Automated speaking assessment: Using language technologies to score spontaneous speech. Routledge. [https://doi.org/10.4324/9781315165103]