The Korean Association for the Study of English Language and Linguistics

Korean Journal of English Language and Linguistics - Vol. 22

[ Article ]
Korea Journal of English Language and Linguistics - Vol. 22, No. 0, pp. 336-354
Abbreviation: KASELL
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Received 28 Jan 2022 Revised 05 Mar 2022 Accepted 28 Mar 2022
DOI: https://doi.org/10.15738/kjell.22..202203.336

2022학년도 수능영어의 체감 난이도 상승 요인에 관한 코퍼스 기반 분석
한지수 ; 이영희
중앙대학교
아주대학교

A corpus-based analysis of the perceived difficulty increase in the 2022 CSAT English Test
Han, Ji-Su ; Younghee Cheri Lee
(co-first author) Graduate Student, Dept. of English Education, Chung-Ang Univ, Tel: 02-820-5391 (jshan111@cau.ac.kr)
(co-first author / corresponding author) Teaching Professor, Dasan University College, Ajou Univ, Tel: 031-219-3053 (cheriberry@ajou.ac.kr)

© 2022 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium provided the original work is properly cited.

Abstract

Since adopting an absolute scale in the College Scholastic Ability Test (CSAT) as of 2017, the CSAT English exam has been in the spotlight for its sheer foreseeable consequences that are not perfectly tailored enough to rank test-takers, drastically lower students’ English savvy, and in turn, inevitably undermine the status of the English language as a core subject. Despite the global pandemic’s impact on educational attainment, worse, the 2022 CSAT English test-takers have lately witnessed a sharp rise in the test’s perceived difficulty compared to the previous year, thereby earning the test the moniker of a ‘hellacious exam.’ Thus, this study aims to delve into the factors that may have caused an increase in its perceived difficulty in the three dimensions: vocabulary load, syntactic pattern density, and information load pertinent to text complexity. To that end, comparable corpora were created using three different CSAT-related texts–the 2022 CSAT English texts, official CSAT test-prep workbooks by EBS, and top-ranked CSAT test-prep materials by commercial publishers–for a corpus-based comparative analysis confined only to gap-filling inference question items with high test difficulties. The results attested no statistically significant difference in any of the three categories, meaning that the text difficulty of the 2022 CSAT English is remarkably similar to just what test-takers have been exposed to in conventional academic settings. Despite the relatively acceptable level of vocabulary variety, syntactic density, and even reduced text complexity, the vast gulf between the ‘genuine’ difficulty and the ‘perceived’ difficulty might have been attributed to insufficient basic core academic skills in English and low text familiarity driven by the EBS materials indirectly linked to the CSAT. Implications will be discussed hinged upon the findings.


Keywords: CSAT, perceived difficulty, vocabulary load, syntactic pattern, information load, text difficulty, lexical complexity, syntactic complexity, token coverage, corpus

References
1. Anthony, L. 2021. AntConc, Version 3.5.8 [computer software]. Tokyo, Japan: Waseda University. Accessed 10 Aug 2021 at https://www.laurenceanthony.net/software
2. CBSi. 2021. Every time there is a controversy about the difficulty of the entrance exam... Does it raise re-test takers? Retrieved 20 December 2021 from https://www.nocutnews.co.kr/news/5674669
3. Chang, J.-Y. 2018. A comparison of 2017-2018 CSAT reading passages via Coh-Metrix: Focusing on descriptive, readability, and easibility measures. Foreign Languages Education 25(4), 81-106.
4. Chang, J.-Y. 2019. A comparison of syntactic complexity in CSAT reading passages before and after the introduction of criterion-referenced evaluation. Journal of the Korea English Education Society 18(2), 161-188.
5. Choi, M. and J.-R. Kim. 2017. An analysis of cohesion and word information among English CSAT question types. The Journal of the Korea Contents Association 17(12), 378-385.
6. Cho, M.-S. and E.-S. Park. 2016. A corpus-based analysis of vocabulary in high school English textbooks, EBS CSAT books, and CSATA. Journal of Learner-Centered Curriculum and Instruction 16(7), 215-233.
7. Coh-Metrix. 2017. Documentation. Retrieved 20 December 2021 from https://www.cohmetrix.com
8. Coxhead, A. 2000. A new academic word list. TESOL Quarterly 34(2), 213-238.
9. Davies, M. 2010. The corpus of contemporary American English as the first reliable monitor corpus of English. Literary and Linguistic Computing 25(4), 447-464.
10. EBS Editorial Department. 2021. EBS CSAT test-prep workbook. EBS.
11. EBSi. 2021. Recent grade cut/error rate. Retrieved 3 December 2021 from https://www.ebsi.co.kr/ebs/xip/xipa/retrievePastGrdCutWrongAnswerRate.ebs?tab=1
12. Gardner, D. 2013. Exploring Vocabulary: Language in Action. New York: Routledge.
13. Goh, G.-Y. and J.-H. Back. 2010. A corpus-based analysis of the vocabulary used in the CSAT English exam and two analogous tests. English Language and Linguistics 16(2), 1-26.
14. Graesser, A. C., D. S. McNamara, M. M. Louwerse and Z. Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers 36(2), 193-202.
15. Hirsh, D. and I. S. P. Nation. 1992. What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a Foreign Language 8(2).
16. Hutchinson, T. and A. Waters. 1987. English for Specific Purposes. Cambridge: Cambridge University Press.
17. Hwang, J.-B. 2014. A recent change in the system of CSAT and an application of absolute evaluation to the English section of CSAT. Conference for improvement on CSAT for the normalization of school English education.
18. Hwang, J.-B. 2018. The 70 % connection to EBS in the English section of CSAT: What’s the problem?
19. IBM Corp. 2021. IBM SPSS Statistics for Windows, Version 28.0 [computer software]. Armonk, NY: IBM Corp.
20. Jung, C. G. 2014. The Archetypes and the Collective Unconscious. Routledge.
21. Kaplan, R. and R. Jr. Baldauf. 2005. Language-in-education policy and planning. In E. Hinkel, ed., Handbook of Research in Second Language Teaching and Learning, 1013-1034. London: Lawrence Erlbaum.
22. KICE. 2018. 2018 Haknyeondo Suneung Yeongeo Jeoldaepyeongga Hakseupannae.
23. Kim, C. H. 2020. A corpus-based comparative analysis of CSAT English from 2015 to 2020 on the basis of criterion-referenced assessment. Multimedia-Assisted Language Learning 23(2), 98-120.
24. Kim, D.-H. and J.-H. Lee. 2014. University students’ reflective perception on their experience of private English education during high school year. Modern English Education 15(3), 241-262.
25. Kim, J. E. and I. C. Choi. 2015. A corpus-based comparative analysis of linguistic difficulty among high school English textbooks, EBS-CSAT prep books, and College Scholastic Ability Test. Multimedia-Assisted language Learning 18(1), 59-92.
26. Kim, J.-R. 2017. A diachronic analysis of English KSAT reading passages. Journal of the Elementary Education Society 27, 63-78.
27. Kim, J.-T. 2019. In the English section of CSAT, absolute evaluation or absolute scale?: A discussion on the adequacy of the evaluation of the English section of CSAT and a suggestion about its evaluative system. The crisis in English education in Korea: from a kindergarten to beyond college, 24-36.
28. Koh, N. and J.-A. Shin. 2017. A comparison of the level of difficulty in the English reading part of the CSAT: Before and after the EBS-CSAT linkage policy. Secondary English Education 10(4), 3-24.
29. Kwon, S.-K. and D. Shin. 2014. The effects of the EBS books-CSAT linkage policy on vocabulary difficulty of the English section in the CSAT. Journal of the Korea English Education Society, 13(4), 97-121.
30. Laufer, B. 1989. What percentage of text-lexis is essential for comprehension? In C. Lauren and M. Nordman, eds., Special Language: From Human Thinking to Thinking Machines, 316-323. Clevedon: Multilingual Matters.
31. Laufer, B. 1992. How much lexis is necessary for reading comprehension? In P. J. L. Arnaud and H. Bejoing, eds., Vocabulary and Applied Linguistics, 126-132. London: Palgrave Macmillan.
32. Lee, B.-M. 2014. An evaluation method of the English section of CSAT: Is this effective? Paper presented at the 64th KEDI Education Policy Forum. Korean Educational Development Institute, April 24.
33. Lee, D.-H. and M. G. Kang. 2015. The analysis of linkage of the Korea educational broadcasting system (EBS) with the college scholastic ability test (CSAT) & trial test for CSAT and the study on improvements for the linkage (in the English Part of the Test). Secondary English Education 8(4), 69-94.
34. Lee, H. 2020. Vocabulary analysis of CSAT English tests and CSAT-EBS preparation coursebooks, with reference to the reading tests. Modern English Education, 21(3), 48-57.
35. Lee, J. W. and J. H. Kwon. 2015. A study on reading test-taking strategies in item types of an English reading test The Journal of Humanities Studies 54(4), 79-100.
36. Lee, K.-N. 2021. A pedagogical discussion based on the investigation into students’ selection tendencies in three basic subjects (Korean, English, and Mathematics) of the high school curricula: Centered on general high schools in Seoul. Korean Journal of English Language and Linguistics 21, 1-18.
37. Lee, Y. C. and T.-Y. Kim. 2022. The paradigm shift in English language teaching in North Korea: A corpus-assisted analysis. Korean Journal of English Language and Linguistics 22, 279-299.
38. Lee, Y.-S. 2018. Absolute evaluation of the English section of CSAT. In papers from the crisis in English education in Korea: School English education and national competitiveness, 3-6.
39. Lee, Y.-W., H. Jung, Y. Yang, J.-W. Lee and H. Pae. 2020. Investigating changing trends in college English education before and after the introduction of the Jeoldaepyeongga system to the KCSAT English section. The Journal of English Language and Literature 66(4), 807-846.
40. Lewis, M. 1993. The Lexical Approach. Hove: Language Teaching Publications.
41. Ministry of Education. 2011. The 2011 CSAT Implementation Plan. Retrieved 21 December 2021 from https://www.moe.go.kr/sn3hcv/doc.html?fn=260b5127ef547f0759a11228cc25676e&rs=/upload/synap/202201
42. Ministry of Education. 2021. The 2021 CSAT Implementation Plan. Retrieved 21 December 2021 from https://www.moe.go.kr/sn3hcv/doc.html?fn=f53ac191cefe714d75953a279d076949&rs=/upload/synap/202201
43. Moon, J.-H. and H.-D. Kim. 2017. An analysis of the linguistic elements of the text in the English reading section of the College Scholastic Ability Test. Modern English Education 18(1), 193-211.
44. Nation, I. S. P. 1990. Teaching and learning vocabulary. New York: Newbury House.
45. Nation, I. S. P. 2005. Teaching and learning vocabulary. In E. Hinkel, eds., Handbook of Research in Second Language Teaching and Learning, 605-620. Routledge.
46. Nation, I. S. P. 2006. How large a vocabulary is needed for reading and listening? Canadian Modern Language Review 63(1), 59-82.
47. Nation, I. S. P. 2013. Learning Vocabulary in Another Language. Cambridge: Cambridge University Press.
48. Nation, I. S. P. 2016. Making and Using Word Lists for Language Learning and Testing. Amsterdam: John Benjamins.
49. Nation, I. S. P. and Beglar. 2007. A vocabulary size test. The Language Teacher 31(7), 9–13.
50. Oh, J.-I. 2014. A corpus-based analysis of the vocabulary levels of English II and EBS college entrance examination preparation coursebooks. Journal of the Korea English Education Society 13(2), 115-132.
51. Schmitt, N. 1997. Vocabulary learning strategies. In D. N. Schmitt and M. McCarthy, eds., Vocabulary: Description, Acquisition and Pedagogy, 199-227. Cambridge: Cambridge University Press.
52. Schmitt, N., T. Cobb, M. Horst and D. Schmitt. 2017. How much vocabulary is needed to use English? Replication of van Zeeland & Schmitt (2012), Nation (2006) and Cobb (2007). Language Teaching 50(2), 212-226.
53. Schmitt, N., X. Jiang and W. Grabe. 2011. The percentage of words known in a text and reading comprehension. The Modern Language Journal 95(1), 26-43.
54. Scott, M. 2016. Wordsmith Tools 7.0. Liverpool: Lexical Analysis Software.
55. Shin, Y. 2019. Analyzing CSAT reading passages by using Coh-Metrix and VocaProfile: Focusing on four years from 2016 to 2019. Journal of Language Sciences 26(4), 109-127.
56. Shulz, R. A. 1981. Literature and readability: Bridging the gap in foreign language reading. The Modern Language Journal 65(1), 43-53.
57. West, M. Ed. 1953. A General Service List of English Words: With Semantic Frequencies and a Supplementary Wordlist for the Writing of Popular Science and Technology. Longman.