Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 19, No. 4, pp.796-816
ISSN: 1598-1398 (Print)
Print publication date 31 Dec 2019
Received 01 Nov 2019 Revised 15 Dec 2019 Accepted 20 Dec 2019
DOI: https://doi.org/10.15738/kjell.19.4.201912.796

영작문평가 상황에서 FACETS와 Coh-Metrix를 활용한 에세이 분석: 학습자 단면을 중심으로

이호
중앙대학교
Analyzing L2 essay with FACETS and Coh-Metrix in L2 writing assessment
Lee, Ho

Abstract

The current study aims to identify Coh-Metrix indices that significantly predict writing proficiency and well discriminate misfit learners from infit and overfit learners. 85 undergraduate and graduate students took an on-line essay test. Two expert raters scored their essays in terms of three analytic areas such as content, organization, and language use. The researcher conducted FACETS analysis whose purpose is to form sub-groups in regard to proficiency level and fit information. Then, the researcher ran Coh-Metrix program to investigate semantic, syntactic, and linguistic features of the students’ essays. The result showed that sentence length, argument overlap, and syntax similarity in adjacent sentences meaningfully discriminated proficient group from less proficient one. In addition, essays written by misfit learners showed lower value of Flesch Reading Ease than essays by counterparts. This study ended with the implication that further studies in L2 writing assessment need to explore learner dimension.

Keywords:

many-facet Rasch model, Coh-Metrix, FACETS, Flesch Reading ease, writing assessment

References

  • 김준구(Kim, J.). 2018. Coh-Metrix를 활용한 중학교 학업성취도 평가 영어 지문의 코퍼스 언어학적 분석(A corpus-based investigation of reading passages in the national assessment of educational achievement English test using Coh-Metrix). ≪중등영어교육≫(Secondary English Education) 11-2, 27-51.
  • 성일호(Sung, I.). 2014. 이독성 공식과 Coh-Metrix를 활용한 우리나라 고등학교 영어교과서 이독성 분석(An analysis of Korean high school English textbooks through readability formulae and Coh-Metrix). ⟪영어영문학연구⟫(Studies in English Language & Literature) 40-4, 299-320. [https://doi.org/10.21559/aellk.2014.40.4.015]
  • 신유선⋅정영경(Shin, Y. S. and Y. Jong). 2018. 코메트릭스(Coh-Metrix)를 활용한 공무원 7급과 9급 영어 시험 지문 분석: 어휘 다양성, 통사적 복잡성, 응집성과 이독성 지수를 중심으로(Analyzing reading passages in the English tests for civil servants with Coh-Metrix: Focusing on lexical diversity, syntactic complexity, cohesion, and readability). ≪영미어문학≫(Studies in British and American Language and Literature) 129, 143-161. [https://doi.org/10.21297/ballak.2018.129.143]
  • 안수진(Ahn, S.). 2018. 한국인 대학생과 영어 원어민 대학생의 영어 설명문, 논설문의 코메트릭스 분석을 통한 비교(An analysis of Coh-Metrix on the differences in English expository and argumentative writing of Korean and native English university student). ⟪새한영어영문학≫(New Korean Journal of English Language and Literature) 60-3, 177-205. [https://doi.org/10.25151/nkje.2018.60.3.009]
  • 이승환(Lee, S-H.). 2013. 코메트릭스(Coh-Metrix)를 이용한 고등학교 영어1과 영어2 교과서 문어 텍스트의 코퍼스 언어학적 난이도 분석(A corpus-based analysis of the difficulty of the reading materials in high school English 1 and 2 textbooks using Coh-Metrix). ≪외국어교육연구≫(Studies in Foreign Language Education) 27-2, 131-148. [https://doi.org/10.16933/sfle.2013.27.2.131]
  • 이창수(Lee, C. S.). 2014. 『한국 중고등학교 교사들의 영어작문 채점 훈련에서 나타나는 채점자 편향과 그 원인』(Rater Bias and Its Source in Secondary School Writing Assessment). 미출간 박사학위 논문, 서울: 서울대학교.
  • 이호⋅김하나(Lee, H. and Ha-Na Kim). 2009. 영작문에서의 대면평가와 익명평가의 비교 연구(Exploring learners’ blind and face-to-face assessment in an L2 argumentative essay-writing test). ≪영어학≫(Korean Journal of English Language and Linguistics) 9-3. 403-427. [https://doi.org/10.15738/kjell.9.3.200909.403]
  • 장소영⋅신동일(Jang, S-Y. and D. I. Shin). 2009. 『언어교육평가 연구를 위한 FACETS 프로그램』(Guidelines of FACETS Program for Language Testing Researchers). 서울: 글로벌콘텐츠(Seoul: Global Contents).
  • 장지연(Chang, J-Y.) 2018. Coh-Metrix를 통한 2017-2018 수능 영어 독해 지문 비교: 기초 산출치, 이독성, 용이성을 중심으로(A comparison of 2017-2018 CSAT reading passage via Coh-Metrix: Focusing on descriptive, readability, and easibility measures). ≪외국어교육≫(Foreign Languages Education) 25-4, 81-106. [https://doi.org/10.15334/FLE.2018.25.4.81]
  • 전문기⋅임인재(Jeon, M-G and I-J. Lim). 2009. Coh-Metrix를 이용한 중학교 1학년 개정 영어 교과서의 코퍼스 언어학적 비교 분석(A corpus-based analysis of middle school English 1 textbooks with Coh-Metrix). ≪영어교육연구≫(English Language Teaching) 21-4, 265-292. [https://doi.org/10.17936/pkelt.2009.21.4.012]
  • 전문기⋅최윤희(Jeon, M-G and Y. Choe). 2019. 코메트릭스를 활용한 한국 대학생들의 영어 논설문과 설명문 요약문 비교 분석(A Coh-Metrix analysis of Korean EFL learners’ summary writings in the English argumentative and expository texts). ≪영어학≫(Korean Journal of English Language and Linguistics) 19-3, 539-559
  • 정미경⋅김남국(Jeong, M-K. and N-G. Kim). 2014. Coh-Metrix를 통한 한국인 EFL학습자와 영어 모국어 화자의 논설문 코퍼스에 나타난 언어적 특성 분석(An analysis of the linguistic features on the corpus of Korean EFL learners and native English speakers with Coh-Metrix). ≪언어학연구≫(Studies in Linguistics) 33, 373-395. [https://doi.org/10.17002/sil..33.201410.373]
  • 홍성우⋅송해성(Hong, S. W. and H. S. Song). 2019. 코메트릭스(Coh-Metrix)를 활용한 2015 개정 고등학교 1학년 영어 교과서 읽기 자료의 난이도 분석(Exploring the difficulty levels of the reading materials in high school English textbooks with Coh-Metrix for the 1st year students). ≪영어어문교육≫(English Language & Literature Teaching) 25-2, 133-161.
  • Aryadoust, V. and S. Liu. 2015. Predicting EFL writing ability from levels of mental representation measured by Coh-Metrix: A structural equation modeling study. Assessing Writing 24, 35-58. [https://doi.org/10.1016/j.asw.2015.03.001]
  • Bond, T. G. and C. M. Fox. 2001. Applying the Rasch Model: Fundamental Measurement in the Human Sciences. NJ: Lawrence Erlbaum. [https://doi.org/10.4324/9781410600127]
  • Crossley, S. and D. S. McNamara, 2011. Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life-Long Learning 21, 170-191. [https://doi.org/10.1504/IJCEELL.2011.040197]
  • Crossley, S. and D. S. McNamara. 2012. Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading 35(2), 115-135. [https://doi.org/10.1111/j.1467-9817.2010.01449.x]
  • Fulcher, G. 2003. Testing Second Language Speaking. Harlow: Pearson.
  • Graesser, A. C., D. S. McNamara and J. M. Kulikowich. 2011. Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher 40, 223-234. [https://doi.org/10.3102/0013189X11413260]
  • Hyland, K. and E. Anan. 2006. Teachers’ perceptions of error: The effects of first language and experience. System 34, 509-519. [https://doi.org/10.1016/j.system.2006.09.001]
  • Johnson, J. S. and G. S. Lim. 2009. The influence of rater language background on writing performance assessment. Language Testing 26, 485-505. [https://doi.org/10.1177/0265532209340186]
  • Kane, M. T. 2013. Validating the interpretations and uses of test scores. Journal of Educational Measurement 50, 1-73. [https://doi.org/10.1111/jedm.12000]
  • Khushik, G. A. and A. Huhta. 2019. Investigating syntactic complexity in EFL learners’ writing across Common European Framework of Reference. Applied Linguistics, 1-28. [https://doi.org/10.1093/applin/amy064]
  • Knoch, U. 2009. Diagnostic assessment of writing: A comparison of two rating scales. Language Testing 26, 275-304. [https://doi.org/10.1177/0265532208101008]
  • Lee, H-K. 2009. Native and nonnative rater behavior in grading Korean students’ English essays. Asia Pacific Educational Review 10, 387-397. [https://doi.org/10.1007/s12564-009-9030-3]
  • McNamara, D. S., S. A. Crossely and P. M, McCarthy. 2010. Linguistic features of writing quality. Written Communication 27, 57-86. [https://doi.org/10.1177/0741088309351547]
  • McNamara, D. S., A. C. Graesser, P. M. McCarthy and Z. Cai. 2014. Automated Evaluation of Text Discourse with Coh-Metrix. UK: Cambridge University Press. [https://doi.org/10.1017/CBO9780511894664]
  • McNamara, T. 1996. Measuring Second Language Performance. NY: Addison Wesley.
  • Schaefer, E. 2008. Rater bias patterns in an EFL writing assessment. Language Testing 25, 465-493. [https://doi.org/10.1177/0265532208094273]
  • Shin, D. I. and H. S. Seol. 2005. Understanding rating patterns through new FACETS program. Foreign Languages Education 12, 191-212.
  • Weigle, S. C. 1998. Using FACETS to model rater training effects. Language Testing 15, 263-287. [https://doi.org/10.1177/026553229801500205]
  • Weigle, S. C. 2002. Assessing Writing. UK: Cambridge University Press. [https://doi.org/10.1017/CBO9780511732997]
  • Xu, W. and M. Liu. 2016. Using Coh-Metrix to analyze Chinese ESL learners’ writing. International Journal of Learning, Teaching and Educational Research 15, 16-26.
  • Yi, J. Y. 2008. Necessity of developing a rating scale for writing assessment in Korean high schools. Foreign Languages Education 15, 261-286.

이호 (Lee, Ho), 교수(Professor)중앙대학교 사범대학 영어교육과(Dept. of English Education, Chung-Ang University)서울시 동작구 흑석로 84(84 Heukseok-ro Dongjak-gu, Seoul, Korea)Email: holee@cau.ac.kr