Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 25, No. 0, pp.1496-1517
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Print publication date 31 Jan 2025
Received 19 Aug 2025 Revised 11 Sep 2025 Accepted 10 Oct 2025
DOI: https://doi.org/10.15738/kjell.25..202511.1496

자동채점 기반 영어 말하기 성적 보고서에 대한 학습자와 교사의 이해도 및 인식 탐색

YunDeok Choi ; Heyoung Kim ; Jin-Hwa Lee ; Min-Chang Sung
(First author) Assistant Professor, Department of English Education, Chungnam National University yundeokchoi@cnu.ac.kr
(Corresponding author) Professor, Department of English Education, Chung-Ang University englishnet@cau.ac.kr
(Co-author) Professor, Department of English Education, Chung-Ang University jinhlee@cau.ac.kr
(Co-author) Associate Professor, Department of English Education, Gyeongin National University of Education mcsung@ginue.ac.kr
An exploratory study on the development of an online score report for automated English speaking assessment


© 2025 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Score reporting plays an important role in supporting the validity of interpretations and uses of assessment results; however, it remains an underexplored topic particularly in domestic research contexts. This exploratory study examines how two user groups-students and teachers-understood and perceived a score report designed for an automated speaking assessment (ASA) system currently under development for Korean public education. A total of 224 Korean secondary school students completed an online survey measuring their understanding and perceptions. Five English teachers participated in online semi-structured interviews. Descriptive statistics, as well as group-independence chi-square and Mann-Whitney U tests, on the survey responses showed that the students overall held low levels of understanding but positive perceptions with no significant differences between school levels and negligible effect sizes. A thematic analysis of the interview transcripts revealed that teachers appreciated individualized feedback on the analytic aspects of speaking performance presented through visual graphs. However, they expressed concerns that the written prose feedback was too general and some terminology lacked specificity for students, providing recommendations for improvement. These findings have implications for the development of a user-friendly score report that enhances the validity of assessment interpretations and uses.

Keywords:

score reporting, understanding, perception, automated speaking assessment, secondary school

Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2023S1A5A2A03086569)

References

  • 교육부(Ministry of Education). 2022. 『2022 개정 영어과 교육과정(교육부 고시 제2022-33호 [별책 14])』(2022 Revised English Curriculum [Ministry of Education Notification No. 2022-33, Appendix 14]). 교육부(Ministry of Education).
  • 김나예⋅김정렬(Kim, N.-Y. and J.-R. Kim). 2018. 초등영어 수행평가에 대한 영어전담교사들의 인식조사(A Perception Study of Elementary School English Teachers on the English Performance Assessment). ≪교사교육연구≫(Teacher Education Research) 57(4), 529–538.
  • 박선영⋅민찬규(Park, S.-Y. and C.-K. Min). 2019. 2015 개정 교육과정 기반 중학교 1학년 영어 말하기 수행평가 시행 방안에 대한 델파이 조사 연구(A Delphi study on the implementation of English-speaking performance assessment for middle school 1st graders based on the 2015 revised English curriculum). ≪영어교과교육≫(Journal of the Korea English Education Society) 18(1), 25–45.
  • 심창용(Sim, C.-Y.). 2024. AI 디지털교과서 활용 영어교육의 기초(The basics of AIDT-powered English education). ≪ESP Review≫ 6(2), 137–154.
  • 이진화⋅최윤덕⋅성민창⋅김혜영(Lee, J.-H., Y. Choi, M.-C. Seong and H. Kim). 2023. 자동채점 기반 영어 말하기 시험 현황 분석(An analysis of the current status of automated scoring-based English speaking tests). ≪영어교육≫(English Teaching) 78(2), 223–244.
  • American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
  • Braun, V. and V. Clark. 2012. Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, and K. J. Sher, eds., APA handbook of research methods in psychology, Vol. 2: Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57-71). American Psychological Association.
  • Brown, G. T., T. M. O'Leary and J. A. Hattie. 2019. Effective Reporting for Formative Assessment: The asTTle case example. In D. Zapata-Rivera ed., Score reporting research and applications (pp. 107-125). Routledge. [https://doi.org/10.4324/9781351136501-11]
  • Creswell, J. W., and V. L. Plano Clark, 2011. Designing and conducting mixed methods research (2nd ed.). Sage.
  • Fulcher, G. 2003. Testing second language speaking. Longman.
  • Goodman, D. P. and R. K. Hambleton. 2004. Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education 17(2), 145-220. [https://doi.org/10.1207/s15324818ame1702_3]
  • Gu, L. and L. Davis. 2019 Providing SpeechRater feature performance as feedback on spoken responses. In K. Zechner and K. Evanini, eds., Automated speaking assessment (pp. 159-175). Routledge. [https://doi.org/10.4324/9781315165103-10]
  • Gu, L., L. Davis, J. Tao and K. Zechner. 2021. Using spoken language technology for generating feedback to prepare for the TOEFL iBT test: A user perception study. Assessment in Education: Principles, Policy and Practice 28(1), 58-76. [https://doi.org/10.1080/0969594X.2020.1735995]
  • Hambleton, R. K. and A. L. Zenisky. 2013. Reporting test scores in more meaningful ways: A research-based approach to score report design. In K. F. Geisinger ed., APA handbook of testing and assessment in psychology (Vol. 3): Testing and assessment in school psychology and education (pp. 479-494). American Psychological Association. [https://doi.org/10.1037/14049-023]
  • Hattie, J. 2009. Visibly learning from reports: The validity of score reports. Online Educational Research Journal, 1-15.
  • Hattie, J. and H. Timperley. 2007. The power of feedback. Review of educational research 77(1), 81-112. [https://doi.org/10.3102/003465430298487]
  • Hsieh, C.-N. 2023. Evaluating the use and interpretation of the TOEIC listening and reading test score report: Perspectives of test takers in Japan (ETS Research Report Series No. RR-23-01). Educational Testing Service. Available online at [https://doi.org/10.1002/ets2.12364]
  • Kim, A. A., M, Chapman, A. Kondo and C. Wilmes. 2020. Examining the assessment literacy required for interpreting score reports: A focus on educators of K-12 English learners. Language Testing 37(1), 54-75. [https://doi.org/10.1177/0265532219859881]
  • Larson-Hall, J. 2010. A Guide to Doing Statistics in Second Language Research Using SPSS. Routledge. [https://doi.org/10.4324/9780203875964]
  • O’Leary, T. M., J. A. Hattie and P. Griffin. 2017. Actual interpretations and use of scores as aspects of validity. Educational Measurement: Issues and Practice 36(2), 16-23. [https://doi.org/10.1111/emip.12141]
  • Sawaki, Y., and R. Koizumi. 2017. Providing test performance feedback that bridges assessment and instruction: The case of two standardized English language tests in Japan. Language Assessment Quarterly 14(3), 234-256. [https://doi.org/10.1080/15434303.2017.1348504]
  • Taylor, L. 2005. Washback and impact. ELT Journal 59(2), 154-155. [https://doi.org/10.1093/eltj/cci030]
  • Vezzu, M., W. VanWinkle and D. Zapata-Rivera. 2012. Designing and evaluating an interactive score report for students (Research Memorandum No. ETS RM-12-01). Educational Testing Service. Available online at https://www.ets.org/Media/Research/pdf/RM-12-01.pdf
  • Wainer, H. 1992. Understanding graphs and tables. Educational Researcher 21, 14-22. [https://doi.org/10.2307/1176346]
  • Zapata-Rivera, D., P. Kannan, C. Forsyth, S. Peters, A. D. Bryant, E. Guo and R. Long. 2018. Designing and evaluating reporting systems in the context of new assessments. In D. Schmorrow and C. Fidopiastis, eds., Augmented cognition: Users and contexts. AC 2018. Lecture notes in computer science (Vol. 10916, pp. 143-153). Springer. [https://doi.org/10.1007/978-3-319-91467-1_12]
  • Zenisky, A. L., and R. K. Hambleton. 2015. A model and good practices for score reporting. In S. Lane, M. R. Raymond, and T. M. Haladyna, eds., Handbook of test development (2nd ed., pp. 585-602). Routledge.