The Korean Association for the Study of English Language and Linguistics

Korean Journal of English Language and Linguistics - Vol. 21
[Paper List] [Goto Volume List]

[ Article ]
Korea Journal of English Language and Linguistics - Vol. 21, No. 0, pp. 375-391
Abbreviation: KASELL
ISSN: 1598-1398 (Print) 2586-7474 (Online)
Received 23 Mar 2021 Revised 20 Apr 2021 Accepted 25 Apr 2021
DOI: https://doi.org/10.15738/kjell.21..202104.375

Exploring the Use of An Artificial Intelligence Chatbot as Second Language Conversation Partners
Dongkwang Shin ; Heyoung Kim ; Jang Ho Lee ; Hyejin Yang
(1st author) Professor, Gwangju National Univ. of Education (sdhera@gamail.com)
(corresponding author) Professor, Chung-Ang Univ. (englishnet@cau.ac.kr)
(co-author) Professor, Chung-Ang Univ. (jangholee@cau.ac.kr)
(co-author) Researcher, Chung-Ang Univ. (hjyang1112@gmail.com)


© 2021 KASELL All rights reserved
This is an open-access article distributed under the terms of the Creative Commons License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Funding Information ▼
Abstract

This study investigated the appropriateness of using artificially intelligent chatbots as conversation partners for second language (L2) learners. 27 Korean high school and 26 college students had a task-oriented conversation with a text-based chatbot, Mitsuku, for 20 minutes. Chat log data were collected and analyzed quantitatively and qualitatively in terms of the quantity of students’ utterances and their vocabulary levels, along with the degree of conversation task success between the chatbot and its users. Both groups finished their tasks, successfully developing conversations with the chatbot and producing double the expected minimum quantity of utterances, although their performances varied individually. Mitsuku’s vocabulary was deemed appropriate for L2 learners' proficiency. The college students used conversational strategies more appropriately than their high school counterparts. Nevertheless, a sentiment analysis showed that the high school students enjoyed talking with Mitsuku to a greater extent than the college students. These results suggest that the chatbot offers L2 learners substantial opportunities as a conversation partner.

Keywords: Artificial Intelligence (AI), chatbot, Mitsuku, conversation task, vocabulary level, task success rate, sentiment analysis, Orange 3.28.0
Acknowledgments

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2018S1A5A2A03037255).

References
1. Abu-Shawar, B. 2017. Integrating CALL systems with chatbots as conversational partners. Computación y Sistemas 21(4), 615–626.
2. AiDreams. 2013. Steve Worswick interview: Loebner 2013 winner. Ai Dreams Forum. Retrieved from https://aidreams.co.uk/forum/index.php?page=Steve_Worswick_Interview_-_Loebner_2013_winner#.XDlXn1UzaM8
3. Alm, A. and L. M. Nkomo. 2020. Chatbot experiences of informal language learners: A sentiment analysis. International Journal of Computer-Assisted Language Learning and Teaching 10(4), 51–65.
4. Batacharia, B., D. Levy, R. Catizone, A. Krotov and Y. Wilks. 1999. Converse: A conversational companion. In Y. Wilks, ed., Machine Conversations, 205–215. Boston/Dordrecht/London: Kluwer.
5. Behnam, B. and Y. Pouriran. 2009. Classroom discourse: Analyzing teacher/learner interactions in Iranian EFL task-based classrooms. Porta Linguarum 12, 117–132.
6. Chon, Y. V. and D. Shin. 2012. Lexical profiles and socioeducational variables of Korean EFL university learners. Korean Journal of Applied Linguistics 28(1), 115–146.
7. Colby, K. 1999. Human-computer conversation in a cognitive therapy program. In Y. Wilks, ed., Machine Conversations. 9–19. Boston/Dordrecht/London: Kluwer.
8. Compton, L. K. L. 2009. Preparing language teachers to teach language online: A look at skills, roles, and responsibilities. Computer Assisted Language Learning 22(1), 73–99.
9. Coniam, D. 2014. The linguistic accuracy of chatbots: Usability from an ESL perspective. Text & Talk 34(5), 545–567.
10. Dahl, D. A., M. Bates, M. Brown, W. Fisher, K. Hunicke-Smith, D. Pallett, C. Pao, A. Rudnicky and E. Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. Proceedings of the Human Language Technology Workshop, 43–48.
11. Demsar, J., T. Curk, A. Erjavec, C. Gorup, T. Hocevar, M. Milutinovic, M. Mozina, M. Polajnar, M. Toplak, A. Staric, M. Stajdohar, L. Umek, L. Zagar, J. Zbontar, M. Zitnik and B. Zupan. 2021. Orange 3.28.0: Data Mining Toolbox in Python [Computer Software]. Retrieved from https://orangedatamining.com/download/#windows
12. Fryer, L. K., D. Coniam, R. Carpenter and D. Lăpușneanu. 2020. Bots for language learning now: Current and future directions. Language Learning & Technology 24(2), 8–22.
13. Grice, M. P. 1989. Logic and conversation. In P. Cole and J. L. Morgan, eds., Syntax and Semantics, Vol. 3: Speech Acts, 242–280. New York: Academic Press.
14. Heatley, A., I. S. P. Nation and A. Coxhead. 2002. Range Program [Computer Software]. http://www.victoria.ac.nz/lals/staff/paul-nation/nation.aspx.
15. Hutto, C. J. and E. Gilbert. 2014. VADER: A Parsimonious rule-based model for sentiment analysis of social media text. Paper presented at the 8th International Conference on Weblogs and Social Media (ICWSM-14).
16. IELTS. 2018. Sample test questions. Retrieved from https://www.ielts.org/-/media/pdfs/115041_speaking_sample_task_-_part_1.ashx?la=en
17. Kanda, T., T. Hirano and D. Eaton. 2004. Interactive robots as social partners and peer tutors for children: A field trial. Human-Computer Interaction 19, 61–84.
18. Krashen, S. 1980. The theoretical and practical relevance of simple codes in L2 acquisition. In R. Scarcella and S. Krashen, eds., Research in L2 Acquisition, 7–18. Rowley, Ma: Newbury House.
19. Krashen, S. 1981. L2 Acquisition and L2 Learning. Oxford: Pergamon Press.
20. Kwon, O.-W., K. S. Lee, Y.-K. Kim and Y. Lee. 2015. GenieTutor: A computer assisted second language learning system based on semantic and grammar correctness evaluations. Proceedings of the EUROCALL 2015, 330–335.
21. Lee, J. H., H. Yang, D. Shin and H. Kim. 2020. Chatbots, ELT Journal 74(3), 338–344.
22. Lu, C.-H., G.-F. Chiou, M.-Y. Day, C.-S. Ong and W.-L. Hsu. 2006. Using instant messaging to provide an intelligent learning environment. Proceedings of the Intelligent Tutoring Systems (ITS) 2006 Lecture Notes in Computer Science: Vol 4053, 575–583.
23. Nation, I. S. P. 2001. Learning Vocabulary in Another Language. Cambridge: Cambridge University Press.
24. Nation, I. S. P. and D. Beglar. 2007. A vocabulary size test. The Language Teacher 31(7), 9–13.
25. Nation, I. S. P. 2012. The BNC-COCA Word Family Lists [Computer Software]. Retrieved from http://www.victoria.ac.nz/lals/about/staff/publications/BNC_COCA_25000.zip
26. Shin. D. 2014. What vocabulary are we learning? The Journal of Foreign Studies 30, 63–95.
27. Shin, D. 2019. Exploring the feasibility of AI chatbots as a tool for improving learners’ writing competence of English. Korean Journal of Teacher Education 35(1), 41–55.
28. Shin, D., Y. V. Chon and H. Kim. 2011. Receptive and productive vocabulary sizes of high-school learners: What next for the basic word list? English Teaching 66(3), 123–148.
29. Stewart, J. 2014. Do multiple-choice options inflate estimates of vocabulary size on the VST? Language Assessment Quarterly 11(3), 271–282.
30. Stewart, J. and D. A. White. 2011. Estimating guessing effects on the vocabulary levels test for differing degrees of word knowledge. TESOL Quarterly 45(2), 370–380.
31. Young, R. and M. Milanovic. 1992. Discourse variation in oral proficiency interviews. Studies in L2 Acquisition 14(4), 403–424.
32. Wang, Y. F. and S. Petrina. 2013. Using learning analytics to understand the design of an intelligent language tutor–Chatbot lucy. Editorial Preface 4(11), 124–131.
33. Wallace, R. S. 2009. The anatomy of ALICE. In R. Epstein, G. Roberts and G. Beber, eds., Parsing the Turing Test, 181–210. Dordrecht: Springer.
34. Weizenbaum, J. 1966. ELIZA: A computer program for the study of natural language communication between man and machine. Communications of the Association for Computing Machinery 9, 36–45.