Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Association for the Study of English Language and Linguistics
[ Article ]
Korea Journal of English Language and Linguistics - Vol. 19, No. 3, pp.452-474
ISSN: 1598-1398 (Print)
Print publication date 30 Sep 2019
Received 25 Jul 2019 Revised 10 Sep 2019 Accepted 19 Sep 2019
DOI: https://doi.org/10.15738/kjell.19.3.201909.452

A Comparative Analysis of Koreans’ English Writings and Google Translations Using Coh-Metrix 3.0

Joonkoo Kim ; Kyuyun Lim*
Ph.D Student, Graduate School of English Education Chung-Ang University/Changdeok Girls’ Middle School 22, Jeongdong-gil, Jung-gu, Seoul, Republic of Korea viali@sen.go.kr
MA Student, Language and Literacy Education Department University of British Columbia 6445 University Blvd, Vancouver, Canada, Tel: +1-604-785-8132 kyuyun.lim@alumni.ac.kr

* Joonkoo Kim is the first author, and Kyuyun Lim is an MA student in language and literacy education.

Abstract

The purpose of the present study is to identify the corpus-based differences between Koreans’ English writings and their corresponding Google translations. For this purpose, the present study utilized Coh-Metrix 3.0 and conducted comparative analyses on two types of writings in terms of 12 benchmarks of text analysis. Coh-Metrix 3.0 provided numeric values for the following selected categories of text analysis: (a) basic counts (i.e., DESSC, DESWC, and DESSL), (b) lexical aspects (i.e., WRDFRQc and LDTTRc), (c) readability (i.e., RDFRE and RDFKGL) (d) syntactic complexity (i.e., SYNLE, SYNNP, and SYNSTRUTa), and (e) cohesion (i.e., CRAFAOa and LSASS1). Each output for 5 categories computed by Coh-Metrix 3.0 was then statistically processed in order to find statistically significant differences. The quantitative findings, given the small sample size associated with lower statistical power and non-normality of some data sets, were interpreted together with results from a robust technique of bootstrapped independent t-tests since the employment of bootstrapping has been empirically justified in the field of applied linguistics (Plonsky 2013, 2014). The overall findings indicated that Google translations tend to produce significantly more words before main verbs and longer sentences compared to human writings. Furthermore, it was also found that Google translations were significantly less readable, but more cohesive. However, there were no significant differences observed in lexical aspects.

Keywords:

Coh-Metrix 3.0, Koreans’ English writings Google translations, human writings, Google Translate

References

  • Ahn, B. and Y. Ma. 2015. A Coh-metrix analysis of elementary school English textbooks. English 21 28(3). 435-460. [https://doi.org/10.35771/engdoi.2015.28.3.019]
  • Ahrenberg, L. 2017. Comparing machine translation and human translation: A case study. Proceedings of The First Workshop on Human-Informed Translation and Interpreting Technology (HiT-IT), 21–28. [https://doi.org/10.26615/978-954-452-042-7_003]
  • Aiken, M. and S. Balan. 2011. An analysis of Google Translate accuracy. Translation Journal 16(2), 12-34.
  • Aryadoust, V. 2016. Understanding the growth of ESL paragraph writing skills and its relationships with linguistic features. Educational Psychology 36(10), 1742-1770. [https://doi.org/10.1080/01443410.2014.950946]
  • Bellos, D. 2011. Is That a Fish in Your Ear? Translation and the Meaning of Everything. New York: Faber and Faber.
  • Castelvecchi, D. 2016. Deep learning boosts Google translate tool. Nature. Retrieved from https://www.nature.com/news/deep-learning-boosts-google-translate-tool-1.20696 [https://doi.org/10.1038/nature.2016.20696]
  • Colina, S. 2009. Further evidence for a functionalist approach to translation quality evaluation. Target 21(2), 235-264. [https://doi.org/10.1075/target.21.2.02col]
  • CORREA, M. 2014. Leaving the ‘peer’ out of peer-editing: Online translators as a pedagogical tool in the Spanish as a second language classroom. Latin American Journal of Content & Language Integrated Learning 7(1), 1-20. [https://doi.org/10.5294/laclil.2014.7.1.1]
  • Costa-jussá, M., M. Farres and J. Pons. 2012. Machine translation in medicine: A quality analysis of statistical machine translation in the medical domain. Paper presented at the Advanced Research in Scientific Areas.
  • Ducar, C. and D. H. Schocket. 2018. Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google Translate. Foreign Language Annals 51(4), 779-795. [https://doi.org/10.1111/flan.12366]
  • Enkin, E. and E. Mejías-Bikandi. 2016. Using online translators in the second language classroom: Ideas for advanced-level Spanish. Latin American Journal of Content & Language Integrated Learning 9(1), 138-158. [https://doi.org/10.5294/laclil.2016.9.1.6]
  • Ferris, D. R., H. Liu, A. Sinha and M. Senna. 2013. Written corrective feedback for individual L2 writers. Journal of Second Language Writing 22(3), 307-329. [https://doi.org/10.1016/j.jslw.2012.09.009]
  • Garcia, I. 2010. Is machine translation ready yet? Target-International Journal of Translation Studies 22(1), 7-21. [https://doi.org/10.1075/target.22.1.02gar]
  • Garcia, I. and M. I. Pena. (2011). Machine translation-assisted language learning: Writing for beginners. Computer Assisted Language Learning 24(5), 471-487. [https://doi.org/10.1080/09588221.2011.582687]
  • Google. 2012, April 26. Breaking down the language barrier - six years in. Retrieved from https://translate.googleblog.com/2012/04/breaking-down-language-barriersix-years.html
  • Graesser, A. C. and D. S. McNamara. 2011. Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science 3(2), 371-398. [https://doi.org/10.1111/j.1756-8765.2010.01081.x]
  • Graesser, A. C., D. S. McNamara, M. M. Louwerse and Z. Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers 36(2), 193-202. [https://doi.org/10.3758/BF03195564]
  • Groves, M. and K. Mundt. 2015. Friend or foe? Google translate in language for academic purposes. English for Specific Purposes 37, 112-121. [https://doi.org/10.1016/j.esp.2014.09.001]
  • Hampshire, S. and C. Porta Salvia. (2010). Translation and the internet: Evaluating the quality of free online machine translators. Quaderns: Revista De Traducci 17, 197-209.
  • I. Solnyshkina, M. V., E. Harkova and S. A. Kiselnikov. 2014. Comparative Coh-metrix analysis of reading comprehension texts: Unified (Russian) state exam in English vs Cambridge first certificate in English. English Language Teaching 7(12), 65. [https://doi.org/10.5539/elt.v7n12p65]
  • Jeon, M-G. and I-J. Lim. 2009. A Corpus-based Analysis of middle school English 1 textbooks with Coh-Metrix. English Language Teaching 21(4), 265-292. [https://doi.org/10.17936/pkelt.2009.21.4.012]
  • Kaltenbacher, M. 2000. Aspects of universal grammar in human versus machine translation. In A. Chesterman, N. San Savador and Y. Gambier eds., Translation in Context, 221-230. Amsterdam: John Benjamins. [https://doi.org/10.1075/btl.39.25kal]
  • Kapadya, A. 2018, Sep 20. Bringing hope to a refugee family, using Google Translate [Blog post]. Retrieved from https://www.blog.google/products/translate/refugee-family-translate/
  • Kim, J. 2014. Continuity problems of elementary and secondary English education: Sequence analysis of English textbooks. Language Research 50(1), 161-184.
  • Kim, J-K. 2018. A Corpus-based investigation of reading passages in the national assessment of educational achievement English test using coh-metrix. Secondary English Education 11(2), 27-51.
  • Kirchhoff, K., A. M. Turner, A. Axelrod and F. Saavedra. 2011. Application of statistical machine translation to public health information: A feasibility study. Journal of the American Medical Informatics Association 18(4), 473-478. [https://doi.org/10.1136/amiajnl-2011-000176]
  • Krishnamoorthy, K., F. Lu and T. Mathew. 2007. A parametric bootstrap approach for ANOVA with unequal variances: Fixed and random models. Computational Statistics & Data Analysis 51(12), 5731-5742. [https://doi.org/10.1016/j.csda.2006.09.039]
  • Kroulek, A. 2017, Feb 28. A Translation Showdown: Man vs Machine Translation. Retrieved February 27, 2019, from https://www.k-international.com
  • LaFlair, G. T., J. Egbert and L. Plonsky. 2015. Bootstrapping. In L. Plonsky ed., Advancing Quantitative Methods in Second Language Research, 135-150. Routledge.
  • Lee, J. 2018. A Coh-metrix analysis of lexical, syntactic and discourse aspects in the newspaper articles of Korean and British university students. The Modern English Education Society 19(4), 17-26. [https://doi.org/10.18095/meeso.2018.19.4.17]
  • Lewis-Kraus, G. 2016, Dec 14. The great A.I. awakening. Retrieved from https://search.proquest.com/docview/1848431038
  • McCarthy, P. M., C. Hall, N. D. Duran, M. Doiuchi, Y. Fujiwara, B. Duncan and D. S. McNamara. 2011. Analyzing journal abstracts written by Japanese, American, and British scientists using Coh-metrix, and the Gramulator. The ESPecialist: Research in Language for Specific Purposes 30(2). 141-173.
  • McNamara, D. S., A. C. Graesser, P. M. McCarthy and Z. Cai. 2014. Coh-Metrix: Automated Evaluation of Text and Discourse. Boston: Cambridge University Press. [https://doi.org/10.1017/CBO9780511894664]
  • Plonsky, L. 2013. Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition 35(4), 655–687. [https://doi.org/10.1017/S0272263113000399]
  • Plonsky, L. 2014. Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform, Modern Language Journal 98(1), 450–470. [https://doi.org/10.1111/j.1540-4781.2014.12058.x]
  • Polio, C. and H. Yoon. 2018a. The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics 28(1), 165-188. [https://doi.org/10.1111/ijal.12200]
  • Ramati, I. and A. Pinchevski. 2018. Uniform multilingualism: A media genealogy of Google translate. New Media & Society 20(7), 2550-2565. [https://doi.org/10.1177/1461444817726951]
  • Rodreguez-Castro, M., S. Salas and T. Benson. 2018. To Google translate™ or not? Newcomer Latino communities in the middle. Middle School Journal 49(2), 3-9. [https://doi.org/10.1080/00940771.2017.1413270]
  • Sheppard, F. 2011. Medical writing in English: The problem with Google translate. La Presse Medicale 40(6), 565-566. [https://doi.org/10.1016/j.lpm.2011.02.024]
  • Tsai, S-C. 2019. Using Google Translate in EFL drafts: A preliminary investigation, Computer Assisted Language Learning 32(5-6), 510-526. [https://doi.org/10.1080/09588221.2018.1527361]
  • Turner, A. M., M. Bergman, M. Brownstein, K. Cole and K. Kirchhoff. 2014. A comparison of human and machine translation of health promotion materials for public health practice: Time, costs, and quality. Journal of Public Health Management and Practice 20(5), 523-529. [https://doi.org/10.1097/PHH.0b013e3182a95c87]
  • Turovsky, B. 2016, April 28. Ten years of Google translate [Blog post]. Retrieved from https://www.blog.google/products/translate/ten-years-of-google-translate, /
  • Wang, Z. and S-A. Lee. 2018. Machine translation versus human translation: The case of English-to-Chinese translation of relative clauses. Language and Information 22(1), 175. [https://doi.org/10.29403/LI.22.1.8]
  • Ye, D. 2013. A Coh-metrix analysis of language varieties between the journal articles of Chinese and American scientists. International Journal of English Linguistics 3(4), 63-70. [https://doi.org/10.5539/ijel.v3n4p63]