Evaluating legal text translation using translation quality assessment: Comparative studies on undergraduates and postgraduates’ translation products

Chairunisa Sonya Sastrawinata; Esti Junining; Sugeng Hariyanto

doi:10.61251/ijoep.v3i2.444

Authors

Chairunisa Sonya Sastrawinata Languages and Literature Department, Faculty of Cultural Studies, Universitas Brawijaya, Indonesia
Esti Junining Languages and Literature Department, Faculty of Cultural Studies, Universitas Brawijaya, Indonesia https://orcid.org/0000-0001-6625-4167
Sugeng Hariyanto English for Tourism Industry Program, Politeknik Negeri Malang, Indonesia https://orcid.org/0000-0001-7492-8096

DOI:

https://doi.org/10.61251/ijoep.v3i2.444

Keywords:

comparative studies, legal texts, translation, translation quality assessment

Abstract

This study examines differences in the quality of legal translation between undergraduate and postgraduate EFL students using the Translation Quality Assessment (TQA) framework. Legal translation requires not only linguistic competence but also a deep understanding of legal concepts and contextual accuracy, making it a complex and specialized field. Although translation competence models and legal translation theories have extensively explained the knowledge and skills required for legal translators, limited empirical research has examined how these theoretical competencies are reflected in the actual translation performance of students at different academic levels. This study addresses this gap by comparing the legal translation quality of undergraduate and postgraduate students using the Translation Quality Assessment (TQA) framework. This research employed a quantitative design involving 33 undergraduate and 6 postgraduate students, who completed a legal translation task comprising 26 sentences. The translated outputs were evaluated based on three criteria: accuracy, readability, and acceptability. Statistical analysis, including the Mann–Whitney U test, was conducted to determine significant differences between the two groups. The findings reveal a statistically significant difference in translation quality (p = 0.009), with postgraduate students outperforming undergraduates. Postgraduate students demonstrated greater ability to apply contextually appropriate and legally informed translation strategies, while undergraduate students tended to rely on literal translation. These results indicate that advanced academic exposure and legal knowledge significantly influence translation performance. This study highlights the importance of integrating legal competence and translation competence to enhance students’ professional readiness in handling legal texts.

Author Biographies

Chairunisa Sonya Sastrawinata, Languages and Literature Department, Faculty of Cultural Studies, Universitas Brawijaya, Indonesia

Chairunisa Sonya Sastrawinata is a postgraduate student in Linguistics at Universitas Brawijaya with a research focus on Translation Studies, particularly legal translation and translation strategies. She holds a bachelor's degree in English Language and Literature from Universitas Brawijaya and has published research on figurative language translation. Her academic interests include translation analysis, legal translation, applied linguistics, and cross-cultural communication. She can be contacted at sastrawinata.sonya@gmail.com.

Esti Junining, Languages and Literature Department, Faculty of Cultural Studies, Universitas Brawijaya, Indonesia

Esti Junining is an Associate Professor in the Department of Languages and Literature, Faculty of Cultural Studies, Universitas Brawijaya. Her expertise includes Applied Linguistics, Translation Studies, English Language Teaching, and Critical Thinking. She holds her doctorate degree in English Language Education in 2013 and has published extensively in linguistics and language education research. She can be contacted at esti@ub.ac.id.

Sugeng Hariyanto, English for Tourism Industry Program, Politeknik Negeri Malang, Indonesia

Sugeng Hariyanto a senior lecturer and professional translator affiliated Politeknik Negeri Malang. His expertise includes Translation Studies, localization, and translation technology. He is also a member of the Competency and Certification Committee of the Indonesian Translators Association (HPI) and has published extensively in translation and language studies. He can be contacted at sugeng.hariyanto@polinema.ac.id.

References

Alghamdi, F., & Alotaibi, H. (2025). Using AI in translation quality assessment: A case study of ChatGPT and legal translation texts. Electronics, 14(19), 1-28. https://doi.org/10.3390/electronics14193893

Alkatheery, E. R. (2023). Google Translate errors in legal texts: Machine translation quality assessment. Arab World English Journal for Translation and Literary Studies, 7(1), 208-219. https://doi.org/10.24093/awejtls/vol7no1.16

Almahasees, Z., Albudairi, Y., Al-Natour, M., Al-Harahsheh, A., & Mahmoud, S. (2025). Challenges and strategies in translating legal terms between English and Arabic: A comparative study of international accords and agreements. Forum for Linguistic Studies, 7(1), 609–625. https://doi.org/10.30564/fls.v7i1.7565

Baker, M. (1992). In other words: A coursebook on translation. Routledge. https://doi.org/10.4324/9780203133590

Baker, M. (2011). In other words: A coursebook on translation (2nd ed.). Routledge.

Briva-Iglesias, V., Camargo, J., & Dogru, G. (2024). Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?. Computer Science: Computation and Language, 1, 1-31. https://doi.org/10.48550/arxiv.2402.07681

Cao, D. (2007). Translating law. Multilingual Matters. https://doi.org/10.21832/9781853599552

Ding, L. (2024). A comparative study on the quality of English-Chinese translation of legal texts between ChatGPT and neural machine translation systems. Theory and Practice in Language Studies, 14(9), 2823-2833. https://doi.org/10.17507/tpls.1409.18

El-daly, H. (2022). Linguistic and socio-cultural approaches to translation: Theoretical and pedagogical reflections. International Journal of Comparative Literature and Translation Studies, 9, 50-72. http://www.ijelr.in/9.4.22/50-72%20Hosni%20Mostafa%20El-daly.pdf

Ginting, D. F., Lubis, S., & Mono, U. (2020). Translation techniques and quality of the translation of legal text on immigration law. International Journal of Humanity Studies (IJHS). 3 (2) 228-238. https://doi.org/10.24071/ijhs.v3i2.2171

Haigh, M. J. (2002). Internationalisation of the curriculum: Designing inclusive education for a small world. Journal of Geography in Higher Education, 26, (1). 49–66. https://doi.org/10.1080/03098260120110368.

Hu, P. C., & Cheng, L. (2016). A study of legal translation from the perspective of error analysis. International Journal of Legal Discourse, 1(1), 235-252. https://doi.org/10.1515/ijld-2016-0007

Kupriyanova, A., Shkilev, R., Litwinowa, M., Vorotilina, T., Tagibov, K., & Shichiyakh, R. (2023). Legal regulation of the reliability and quality of translations of official documents and texts. Journal of Law and Sustainable Development, 11(11), 1-16. https://doi.org/10.55908/sdgs.v11i11.1330

Lesznyak, M., & Balogh, D. (2019). Comparative analysis of translations prepared by students with and without legal qualifications. Comparative Legilinguistics, 37, 85–115. https://doi.org/10.14746/cl.2019.37.3

Mahapatra, S., Datta, D., Soni, S., Goswami, A., & Ghosh, S. (2023). MILPaC: A novel benchmark for evaluating translation of legal text to Indian languages. ACM Transactions on Asian and Low-Resource Language Information Processing, 24, 1 - 30. https://doi.org/10.1145/3748313

Manipuspika, Y. (2021). The effectiveness of Translation Quality Assessment (TQA) models in the translation of Indonesian students. International Journal of Linguistics, Literature and Translation, 4 (5), 287-297. https://doi.org/10.32996/ijllt.2021.4.5.32

Molina, L., & Albir, A.H. (2002). Translation techniques revisited: A dynamic and functionalist approach. Meta, 47(4), 498–512. https://doi.org/10.7202/008033ar

Muhaya, R. E., & Nugroho, R. (2024). Features of legal language and its translation analysis in Indonesian-English “Settlement Termination Agreement”. Journal of English Language Teaching and Linguistics, 9(1), 47-63. https://doi.org/10.21462/jeltl.v9i1.1205

Munday, J. (2008). Introducing translation studies: Theories and applications (2nd ed.). Routledge

Newmark, P. (1998). A textbook of translation. Prentice Hall.

Pascarina, H., & Saefudin, Y. (2019). Legal text translation: Translation quality of H.L.A. Hart’s the concept of law. Atlantis Press. https://doi.org/10.2991/icglow-19.2019.89

Prieto Ramos, F., & Guzmán, D. (2024). The impact of specialised translator training and professional experience on legal translation quality assurance: an empirical study of revision performance. Interpreter and Translator Trainer, 18, 313 - 337. https://doi.org/10.1080/1750399x.2024.2344948

Pungă, L., & Ungurean, D. (2022). Quality assessment in legal translation: A case study. Romanian Journal of English Studies, 19, 22 - 29. https://doi.org/10.2478/rjes-2022-0004

Ramos, P. F. (2011). Developing legal translation competence: An integrative process-oriented approach. Comparative Linguistics, 5, 7-21. https://doi.org/10.14746/cl.2011.5.01

Ramos, P. F. (2014). Legal translation studies as an interdiscipline: Scope and evolution. Meta, 59(2), 260–277. https://doi.org/10.7202/1027475ar

Rivera-Trigueros, I. (2021). Machine translation systems and quality assessment: a systematic review. Language Resources and Evaluation, 56, 593 - 619. https://doi.org/10.1007/s10579-021-09537-5

Šarčević, S. (2000). New approach to legal translation (Reprinted ed.). Kluwer Law International.

Šarčević, S. (2012). Challenges to the Legal Translator. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199572120.013.0014

Seo, J. M. (2023). A Study on the Translation Strategies of Legal Terminology in Korean-English Translation – Centered around Legal Terms in the Intellectual Property Law. Asia-pacific Journal of Convergent Research Interchange, 9(7), 375-384. https://doi.org/10.47116/apjcri.2023.07.30

Șimon, S., Dejica, D., Fărcașiu, M. A., & Kilyeni, A. (2022). New trends in translation and interpreting studies: Linguistic accessibility in Romania. Open Linguistics, 8, 459–474. https://doi.org/10.1515/opli-2022-0217

Stepanova, V. V. (2017). Translation strategies of legal texts (English-Russian). Procedia - Social and Behavioral Sciences, 237, 1329–1336. https://doi.org/10.1016/j.sbspro.2017.02.218

Wiesmann, E. B. M. (2019). Machine translation in the field of law: A study of the translation of Italian legal texts into German. Comparative Legilinguistics, 37, 117-153. https://doi.org/10.14746/cl.2019.37.4

Williams, M. (2001). The application of argumentation theory to translation quality assessment. Meta: Journal Des Traducteurs, 46(2), 326–344. https://doi.org/10.7202/004605ar

Zeifert, M., & Tobor, Z. (2022). Legal translation versus legal interpretation: A legal-theoretical perspective. Int J Semiot Law 35, 1671–1687. https://doi.org/10.1007/s11196-021-09837-7

Yu, A. (2023). Quality evaluation of C-E translation of legal texts by mainstream machine translation systems—an example of DeepL and Metasota. English Language Teaching and Linguistics Studies, 5(2), 180-199. https://doi.org/10.22158/eltls.v5n2p180