Automated assessment of English written assignments using large language models

Abstract

The gap between the labour intensity of expert grading of extended written assignments in English and the demand for large-scale, rapid assessment generates a request for automation whose reliability remains contested. The aim of the study is to quantify the agreement between the scores produced by large language models and expert scores and to identify the conditions under which such agreement reaches a practically acceptable level. Simulation modelling of the grading procedure was carried out on a corpus of 600 assignments at CEFR levels A2-C1, together with a comparative analysis of five automatic configurations and the computation of agreement indicators – the quadratic weighted kappa, the shares of exact and adjacent agreement, the mean absolute error and the linear correlation. The flagship proprietary model attains a quadratic kappa of 0.77 with exact agreement of 54,0% and adjacent agreement of 89,6%, whereas smaller open models and the classical regressor remain within the interval 0.58-0.71. Agreement is heterogeneous across criteria: formal parameters – spelling, punctuation, grammar – are reproduced more accurately than content-related ones, namely task achievement and cohesion. Enriching the request with a rubric, reference samples, step-by-step reasoning and aggregation of repeated runs raises the kappa from 0.55 to 0.80. The obtained values stay below the inter-rater reliability ceiling of 0.86, which positions automatic scoring as an instrument of preliminary and auxiliary ranking rather than autonomous assignment of the final mark.