Apple researchers have published a paper on Apple Machine Learning Research comparing how well large language models and humans assess machine translation quality, concluding that while LLMs can beat human agreement in some tasks, both are unreliable for most real-world settings.
The study, by Apple scientists Hala Almaghout, Christian Federmann and Qin Gao, tests LLMs on two common translation evaluation frameworks: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The team used a long-context test set spanning 70 language pairs along with the public WMT23 and WMT25 benchmarks.
In certain evaluation scenarios, the agreement between an LLM and a human annotator climbed higher than the agreement between two human annotators. But performance varied significantly across annotation schemes, language pairs and domains. The researchers note that humans have particular trouble with fine-grained MQM annotations and low-resource languages, while LLMs often miss minor errors, confuse language variants and struggle with error span annotation. The paper suggests pairing humans with LLMs in collaborative pipelines could help address these reliability gaps.
Why it matters
Apple’s findings add a measured voice to debates about whether LLMs can replace human judgement in machine translation evaluation. Our view: the results confirm that neither humans nor LLMs are consistently trustworthy for detailed annotation tasks, which makes hybrid human-model workflows the most practical path forward for now.