---
title: "Apple study finds LLMs match humans on some translation checks, but both fall short"
description: "A new paper from Apple researchers shows large language models can outperform human annotators in certain machine translation quality tasks, but neither is reliable in most cases."
url: https://mactoreality.com/apple-study-finds-llms-match-humans-on-some-translation-checks-but-both-fall-short
published: 2026-10-13T07:14:27Z
updated: 2026-10-13T07:14:27Z
author: "Mac to Reality Newsdesk"
category: News
tags: ["Apple Intelligence & Siri","Software"]
publisher: Mac to Reality
ai_assisted: true
apple_heat_impact: 0
---

# Apple study finds LLMs match humans on some translation checks, but both fall short

*A new paper from Apple researchers shows large language models can outperform human annotators in certain machine translation quality tasks, but neither is reliable in most cases.*

**Key points**

- Apple researchers evaluated LLMs on MQM and ESA schemes using data covering 70 language pairs.
- LLMs exceeded human-human agreement in some evaluation tasks.
- Both humans and LLMs struggle with fine-grained error types and low-resource languages.

Apple researchers have [published a paper on Apple Machine Learning Research](https://machinelearning.apple.com/research/llm-translation-quality-annotation) comparing how well large language models and humans assess machine translation quality, concluding that while LLMs can beat human agreement in some tasks, both are unreliable for most real-world settings.

The study, by Apple scientists Hala Almaghout, Christian Federmann and Qin Gao, tests LLMs on two common translation evaluation frameworks: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The team used a long-context test set spanning 70 language pairs along with the public WMT23 and WMT25 benchmarks.

In certain evaluation scenarios, the agreement between an LLM and a human annotator climbed higher than the agreement between two human annotators. But performance varied significantly across annotation schemes, language pairs and domains. The researchers note that humans have particular trouble with fine-grained MQM annotations and low-resource languages, while LLMs often miss minor errors, confuse language variants and struggle with error span annotation. The paper suggests pairing humans with LLMs in collaborative pipelines could help address these reliability gaps.

## Why it matters

Apple’s findings add a measured voice to debates about whether LLMs can replace human judgement in machine translation evaluation. Our view: the results confirm that neither humans nor LLMs are consistently trustworthy for detailed annotation tasks, which makes hybrid human-model workflows the most practical path forward for now.

## Sources

- [Apple Machine Learning Research: Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged](https://machinelearning.apple.com/research/llm-translation-quality-annotation)

*This report was written by the Mac to Reality newsdesk with AI assistance from the sources listed above, under the editorial policy at https://mactoreality.com/editorial-policy.*