A Translation Quality Score Is Not Enough to Approve an InDesign Document
I would not approve a translated InDesign document on a segment quality score alone. Two papers published this month examine the performance and reliability of automated translation evaluation.
This article is also available as a short video on YouTube: A Translation Quality Score Is Not Enough to Approve an InDesign Document
The findings of the WMT26 shared task on automated translation quality evaluation compared evaluation systems on 21 translation directions. On that benchmark, unsupervised LLM-as-a-judge approaches outperformed traditional and supervised metrics across all three primary tasks: marking error spans, predicting a quality score, and identifying translations with no errors.
A paper by researchers at Apple, accepted at EMNLP 2026, compared LLM and human annotators on two annotation schemes, MQM and ESA, using a long-context test set of 70 language pairs and data from WMT23 and WMT25. The authors report that agreement between LLMs and humans sometimes exceeds agreement between human annotators, but both vary across schemes, language pairs, and domains and remain unreliable in most settings.
The findings need not conflict. Better performance than other automated metrics can coexist with unreliable agreement with human annotators.
Neither abstract addresses IDML extraction or merge. My own tests examined those operations.
Three checks a segment score does not cover
The WMT task scores translations at the segment level. For an InDesign handoff, I check three things that a score on those segments does not tell me.
Whether the content that needs translation was extracted. Text on hidden layers, master pages, or the pasteboard is in the IDML archive whether or not it shows on the page. In Okapi's IDML filter, separate settings control whether each of those locations is extracted. Some of that text may not need translation. If text that does need it is left out of extraction, it never reaches an evaluator.
Whether the merged file kept its structure. In a test with Okapi Tikal and a synthetic IDML file, I left the target of a footnote empty and merged the file. The merge finished without an error or a warning. The footnote came back as an empty <Footnote/> element, with its paragraph structure gone. In the same test, a table cell and an ordinary story behaved the same way. The details are in Missing Footnotes, Empty Table Cells, and Stale Index Pages After IDML Translation.
An empty-target check on the bilingual file can catch that case before the merge. After the merge, comparing the source and translated InDesign files story by story, with table sizes and footnote counts, can reveal structural losses. Neither check requires reading the target language. The checklist is in Comparing InDesign Files After Translation: What to Check.
Whether the text appears on the page. Overset text remains in the story. Confirming that the text is present does not establish that it appears on the page. InDesign marks an overset frame with a red indicator. In my test, the Preflight panel's [Basic] profile listed overset table cells under TEXT > Overset text.
Using the scores
Before using an LLM judge to decide which segments a person reviews first, test it on your own material: which real errors it misses and which of its flags are wrong, for each language.
If your workflow already includes the document checks described above, a segment score gives no reason to remove them.
Sources
- Lavie et al., Findings of the WMT26 Shared Task on Automated Translation Quality Evaluation Systems: Compact Open Models are Competitive Quality Evaluators, Proceedings of the Eleventh Conference on Machine Translation (WMT), 2026.
- Almaghout, Federmann, and Gao, Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged, arXiv:2610.10918, accepted at EMNLP 2026.