Machine Translation Quality Evaluation
ISO 5060 provides a framework for evaluating the quality of machine translation output - metrics, methods, and benchmarking for AI translation systems.
- MT Quality Metrics
- Evaluation Methods
- Benchmarking
- Error Classification
- Human vs. Automated Evaluation
Free access - just sign in
ReadSafety.com is completely free. Create a free account or sign in to unlock all ISO standard guides, the interactive Periodic Table, and the knowledge blog.
About ISO 5060:2024 - Machine Translation Quality Evaluation
ISO 5060:2024 provides a framework for evaluating the quality of machine translation (MT) output. As AI-powered translation systems become increasingly capable and widely used, the need for standardized methods to measure, compare, and benchmark MT quality has become critical for language service providers, MT developers, and organizations deploying MT in their workflows.
The standard covers the full range of MT quality evaluation approaches: human evaluation (where human raters assess MT output quality), automatic evaluation (where software metrics are used to score MT output), and hybrid approaches combining human and automatic evaluation. It specifies requirements for the design of evaluation studies, the selection of evaluation metrics, and the reporting of evaluation results.
ISO 5060 addresses the major automatic MT evaluation metrics including BLEU (Bilingual Evaluation Understudy), TER (Translation Edit Rate), chrF (character n-gram F-score), COMET (Crosslingual Optimized Metric for Evaluation of Translation), and BLEURT. The standard provides guidance on the appropriate use of each metric, their limitations, and the conditions under which they provide reliable quality signals.
For human evaluation, ISO 5060 covers evaluation frameworks including MQM (Multidimensional Quality Metrics), which provides a hierarchical taxonomy of translation error types (accuracy errors, fluency errors, terminology errors, style errors, and locale convention errors), and the Direct Assessment (DA) method, which uses continuous quality scales rather than error annotation.
ISO 5060 is used by MT system developers, translation service providers, and organizations deploying MT to establish objective quality baselines, compare MT systems, and monitor MT quality over time. It is used alongside ISO 18587 (post-editing of MT output) and ISO 11669 (translation project framework).
Source: ReadSafety.com - Free ISO Educational Resource by USQC · readsafety.com/iso/5060