Jesús Giménez and Lluís Màrquez
 A a common workbench on which automatic MT evaluation metrics can be robustly used and combined for the purpose of MT system development. Current version includes a rich set of metrics operating at different linguistic levels (lexical, shallow syntactic, syntactic, and shallow semantic).
Technical Requirements:
  • Individual metrics (BLEU, NIST, GTM, ROUGE, METEOR, ...)
  • Linguistic processors:
           - C&C Tools
                                - SwiRL
           - BIOS
           - Charniak Parser
           - MINIPAR dependency parser
           - SVMTool
  • Lexical Similarity
  • Syntactic Similarity
                 - Shallow parsing
                 - Dependency parsing
                 - Constituency parsing
  • Semantic Similarity
                 - Named entities
                 - Semantic roles
                 - Discourse representations
  • QARLA Framework
 IQmt allows for evaluation of translation quality aspects at different linguistic levels. Metric scores can also be combined into a single measure of quality, either through QARLA, of via arithmetic mean.


  • Jesús Giménez and Enrique Amigó. IQMT: A Framework for Automatic Machine Translation Evaluation. Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC'06). Genoa, Italy, 22-28 May. 2006. Departament Research Technical Report (LSI-07-29-R).
  • Jesús Giménez and Lluís Màrquez. Linguistic Features for Automatic Evaluation of Heterogeneous MT Systems. Proceedings of WMT 2007 (ACL'07).
  • Jesús Giménez and Lluís Màrquez. A Smorgasbord of Features for Automatic MT Evaluation. Proceedings of the 3rd ACL Workshop on Statistical Machine Translation (shared evaluation task).


Scroll to Top