To start with lets explain what the FEMTI is. The Framework for Machine Translation Evaluation in ISLE is a resource that helps MT evaluators define contextual evaluation plans. It consist on two interrelated classifications:
- It lists possible characteristics of the contexts of use that are applicable to MT systems.
- It lists the possible characteristics of an MT system, along with the metrics that were proposed to measure them.
FEMTI proposes a set of quality characteristics that are relevant to that context, using its embedded knowledge base. Evaluators can modify this set of quality characteristics and select evaluation metrics for each of them, by browsing the second classification. Evaluators can then print the evaluation plan and execute the evaluation.
According to FEMTI the main characteristics of a translation task are the following:
- Assimilation: “The ultimate purpose of the assimilation task (of which translation forms a part) is to monitor a (relatively) large volume of texts produced by people outside the organization, in (usually) several languages.”
- Dissemination: “The ultimate purpose of dissemination is to deliver to others a translation of documents produced inside the organization.”
- Communication: “The ultimate purpose of the communication task is to support multi-turn dialogues between people who speak different languages. The translation quality must be high enough for painless conversation, despite possible syntactically ill-formed input and idiosyncratic word and format usage. The ultimate purpose of dissemination is to deliver to others a translation of documents produced inside the organization.”
- Eduard Hovy, Margaret King & Andrei Popescu-Belis. (2003). FEMTI – a Framework for the Evaluation of Machine Translation in ISLE. Retrieved May 5, 2008, 11:24 from http://www.issco.unige.ch:8080/cocoon/femti/st-home.html
- Characteristics: Retrieved May 5, 2008, 11:39 from http://www.isi.edu/natural-language/mteval/framed-glossary.html