This page is taken from an Appendix C of Technical Report MCCS-96-300 Evaluation of the Mikrokosmos System as of 1996.
Since the time of the initial testing of Mikrokosmos, Mikrokosmos has been improved significantly, both in the quality and accuracy of its static knowledge sources and its testing environment and toolkits. As a result, we are getting much better results in word sense disambiguation. For example, Mikrokosmos is able to disambiguate open class words in Spanish texts with better than 90% correctness.
This addendum presents the latest results, the current testing environment that minimized manual checking, and lists some of the remaining problems that we are currently working on.
For example:
Total # constraints checked in Text 1: 4857
Average # constraints per sentence: 231
Average # constraints per word: 14
Checking each constraint involves traversing a path through several concepts in the ontology. It is practically impossible to deal with this amount of data manually.
We designed a new testing program with the following functionality:
Input Sentence: Doctor Andreu cuya fama...
Step 1. Bind: 2 binding problems noted: OBTENER and ESTAR.OBTENER did not bind because there is no OBJ in the f-str (details of syn-struc's and f-str's)....
Step 2. Run-analyzer: Output TMR....
Correct senses chosen for: ÁREA,...
Incorrect senses chosen for: DOCTOR-ANDREU, OBTENER,..
Correct senses unknown for: DE, PARA,...
NULL-SEM: EN76 by SEDE-CENTRAL-N1
Note: A NULL-SEM is a mechanism used to indicate a situation where one word ``consumes'' the meaning of another thereby nullifying any lexical semantics associated with the second word. For example, a verb may subcategorize for a preposition (e.g., EN above) and hence eliminate the lexical meanings of the preposition from consideration.
Step 3. Explain-disambiguation DOCTOR-ANDREU:Word: Doctor-Andreu
Sense chosen: Sense1 (HUMAN)
Correct Chosen: Sense2 (ORGANIZATION)
Other Senses: NIL
Details of constraints, scores, paths, etc.
Step 4. Get-sem Doctor-Andreu: to access lex-maps, etc.
It can be seen from the above that the new testing environment provides high-level functions such as explain-binding and explain-disambiguation. Moreover, it eliminates the need for separate tools for accessing the lexicon and the ontology by providing accessor functions (such as get-sem) to relevant information.
Table 1 above shows the latest disambiguation results from Mikrokosmos. These are results from analyzing four real-world texts which had, on an average, 17 sentences with over 21 words per sentence. It can be seen that, on an average, we get 91% correct disambiguation of open class words, counting only ambiguous words. That is, of all the ambiguous open-class words in the texts, Mikrokosmos selects the right sense 91% of the time. If we count all open class words, the percentage correctness goes up to about 97%.
It is obvious from the Table that the performance on the first and third texts (Roche and Matra-Hachette) was worse than the performance on the other two texts. The first and third texts had longer sentences, many more ambiguous words, and constructs that make disambiguation hard (e.g., ambiguous words embedded in appositions). Moreover, just a handful of difficult words led to significantly worse performance in these texts. For example, the word operacion occurred several times in these texts and was hard to disambiguate between its WORK-ACTIVITY, MILITARY-OPERATION, SURGERY, and FINANCIAL-TRANSACTION senses.
It can also be noted from Table 1 that syntactic information contributed to about 38% of word sense disambiguation (18 out of 48, on an average, were disambiguated by syntax; some of these could be incorrect and are included in calculating the total percentage of correct and incorrect disambiguation). Often, syntactic binding eliminates word senses and makes an ambiguous word unambiguous in its syntactic context.
In comparison, the best results before the latest round of testing and clean-up were:
62% correct counting only ambiguous open-class words and 76% overall.
The above results are only for disambiguating open-class words. We are currently working on building hierarchical discrimination trees for prepositions and other closed-class entries in the lexicon in order to get good results in disambiguation closed-class meanings.
total number of open class words: 104
number of words with only one sense: 78
number of words with only one senses after syntax: 87
number ambiguous words after syntax: 17
number ambiguous words correctly disambiguated: 14
number ambiguous words incorrectly disambiguated: 3
i.e., 88.5% correct counting only ambiguous open class words and
97.1% correct counting all open class words. These numbers are almost the same as the numbers for training texts shown earlier in Table 1.
Here is how we tested Mikrokosmos on the new text:
Unknown words (which were mapped to ALL) were treated as unambiguous. There were a total of 19 such words of which 12 appeared to be proper names. Only 3-4 of the 19 words may in fact be ambiguous.
In a test set, it was found that only 174 out of 642 open-class words in the lexicon are ambiguous.
We added 40 new word senses to about 30 of these words. Here are the results:
BEFORE (109 words were ambiguous):
Correctly disambiguated: 91 wordsIncorrectly disambiguated: 18 words
Percent correct: 83.5%
AFTER (139 words were ambiguous):
Correctly disambiguated: 111 wordsIncorrectly disambiguated: 28 words
Percent correct: 79.9%
This translates to about 67% correct on newly acquired ambiguous words and 80% for all ambiguous words.
The above numbers do not include the contribution of syntax. If we had included ambiguities resolved by syntax, the numbers reported above would be significantly higher and quite comparable to the numbers presented in the previous sections for training and new texts.