Table 9 shows the latest disambiguation results from Mikrokosmos. These are results from analyzing four real-world texts which had, on an average, 17 sentences with over 21 words per sentence. It can be seen that, on an average, we get 91% correct disambiguation of open class words, counting only ambiguous words. That is, of all the ambiguous open-class words in the texts, Mikrokosmos selects the right sense 91% of the time. If we count all open class words, the percentage correct goes up to about 97%.
It is obvious from the Table that the performance on the first and third texts (Roche and Matra-Hachette) was worse than the performance on the other two texts. The first and third texts had longer sentences, many more ambiguous words, and constructs that make disambiguation hard (e.g., ambiguous words embedded in appositions). Moreover, just a handful of difficult words led to significantly worse performance in these texts. For example, the word operacion occurred several times in these texts and was hard to disambiguate between its WORK-ACTIVITY, MILITARY-OPERATION, SURGERY, and FINANCIAL-TRANSACTION senses.
It can also be noted from Table 1 that syntactic information contributed to about 38% of word sense disambiguation (18 of 48, on an average, were disambiguated by syntax). Often, syntactic binding eliminates word senses and makes an ambiguous word unambiguous in its syntactic context.
One potential flaw with these tests was that the four texts were the same ones used in lexicon and ontology acquisition. This flaw is not as great as it may seem, though. In research such as statistical modeling, when the data is used to actually train the statistical weights in order to obtain the best coverage, testing on the training data does not make any sense. In our work, however, we do not ``tweak'' parameters in order to obtain more favorable results. Revisions occur almost exclusively when some lexicon or ontology entry is found to be incorrect. For instance, in the lexicon, an incorrect or incomplete syntactic specification could be eliminating one or more senses of a word, or a mapping to an incorrect concept might be causing unnecessarily low constraint scores. These are the kinds of errors the Mikrokosmos team focused on when developing the knowledge sources.
In order to confirm that our revision techniques were not unduly affecting the results, we tested the Mikrokosmos analyzer on a new text, previously unseen by us. The results are very promising and quite comparable to the ones above.
total number of open class words: 104
number of words with only one sense: 78
number of words with only one senses after syntax: 87
number ambiguous words after syntax: 17
number ambiguous words correctly disambiguated: 14
number ambiguous words incorrectly disambiguated: 3
i.e., 88.5% correct counting only ambiguous open class words and 97.1% correct counting all open class words. These numbers are almost the same as the numbers for training texts shown earlier in Table 1.
With regards to the Hunter-Gatherer system, these results are presented as evidence that this research can be used profitably in real-life problems. Before HG was implemented, the testing/revision cycle in the project was extremely tedious. Analyzing even small sentences or phrases often took several minutes, and some of the sentences could not be analyzed in whole at all. After HG was implemented, the testing cycle was improved greatly. Not only did the speed increase, but ``what-if'' experiments were then simplified, and to a large extent, made practical. Furthermore, a 97% success rate in disambiguating open-class words can be taken as strong proof of the reliability of the Hunter-Gatherer system.