Evaluating Students' Summaries with GETARUNS

Evaluating summaries is currently performed by the use of statistically-based tools which lack any linguistic knowledge and are unable to produce grammatical and semantic judgements (Landauer et al., 1997). However, summary evaluation needs precise linguistic information with a much finer-grained coverage than what is being offered by currently available statistically based systems. We assume that the starting point of any interesting application in these fields must necessarily be a good syntactic-semantic parser. In this paper we present the system for text understanding called GETARUNS, General Text and Reference Understanding System (Delmonte, 2003). The heart of the system is a rule-based top-down parser, which uses an LFG oriented grammar organization. Lately, a less constrained version of the parser for the application field of text summarization has been developed, which allows the system to recover gracefully from failures. To this end, the parser is couple with another concurrent parsing processes: a partial or shallow parse is always produced and used to recover from complete failures. GETARUNS, has a highly sophisticated linguistically based semantic module which is used to build up the Discourse Model. Semantic processing is strongly modularized and distributed amongst a number of different submodules which take care of Spatio-Temporal Reasoning, Discourse Level Anaphora Resolution. Evaluation taps information from the Discourse Model and uses Predicate Argument Structures (PAS) to detect students’ understanding of the text to summarize. It also uses the output of the Anaphora Resolution Module to check for most relevant topics in the text which the student should have addressed in his/her summary. The system uses a Topics-Stack while processing the text in order to corefer referential expressions: The Topic-Stack Hierarchy gauges nominal heads as either Main, Secondary or Potential Topic. This grading is used as a score that allows the system to detect the most relevant entities in the text at the end of the computation.