Natural Language Processing
Natural Language Processing (NLP) is both a modern computational technology and a method of investigating and evaluating claims about human language itself. Some prefer the term Computational Linguistics in order to capture this latter function, but NLP is a term that links back into the history of Artificial Intelligence (AI), the general study of cognitive function by computational processes, normally with an emphasis on the role of knowledge representations, ie the need for representations of our knowledge of the world in order to understand human language with computers.
Natural Language Processing is the use of computers to process written and spoken language for some practical, useful, purpose: to translate languages, to get information from the web on text data banks so as to answer questions, to carry on conversations with machines (eg to get advice about pensions, etc). These are only examples of major types of NLP, and there is also a huge range of lesser but interesting applications, eg getting a computer to decide if one newspaper story has been rewritten from another.
NLP is not simply applications but the core technical methods and theories that the major tasks above divide up into, such as Machine Learning techniques (ie automating the construction and adaptation of machine dictionaries, modelling human agents' beliefs and desires etc). This last is closer to Artificial Intelligence, and is an essential component of NLP if computers are to engage in realistic conversations: they must, like us, have an internal model of the humans they converse with.
In the Companions project, NLP plays a key role primarily in providing the technology to perform Natural Language Understanding / Analysis and Natural Language Generation. The primary interface of a dialogue agent in a project like Companions uses natural language analysis to process the language at the input side and natural language generation to generate coherent responses at the output side, complemented by ASR (input front-end) and TTS (output back-end).
Natural Language Understanding
In the language understanding area, it is clear that 'deep understanding' is today achieved by rule-based methods only if the task addressed is very limited, while robust performance requires a system to incorporate quantitative techniques, and in particular statistical and machine learning-based techniques to allow for system adaptation.
Several projects have looked at various sides of the problem, including analysis of predicate-argument structure (PropBank), semantic annotation of verbs (FrameNet), and complex formal representation of intertwined language features (such as structure, semantics, and information structure in the Prague Dependency Treebank project).
Parsers exist for all the Companions project languages, with the current most efficient for English and Swedish being the functional dependency-based ones from Connexor. A parser for Czech is described by Collins et al. (1999). Robust semantic processing has been achieved for medium-scale conversational ASR systems, eg in Verbmobil, Siridus, Amities, Comic and Dumas.
In general, the performance of all these parsers (75-92%, depending on the attribute measured) is below what a smooth and natural dialogue system needs, therefore improvement on this front is necessary, along with solving all the integration issues.
Natural Language Generation
In the NL generation area, despite long history of research, most focus has been on language formation (such as the synthesis of a correct word form for inflective languages), or auxiliary word insertion etc. (in systems such as FUF for English).
Morphological generator exists for Czech and Swedish within the consortium, and one can be easily obtained for English. However, little attention has been devoted to the core problem of generation, namely going from a formal representation of meaning to the syntactic (and then surface) form; in the European context, the AGILE project (2000-2002) has been tackling these issues in a limited domain.
Overall performance of text-generation systems is low, and it is difficult to measure: only metrics similar to those of MT (BLEU, NIST) can be applied, but only with caution.
Updated: 07 January 2008 16:39 PM