Home » Academic
Academic
PhD thesis (defended in March 2010)
The Structure of Real User-System Dialogues in Interactive Question Answering
When users engage in (typed) conversations with an Interactive Question Answering (IQA) system, user questions are typically not asked in isolation. The questions’ context, i.e., the preceding interactions, should be useful for understanding Follow-Up Questions (FU Qs) and helping the system pinpoint the correct answer. In this work, we study how much context, and what elements of it, should be considered to answer FU Qs. We harness Logistic Regression Models (LRMs), both for learning which aspects of dialogue structure are relevant to answering FU Qs, and for comparing the accuracy with which the resulting IQA systems can correctly answer these questions. Unlike much of the related research in IQA, which uses artificial collections of user questions, our work is based on real user-system dialogues we collected via a chatbot-inspired help-desk IQA system we deployed on the web site of our University library.
Our statistical modeling experiments integrate a wide array of shallow and deep features, each of which describing a specific relation that holds between two utterances (i.e., user questions or system answers). These relations are based on lexical similarity, as proposed in the Question Answering literature for mapping questions to their correct answers, and different theories of discourse and dialogue coherence, respectively. The experimental results demonstrate which of the proposed features hold up against empirical evidence from realistic IQA dialogue data. In a nutshell, the best LRMs for describing IQA dialogue structure combine shallow and deep utterance-utterance relations; also, the best models distinguish different FU Q types, where we show that this classification can be done implicitly and automatically using the same set of shallow and deep features we use for mapping FU Qs to their correct answer.
The implications of this work are two-fold. For the dialogue and discourse research community, concerned with theories of text coherence, we provide clues as to which automatically implementable theories of inter-utterance coherence hold up empirically in realistic IQA dialogues. On the other hand, the IQA research community could benefit from our results for learning how to automatically distinguish different types of FU Qs, and how to formulate answer pinpointing strategies for each particular FU Q type. More specifically, our work is a practical study of how a real IQA system can tackle the problem of context fusion, and as a result, improve the accuracy of selecting the correct answer to FU Qs.
Please visit the BoB dialogue corpus web-site, providing more information on the dialogue data that was used as a basis for this research.
Publications
Raffaella Bernardi, Manuel Kirschner and Zorana Ratkovic. Context Fusion: The Role of Discourse Structure and Centering Theory. In Proc. of the Seventh conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta. 2010.
pdf
Raffaella Bernardi and Manuel Kirschner. From artificial questions to real user interaction logs: Real challenges for Interactive Question Answering systems. In Proc. of Workshop on Web Logs and Question Answering (WLQA’10), Valletta, Malta. 2010.
pdf
Manuel Kirschner, Raffaella Bernardi, Marco Baroni and Le Thanh Dinh. Analyzing Interactive QA Dialogues using Logistic Regression Models. In Proc. of XIth International Conference of the Italian Association for Artificial Intelligence (AI*IA’09), Reggio Emilia, Italy. 2009.
pdf
Manuel Kirschner and Raffaella Bernardi. Exploring Topic Continuation Follow-up Questions using Machine Learning. In Proc. of NAACL HLT 2009: Student Research Workshop, Boulder, CO. 2009.
pdf
Raffaella Bernardi and Manuel Kirschner. Context Modeling for IQA: The Role of Tasks and Entities. In Proc. of Workshop for Knowledge and Reasoning for Answering Questions (KRAQ’08), Manchester, UK. 2008.
pdf
Manuel Kirschner and Raffaella Bernardi. Context Modeling for IQA: The Role of Tasks and Entities. Poster presented at LCT student session in Bolzano, 15. May 2008 (unpublished).
pdf
Manuel Kirschner and Raffaella Bernardi. An Empirical View on IQA Follow-up Questions. In Proc. of the 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium. 2007.
pdf
Manuel Kirschner. Applying a Focus Tree Model of Dialogue Context
to Interactive Question Answering. In Proc. of the ESSLLI’07 Student Session, Dublin, Ireland. 2007.
pdf
Manuel Kirschner. The BoB IQA system: A Domain Expert’s Perspective. In Proc. of the 11th Workshop on the Semantics and Pragmatics of Dialogue (SemDial’07), Rovereto, Italy. 2007.
pdf
Manuel Kirschner. Building a Multi-lingual Interactive Question-Answering System for the Library Domain. In Proc. of the 10th Workshop on the Semantics and Pragmatics of Dialogue (Brandial’06), Potsdam, Germany. 2006.
pdf
Harald Hüning, Manuel Kirschner, Fritz Class, André Berton, and Udo Haiber. Embedding Grammars into Statistical Language Models. In Proc. of Interspeech’05, pages 1313–1316, Lisbon, Portugal. 2005.
Additional Conferences, Talks, Summer Schools
- 29th Student Conference of Linguistics (StuTS), 2001 (Saarbrücken)
- 11th Student Conference of Computational Linguistics (TaCoS), 2002 (Potsdam) (member of local organizing committee)
- EACL 2006 (Trento)
- ESSLLI 2006 (Málaga)
- ESSLLI 2007 (Dublin)
- Potsdam Fall School in Computational Linguistics 2007 (organized by the “Deutsche Gesellschaft für Sprachwissenschaft”)
- Invited talk at CIMeC PhD – CLIC Research Seminar (Rovereto), 28. Feb. 2008
- ESSLLI 2008 (Hamburg): Co-chair (Language & Computation) for the ESSLLI Student Session (StuS)