Where academic tradition
meets the exciting future

You are here: TUCS > PUBLICATIONS > Publication Search > On Evaluation of Automatically...

On Evaluation of Automatically Generated Clinical Discharge Summaries

Hans Moen, Juho Heimonen, Laura-Maria Murtola, Antti Airola, Tapio Pahikkala, Virpi Terävä, Riitta Danielsson-Ojala, Tapio Salakoski, Sanna Salanterä, On Evaluation of Automatically Generated Clinical Discharge Summaries. In: Ellen A. A. Jaatun, Elizabeth Brooks, Kirsti Berntsen, Heidi Gilstad, Martin Gilje Jaatun (Eds.), Proceedings of the 2nd European Workshop on Practical Aspects of Health Informatics (PAHI 2014), Ceur Workshop Proceedings 1251, 101–114, WISU Verlag Aachen, 2014.

Abstract:

Proper evaluation is crucial for developing high-quality computerized text summarization systems. In the clinical domain, the specialized information needs of the clinicians complicates the task of evaluating automatically produced clinical text summaries. In this paper we present and compare the results from both manual and automatic evaluation of computer-generated summaries. These are composed of sentence extracts from the free text in clinical daily notes - corresponding to individual care episodes, written by physicians concerning patient care. The purpose of this study is primarily to find out if there is a correlation between the conducted automatic evaluation and the manual evaluation. We analyze which of the automatic evaluation metrics correlates the most with the scores from the manual evaluation. The manual evaluation is performed by domain experts who follow an evaluation tool that we developed as a part of this study. As a result, we hope to get some insight into the reliability of the selected approach to automatic evaluation. Ultimately this study can help us in assessing the reliability of this evaluation approach, so that we can further develop the underlying summarization system. The evaluation results seem promising in that the ranking order of the various summarization methods, ranked by all the automatic evaluation metrics, correspond well with that of the manual evaluation. These preliminary results also indicate that the utilized automatic evaluation setup can be used as an automated and reliable way to rank clinical summarization methods internally in terms of their performance.

BibTeX entry:

@INPROCEEDINGS{inpMoHeMuAiPaTeDaSaSa14a,
  title = {On Evaluation of Automatically Generated Clinical Discharge Summaries},
  booktitle = {Proceedings of the 2nd European Workshop on Practical Aspects of Health Informatics (PAHI 2014)},
  author = {Moen, Hans and Heimonen, Juho and Murtola, Laura-Maria and Airola, Antti and Pahikkala, Tapio and Terävä, Virpi and Danielsson-Ojala, Riitta and Salakoski, Tapio and Salanterä, Sanna},
  volume = {1251},
  series = {Ceur Workshop Proceedings},
  editor = {Jaatun, Ellen A. A. and Brooks, Elizabeth and Berntsen, Kirsti and Gilstad, Heidi and Jaatun, Martin Gilje},
  publisher = {WISU Verlag Aachen},
  pages = {101–114},
  year = {2014},
  keywords = {Summarization Evaluation, Text Summarization, Clinical Text Processing, NLP},
  ISSN = {1613-0073},
}

Belongs to TUCS Research Unit(s): Turku BioNLP Group

Publication Forum rating of this publication: level 1

Edit publication