Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
作者： Eric P. Xing , Xiaodan Liang , Zhiting Hu , Christy Y. Li
DOI:
关键词: Natural language processing 、 Artificial intelligence 、 Reinforcement learning 、 Report generation 、 Computer science 、 Bridging (programming) 、 Sentence
摘要: Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level …
arxiv.org参考文章(45)
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio, None, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention international conference on machine learning. ,vol. 3, pp. 2048- 2057 ,(2015)
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan, Show and tell: A neural image caption generator computer vision and pattern recognition. pp. 3156- 3164 ,(2015) , 10.1109/CVPR.2015.7298935
Andrej Karpathy, Li Fei-Fei, Deep visual-semantic alignments for generating image descriptions computer vision and pattern recognition. pp. 3128- 3137 ,(2015) , 10.1109/CVPR.2015.7298932
Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Trevor Darrell, Kate Saenko, Long-term recurrent convolutional networks for visual recognition and description computer vision and pattern recognition. pp. 2625- 2634 ,(2015) , 10.1109/CVPR.2015.7298878
Ramakrishna Vedantam, C. Lawrence Zitnick, Devi Parikh, CIDEr: Consensus-based image description evaluation computer vision and pattern recognition. pp. 4566- 4575 ,(2015) , 10.1109/CVPR.2015.7299087
Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu, BLEU Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02. pp. 311- 318 ,(2001) , 10.3115/1073083.1073135
Yi Hong, Charles E. Kahn, Content analysis of reporting templates and free-text radiology reports Journal of Digital Imaging. ,vol. 26, pp. 843- 849 ,(2013) , 10.1007/S10278-013-9597-4
Ronald J. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning Machine Learning. ,vol. 8, pp. 229- 256 ,(1992) , 10.1007/BF00992696
Alon Lavie, Satanjeev Banerjee, METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments meeting of the association for computational linguistics. pp. 65- 72 ,(2005)
Jan M. L. Bosmans, Joost J. Weyler, Arthur M. De Schepper, Paul M. Parizel, The Radiology Report as Seen by Radiologists and Referring Clinicians: Results of the COVER and ROVER Surveys Radiology. ,vol. 259, pp. 184- 195 ,(2011) , 10.1148/RADIOL.10101045