干旱区生态环境知识资源中心(Arid): Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease

Arid

DOI	10.1016/j.compmedimag.2023.102303
	Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease
	Gao, Xingyu; Shi, Feng; Shen, Dinggang; Liu, Manhua
通讯作者	Liu, MH
来源期刊	COMPUTERIZED MEDICAL IMAGING AND GRAPHICS
ISSN	0895-6111
EISSN	1879-0771
出版年	2023
卷号	110
英文摘要	Multimodal images such as magnetic resonance imaging (MRI) and positron emission tomography (PET) could provide complementary information about the brain and have been widely investigated for the diagnosis of neurodegenerative disorders such as Alzheimer's disease (AD). However, multimodal brain images are often incomplete in clinical practice. It is still challenging to make use of multimodality for disease diagnosis with missing data. In this paper, we propose a deep learning framework with the multi-level guided generative adversarial network (MLG-GAN) and multimodal transformer (Mul-T) for incomplete image generation and disease classification, respectively. First, MLG-GAN is proposed to generate the missing data, guided by multi-level information from voxels, features, and tasks. In addition to voxel-level supervision and task-level constraint, a feature-level auto-regression branch is proposed to embed the features of target images for an accurate generation. With the complete multimodal images, we propose a Mul-T network for disease diagnosis, which can not only combine the global and local features but also model the latent interactions and correlations from one modality to another with the cross-modal attention mechanism. Comprehensive experiments on three independent datasets (i.e., ADNI-1, ADNI-2, and OASIS-3) show that the proposed method achieves superior performance in the tasks of image generation and disease diagnosis compared to state-of-the-art methods.
英文关键词	Multimodal brain images Generative adversarial network Transformer Image generation Disease diagnosis
类型	Article
语种	英语
收录类别	SCI-E
WOS记录号	WOS:001098257500001
WOS关键词	ESTIMATING CT IMAGE ; CLASSIFICATION ; REPRESENTATION ; ROBUST ; GAN
WOS类目	Engineering, Biomedical ; Radiology, Nuclear Medicine & Medical Imaging
WOS研究方向	Engineering ; Radiology, Nuclear Medicine & Medical Imaging
资源类型	期刊论文
条目标识符	http://119.78.100.177/qdio/handle/2XILL650/395810
推荐引用方式 GB/T 7714	Gao, Xingyu,Shi, Feng,Shen, Dinggang,et al. Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease[J],2023,110.
APA	Gao, Xingyu,Shi, Feng,Shen, Dinggang,&Liu, Manhua.(2023).Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease.COMPUTERIZED MEDICAL IMAGING AND GRAPHICS,110.
MLA	Gao, Xingyu,et al."Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease".COMPUTERIZED MEDICAL IMAGING AND GRAPHICS 110(2023).