TransFuse++: A Hybrid CNN-Transformer Architecture with Cross-Attention, Temporal Modeling, and Uncertainty Estimation for Medical Image Segmentation

Masoud Noroozi¹   Sayna Jamaati²   Hamed Aghapanah³   Ali Saeeidi Rad⁴   Mahsa Asadi Anar⁵   Ali Darzi⁶   Mahla Shokouhfar⁷   Helia Sadat Kazemi⁸   Mohammadreza Ghahari⁹   Mohammad Saeed Soleimani Meigoli¹⁰   Jafar Majidpour¹¹   Hossein Arabi¹²   Ali Reza Karimian¹³  

1- گروه مهندسی پزشکی، دانشکده مهندسی، دانشگاه اصفهان 2- دانشگاه صنعتی شریف 3- دانشگاه علوم پزشکی اصفهان 4- گروه مهندسی پزشکی، دانشکده مهندسی، دانشگاه اصفهان 5- دانشگاه علوم پزشکی شهید بهشتی 6- دانشگاه علوم پزشکی شهید بهشتی 7- دانشگاه علوم پزشکی شهید بهشتی 8- دانشگاه علوم پزشکی شهید بهشتی 9- دانشگاه علوم پزشکی تهران 10- دانشگاه علوم پزشکی فسا 11- دانشگاه راپارین 12- دانشگاه ژنو 13- گروه مهندسی پزشکی، دانشکده مهندسی، دانشگاه اصفهان

Hybrid CNN-Transformer Architecture،Medical Image Segmentation،Cross-Attention Mechanism،Temporal Modeling،Uncertainty Estimation،Deep Learning

Similar to any automated image analysis, precise and credible segmentation is indispensable; however, current convolutional or transformer-based networks alone cannot produce the necessary satisfactory spatial resolution, paired with anatomical global thought and volumetric consistency, in medical practice. We introduce TransFuse++, a promising step that offers incremental improvements to the TransFuse architecture, which closely integrates a CNN branch for local texture representation and a Vision Transformer branch for long-term context encoding. The model is further improved with three new modules: a cross-attention mechanism that explicitly aligns the slices, a model with a gated recurrent unit (GRU) that propagates time through stacked sequences, and Monte Carlo dropout that produces pixel-based uncertainty maps. Furthermore, context-dependent cross-skip attention fusion, along with an enhanced BiFusion head, enables more precise boundary delineation without a notable increase in computational cost in the default (no-MC) configuration. When trained in an end-to-end manner on a cardiac X-ray (1,717 frames), a lung CT (6,766 slices) and MR of cirrhotic liver (more than 600 volumes) datasets with an identical 5-fold cross validation protocol, we observe improvements of up to 4.1 percentage points in terms of Dice coefficient and 4.3 percentage points in terms of mIoU, obtaining Dice = 0.972 ± 0.002 (X-ray), 0.989 ± 0.001 (CT) and 0.923 ± 0.061 (MRI). The analysis using Spearman correlation revealed that the per-image Dice coefficients of competing models were either not related to each other (|ρ| ≤ 0.06, Bonferroni-adjusted p > 0.005) or that they introduced complementary specific errors in addition to existing architectures. Ablation experiments validate that cross-attention, temporal modeling, and cross-skipping fusion are key contributors to these incremental improvements. Meanwhile, the uncertainty maps help identify ambiguous areas, making TransFuse++ a promising step toward a confidence-aware segmentation approach for multimodal radiology.