arXiv:2611.13564v1 Announce Type: cross Abstract: Objective: Concept-based interpretation has emerged as a promising approach for explaining blackbox models using human-understandable, high-level concepts. However, most existing methods require annotated concept labels to train the model to identify concepts in the input, and obtaining such annotations in clinical text is often expensive. This study aims to enable concept-based interpretation for language models without concept supervision on clinical prediction tasks. Methods: We propose a novel framework that enables concept-based interpretation without concept supervision through a tailored cross-attention bottleneck. Clinical text and the text descriptions of medical concepts are first tokenized and embedded by a pretrained language model. A cross-attention module then estimates the presence of each concept for each text token, producing concept relevance scores that capture how concepts are involved throughout the note. These scores are pooled and used in a linear prediction layer to model clinical outcomes in an interpretable manner. Results: The performance of the proposed framework was compared against blackbox models across two tasks: chest trauma severity stratification using a private dataset and an in-hospital mortality prediction task using the publicly available MIMIC-III dataset. The proposed framework achieved AUROC values within 0.02 of the corresponding blackbox models. The framework also enabled cohort- and patient-level interpretation, capturing the overall concept importance across the cohort and patient-specific concept contributions. Conclusion: Our proposed framework realizes concept-based interpretation without requiring ground-truth concept presence labels. It improves the transparency of clinical text predictions while maintaining strong performance in clinical prediction tasks.