ISCAP Proceedings: Abstract Presentation
Design and Evaluation of a Human-Centered Explainable AI Dashboard for Cybersecurity Education
Steve Schilhabel
University of Wisconsin Oshkosh
Abstract
While artificial intelligence (AI) is applied more often in cybersecurity training and instruction, the black-box nature of AI systems’ decision-making processes can detract from their value for education and impair the trust of the users. In this paper, we share how we designed, prototyped, and evaluated a Human-Centered Explainable AI (HC-XAI) dashboard as a tool for supporting and empowering cybersecurity learning with a focus on phishing threat identification. The artifact was developed in response to educational challenges from the field, following the principles of Design Science Research (DSR). It incorporates three complementary explanation modalities, including: (1) rule-based logic; (2) natural language explanations, generated using large language models; and (3) visual heatmap visualizations of token-level attention. The HC-XAI dashboard was rigorously evaluated using a two-phase methodology, which included both an expert heuristic walkthrough and a mixed-methods user study with a sample of 23 cybersecurity students. The findings demonstrate the dashboard’s strong usability, positive impact on learner trust, and variation in user preference across the three modalities. The paper’s contributions are fourfold: (1) to the literature, the study shares how to operationalize and evaluate key design choices of HC-XAI; (2) to cybersecurity training practice, the work presents actionable directions for educational program designers and teachers on how to use explainable AI for improving IS and cybersecurity students’ education and learning; (3) to DSR research, the work makes a unique contribution in the context of user training and shows how design artifacts can support not only technical practice but classroom activities as well; and (4) to the research community, the study provides a robust HC-XAI dashboard artifact as a proof-of-concept to inspire and support applied, student-centered research.