ISCAP Proceedings: Abstract Presentation
Teaching IS Subjects with an Intelligent Textbook – Effects on Learning and Student Perceptions
Kazuo Nakatani
Florida Gulf Coast University
Yabing Jiang
Florida Gulf Coast University
Abstract
Recent breakthroughs in generative artificial intelligence (GenAI) are transforming industries and driving higher education institutions to explore its meaningful and responsible integration into academic practice. Researchers are shifting from studying the benefits, risks, perceptions, and attitudes associated with GenAI adoption to exploring its practical applications in teaching (Xiao et al., 2023; Yusuf et al., 2024). One recent development is the adoption of fifth-generation intelligent textbooks and features enabled by large language models (Sosnovsky et al., 2025).
Crossley et al. (2025) found that AI-enhanced texts lead to learning gains in a computer science course. Kestin et al. (2025) showed that students are more engaged and motivated and learn significantly more with a GenAI powered AI tutor in a physics course. Along this line of research, we examine the impact of adopting an intelligent coursepack on student learning and perceptions in an undergraduate IS core course, collaborating with Edvisor.ai, a provider of fifth-generation intelligent textbooks. This coursepack provides a controlled AI learning environment, where AI-generated content is confined to the scope specified by instructors, and two AI-supported features called Reading Assistant (RA) and Quiz Recovery (QR) to improve student engagement and learning outcomes. RA and QR function as personalized AI learning companions: RA can summarize, explain, and give examples of any highlighted text; QR lets students recover partial grades when they develop and demonstrate understanding through engaging conversations (edvisor.ai.com). The faculty members teaching the course developed a list of essential terms, concepts, and frameworks aligned with the course learning outcomes. Edvisor.ai then generated the coursepack for Fall 2026 using this list and multiple AI models. The coursepack was verified, revised, and approved by instructors.
Research design and data collection We adopted the Kirkpatrick model (Kirkpatrick, 1996), which measures training effectiveness at four levels (reaction, learning, behavior, and results), to assess the effectiveness of this intelligent coursepack. The Kirkpatrick Model is widely used to evaluate training and learning activities (Alsalamah & Callinan, 2022) and has been adapted to assess educational effectiveness in higher education settings by Praslova (2010). In this study, we will evaluate the impact of the Edvisor coursepack focusing on Level 1 reaction and Level 2 learning. Level 1 measures provide instructors with valuable feedback on students’ satisfaction with and interest in using AI features, as well as insights into areas for improvement. We designed post-course survey questionnaires, adapted from literature, to measure self-efficacy (Pintrich et al.,1991; Artino & McCoach, 2008), motivation (Sun et al., 2019), and engagement (Handlesman et al., 2005; Dixson, 2015). Level 2 learning is measured through pre- and post-tests for each module and the final exam. Activity logs will be collected from the Edvisor platform as additional measures of engagement.
Discussion and implications Result of the study may provide empirical evidence for the effectiveness of fifth-generation intelligent textbook in supporting student learning and enhancing student engagement, motivation, and self-efficacy. Findings from the study may also provide guidance on the design, implementation, and improvement of intelligent textbooks.
Reference
Alsalamah, A., & Callinan, C. (2022). The Kirkpatrick model for training evaluation: Bibliometric analysis after 60 years (1959–2020). Industrial and Commercial Training, 54(1), 36–63. https://doi.org/10.1108/ICT-12-2020-0115.
Artino, A. R., & McCoach, D. B. (2008). Development and initial validation of the Online Learning Value and Self-Efficacy Scale. Journal of Educational Computing Research, 38(3), 279–303. https://doi.org/10.2190/EC.38.3.c
Crossley, S., Choi, J. S., Morris, W., Holmes, L., & Joyner, D. (2025). AI enhanced intelligent texts and learning gains. CEUR Workshop Proceedings, 4010, 17–24. https://ceur-ws.org/Vol-4010/itb25_s1s2.pdf
Dixson, M. D. (2015). Measuring student engagement in the online course: The Online Student Engagement Scale (OSE). Online Learning, 19(4). https://olj.onlinelearningconsortium.org/index.php/olj/article/view/561/165
Handelsman, M. M., Briggs, W. L., Sullivan, N., & Towler, A. (2005). A measure of college student course engagement. The Journal of Educational Research, 98(3), 184–192. https://doi.org/10.3200/JOER.98.3.184-192
Kestin, G., Miller, K., Klales, A., et al. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, Article 17458. https://doi.org/10.1038/s41598-025-97652-6
Kirkpatrick, D. (1996). Great ideas revisited: Revisiting Kirkpatrick’s four-level model. Training & Development, 50, 54–57.
Pintrich, P. R., Smith, D. A. F., Carcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). https://files.eric.ed.gov/fulltext/ED338122.pdf
Praslova, L. (2010). Adaptation of Kirkpatrick’s four-level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215–225. https://doi.org/10.1007/s11092-010-9098-7
Sosnovsky, S., Brusilovsky, P., & Lan, A. (2025). Intelligent textbooks. International Journal of Artificial Intelligence in Education, 35, 967–986. https://doi.org/10.1007/s40593-024-00451-9
Sun, Y., Ni, L., Zhao, Y., Shen, X. L., & Wang, N. (2019). Understanding students’ engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 50(6), 3156–3174. https://doi.org/10.1111/bjet.12724
Xiao, P., Chen, Y., & Bao, W. (2023). Waiting, banning, and embracing: An empirical analysis of adapting policies for generative AI in higher education. arXiv. https://arxiv.org/pdf/2305.18617
Yusuf, A., Pervin, N., Román-González, M., & Noor, N. M. (2024). Generative AI in education and research: A systematic mapping review. Review of Education, 12, Article e3489. https://doi.org/10.1002/rev3.3489