ISCAP Proceedings: Abstract Presentation
Assessing Programming In The Age of AI
Katelyn Walls
Elon University
Emily Yttri
Elon University
Abstract
Generative Artificial Intelligence (GenAI) is completely transforming both the ways in which programming is taught and the way in which students’ programming competency is assessed. Popular large language models (LLMs) can solve an introductory programming problem with almost 95% accuracy (Kiesler & Schiffner, 2023). Traditionally, assessment was conducted by running the code and verifying that the output matched the solution. However, this approach no longer allows instructors to distinguish between what was learned and what was generated by GenAI.
Research on GenAI use in programming education is broad, with several avenues exploring overreliance, inaccuracy, academic integrity, and implementation barriers. (Elnaffar et al., 2025). Overall, studies typically report better student outcomes in introductory programming courses when GenAI is incorporated, but these improved outcomes are dependent on the appropriate integration of AI by instructors (Alanazi et al., 2025). Additionally, recent pedagogy research suggests that structural redesigns in assessments are necessary with GenAI (Corbin et al., 2025).
We propose a conceptual framework that restructures assessment around reasoning, judgment, and metacognition. First, the focus of assessment must shift from output-oriented to process-oriented. We assess how students approach and decompose a problem, including the logic behind their approach, the assumptions that they make, and how well they prompt GenAI. We also consider the extent to which students are able to verify, justify, and adapt the GenAI-generated code to a specific context. Then, we need to ensure that students understand what the code does and how it works. This requires transparent documentation showing what the students created themselves and what was contributed by GenAI. By focusing on distinctly human skills, we believe instructors can move toward more effective assessment of programming.
Assessment redesigns built on this framework could be implemented in a range of ways. For example, students could partake in a live building session with an oral defense, where students solve a programming problem with GenAI assistance visible on their computer screen and then defend their code against real-time instructor questioning. Another possible implementation is an AI critique-and-fix task, in which students could identify, fix, and justify changes to GenAI-generated code (Bearman et al., 2024). As a potential reworked capstone, students could solve a real-world business problem and submit their code, their GenAI conversation, and a reflective memo.
GenAI has made coding more accessible, but it has also diminished conceptual understanding and troubleshooting ability. To ensure that students develop programming competencies in the age of GenAI, we must address the human components. By shifting assessments to higher-order skills (Bloom et al., 1956), we believe instructors and students in introductory programming courses can balance GenAI use with deeper comprehension.
References
Alanazi, M., Soh, B., Samra, H., & Li, A. (2025). The Influence of Artificial Intelligence Tools on Learning Outcomes in Computer Programming: A Systematic Review and Meta-Analysis. Computers, 14(5), 185. https://doi.org/10.3390/computers14050185
Bearman, M., Tai, J., Dawson, P., Boud, D., & Ajjawi, R. (2024). Developing evaluative judgement for a time of generative artificial intelligence. Assessment & Evaluation in Higher Education, 49(6), 893–905. https://doi.org/10.1080/02602938.2024.2335321
Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, NY: David McKay Company.
Corbin, T., Dawson, P., & Liu, D. (2025). Talk is cheap: Why structural assessment changes are needed for a time of GenAI. Assessment & Evaluation in Higher Education, 50(7), 1087–1097. https://doi.org/10.1080/02602938.2025.2503964
Elnaffar, S., Rashidi, F., & Abualkishik, A. Z. (2025). Teaching with AI: A systematic review of chatbots, generative tools, and tutoring systems in programming education. arXiv Preprint arXiv:2510.03884.
Kiesler, N., & Schiffner, D. (2023). Large Language Models in Introductory Programming Education: ChatGPT’s Performance and Implications for Assessments (arXiv:2308.08572). arXiv. https://doi.org/10.48550/arXiv.2308.08572