Assessment and Development of
Prior Knowledge for
IS Learning Effectiveness:
Reflections on Practice

Nelly Todorova
Annette Mills
Department of Accountancy, Finance and Information Systems
University of Canterbury
Christchurch, New Zealand
Abstract
Ramsden (1992) suggests that excellence in teaching requires constant attention to how a subject is being understood by students and the ability to use the assessment to change instruction so that it more accurately addresses student's errors and misconceptions. This paper is therefore premised on the need to more accurately understand what students know about a subject (prior knowledge), to enrich their prior knowledge and to use this understanding to inform teaching. Based on an extensive literature review, an iterative learning model is introduced which incorporates this process of enrichment, evaluation and activation of prior knowledge on a particular subject or learning unit. The paper also reports on the adoption of the model in a first year introductory IS course and concludes with teacher reflections on the use of the model and suggestions for further work.
Keywords: prior knowledge, IS education, just-in-time teaching, student learning


1.  INTRODUCTION
Learning is an outcome of student activities which depends on two categories of interdependent factors: the teaching context and student specific factors. While effective teaching facilitates student learning, desired outcomes cannot be achieved without student motivation and participation.
One of the key student factors influencing learning outcomes is the relevant knowledge that a student has about a particular subject or topic prior to a learning event (Ausubel 1968; Biggs 2003; Dochy & Alexander 1995). An assessment of prior knowledge can therefore provide valuable information to determine the appropriate guidance needed by learners. Hence for teaching to be effective, teachers need to be able to determine the learner's level of relevant knowledge including their correct knowledge and misconceptions, and adjust their teaching accordingly.
This paper therefore examines current understanding of prior knowledge and its role in student performance and learning. Based on an analysis of the literature, the paper presents an iterative learning model for assessment, development and management of prior knowledge which was then applied in a first-year IS course. The application of the model is then examined using teacher reflection. The paper concludes with recommendations future research.
2.  PRIOR KNOWLEDGE
Prior knowledge may be defined as the knowledge, skills, or ability that a learner brings to the learning encounter. This includes actual knowledge that is available before the learning event, is declarative or procedural, partly explicit and partly tacit, dynamic in nature, and stored in the knowledge base (Dochy and Alexander 1995). Prior knowledge may also include correct understandings and incorrect understandings (or misconceptions).
Research suggests that the variance in students' prior knowledge is one of the strongest factors influencing educational achievement, understanding of lecture material and the potential for meaningful learning (e.g. Ausubel, 1968; Beckwith, 1991; Hadwin et al, 1999; Yates & Chandler, 1991).  For example, O'Donnell and Dansereau (2000) found in the context of cooperative learning that high prior knowledge learners outperformed low prior knowledge learners. Hmelo et al. (2000) in a study of 4th year medical students designing a clinical trial, also found qualitative differences between high and low prior knowledge groups of learners. The authors concluded that situating learning in authentic activity was especially important for low prior knowledge students.  Ausubel (1968) also suggests that meaningful learning is more likely to take place if the learning task can be related to what the learner already knows; conversely, rote learning is more likely if the learner lacks the relevant prior knowledge needed to make the learning task meaningful.
Laurillard (1993) also argues that “it is impossible for teaching to succeed if it does not address the current forms of student understanding of a subject". However, it is important to note that prior knowledge can have positive and/or negative effects on learning. Existing knowledge can have a cumulative impact on individual development thereby accelerating the learning process. However, existing knowledge can inhibit learning if it contains misconceptions (i.e. faulty beliefs or knowledge based on misinformation). In recognition of this twofold impact of prior knowledge, the learning process should be directed so that it 1) builds on positive and consistent prior knowledge and 2) eliminates or reduces the impact of misconceptions. Ausubel (1968) therefore emphasises the importance of checking on the prior knowledge – what students bring into a course – and using this to inform teaching.
Where the lecturer is unable to accurately ascertain the level of prior knowledge relevant to course content, it becomes difficult for students to successfully participate in exercises that require the application of prior knowledge. Furthermore, since lecturers will often address groups with different levels of prior knowledge this can cause problems in determining the level at which teaching should be targeted (Entwistle, 1998).  In an effort to ensure that students have a common understanding of concepts, the lecturer may find him/herself having to spend valuable lecture time conveying large amounts of information, rather than focusing on higher level goals of analysis and synthesis (Jenkins, 1994).  It is therefore important to not only determine existing knowledge but to also ensure that there is an existing level of shared awareness of required concepts.
One of the key difficulties in applying a learning strategy targeted at the level of a students' prior knowledge is the lack of information regarding these levels. The assessment of prior knowledge is generally a difficult task, and any assessment undertaken is likely to provide only a snapshot or a portion of prior knowledge at a particular point in time (Glaser, 1976). Attempts to investigate prior knowledge include the use of assessment types such as multi-choice (MC) questioning, open/closed tests, recognition tests, and free recall (Dochy and Alexander 1995). Novak et al. (1999) also describes how pre-class "warm-up exercises" consisting of short essay and multi-choice questions are used in physics teaching to help create, elicit, assess and activate relevant prior knowledge.  These warm-up exercises aim to encourage students to prepare regularly for classes and help teachers identify student difficulties and misconceptions in time to adjust their teaching accordingly. Like Ausubel (1968), Novak et al (1999) also conclude that acquiring information about the student's levels of understanding before a class allows faculty to reflect and prepare the lesson appropriately. For such exercises to be effective, it is evident from the literature that elicitation of prior knowledge needs to be carried out on a regular basis (Novak, Patterson et al. 1999; Marrs, Blake et al. 2003)
Finally, Yates and Chandler (1991) argue that possessing knowledge is not equivalent to using this knowledge for achieving cognitive, learning or mnemonic goals. As knowledge is inert, failure to activate prior knowledge in a problem solving environment obstructs the learning process. Novak et al. (1999) describe how interactive lecture sessions are built around student responses to pre-class "warm-up" exercises can be used to engage students in a feedback loop as a part of classroom discussion.  Willoughby et al. (1993) employ an interrogation technique to activate existing prior knowledge and boost learning. Christen and Murphy (1991) argue that challenging the students to call on their prior knowledge transcends the learning process from memorisation to meaningful learning.
3.  THE ITERATIVE LEARNING MODEL
The preceding discussion of the literature on learning and prior knowledge demonstrates that in order to leverage students’ prior knowledge educators need to:
1) build on positive and consistent prior knowledge and
2) eliminate or reduce the impact of misconceptions and 3) assess the level of knowledge to allow staff to reflect and prepare their delivery accordingly. The review of the literature also demonstrates that the development, assessment and activation of prior knowledge needs to be a continuous process.
In line with the above objectives to improve the effectiveness of the teaching process, a four stage evolutionary model of learning is proposed that builds on and utilises students' prior knowledge of relevant material (Figure 1). The model supports the concepts of determination and activation of prior knowledge. In the first two stages of the model, prior knowledge is built and assessed. The third and fourth stages demonstrate how teaching can be adjusted to leverage and activate prior knowledge.



















In the first stage (Enrich), students are provided with recommended readings for the forthcoming lecture. This encourages students to develop a common knowledge base prior to the learning event. This will ensure a minimum level of prior knowledge for all students.
The second stage (Evaluate) assesses students’ understanding of the provided material to determine their prior knowledge before the learning events. If students are given credit for this assessment, this stage also provides an incentive to prepare the lecture material.
In the third stage (Reflect), the lecturer's understanding of students' prior knowledge is used to inform teaching practice in the lecture sessions (Ausubel, 1968). While assessment often provides feedback to students on how they should learn (or have failed to learn), this form of assessment is designed to provide feedback to teachers prior to the lecture session on how to teach. The lecturer is able to tailor the session to appropriately balance content delivery, clarification of misconceptions, and analysis and application of concepts.
The final stage (Activate) of the Iterative Learning Model involves the activation of prior knowledge. During the lecture, students are challenged to recall prior knowledge as they apply it to contextual questions (e.g. problem-solving scenarios) and link new concepts to pre-existing ones. Activation of knowledge allows the students to appreciate "how and when existing mental elements can bear upon new demands" (Yates and Chandler, 1991). The declarative knowledge accumulated through prior reading is transformed into procedural knowledge that is bound by context (e.g. through case-based scenarios). This further enriches student knowledge in the subject area and creates a progressive accumulation of knowledge. Hence, learning becomes an iterative process.
The model presented in this section was applied to a first year introductory university course in Information Systems. The following section reports on the methods and tools of adoption.
4.  THE CASE STUDY
Introduction to Information Systems and Technology (IntroIST) is a semester-length undergraduate first-year course. The course is supported by a prescribed textbook and required readings.
The course has an enrolment of approximately 480 students, with approximately 300 students enrolling in the first semester, and the remainder in the second semester. There are no pre-requisites for entry into this course which means that there is no expectation of any prior knowledge in the area. Although most of the students have used computers and applications such as MS-Word and MS-Excel, most do not have a significant knowledge of IST principles and concepts.
80% of the students taking IntroIST are business majors (e.g. Accounting, Management, Economics).  At least 50% of the student population intend to major in accounting. The student population therefore consists of primarily of non-IS majors with fewer than 20% intending to major or minor in Information Systems (IS) or Computer Science. IntroIST (or an equivalent substitute) is a required course for all IS-majors, IS-minors, and Accounting-majors only; however Accounting-majors are not required to take any other IS-subject. This shows that there is a significant variation in student motivation and their preconceptions of the importance and difficulty of the subject area.
Approximately 60% of the students taking IntroIST will also have taken or be concurrently enrolled in higher-level courses, with 5% being concurrently enrolled in higher-level IS or CS courses. This demonstrates a variance in students’ experience of learning activities and techniques. These characteristics therefore lead to significant differences in the prior knowledge of the student population.
The following discusses, in the context of the above case description, the adoption of the four-stage iterative learning model introduced in this paper.
Stage 1: Enrich
Every week students are given the specific parts of the textbook they need to read and the topics of greater importance for the following week. This offers some guidance to students on how to organize their study prior to the lecture.
The readings normally consist of both theoretical definitions and facts (declarative knowledge) and short cases (contextual knowledge).
Stage 2: Evaluate
In this stage students are required to take a weekly on-line (WebCT-based) assessment based on the provided material for the following week. While the technique of on-line testing is not new, these tests focus not on an assessment of students' understanding of taught material, but on determining the students' prior understanding of forthcoming material and using this to inform teaching.
The assessments are normally available for one week prior to the target teaching week. Students are given 10 minutes in which to complete five (5) multi-choice questions randomly selected from a pool of questions.  To encourage students to do the assignments, they are given credit using the best 10 of their submitted assessments. As a result students are encouraged to participate without fear of failure. 75% of the students on average complete each on-line assignment and 83% of the students completing at least 10 of the assessments over the length of the course. Submissions are normally due 24-hours before the lecture. All submissions are marked on-line and graded on the basis of correctness; however the results and detailed feedback are not released until the assessment period has expired.
Stage 3: Reflect
In this stage the lecturer is provided with feedback on students’ understanding of the provided material. Prior to the lecture session, the summarised assessment results are reviewed by the lecturer to distinguish those topics on which students have performed well or poorly. The distribution of incorrect responses for each question is also reviewed. An analysis of the responses contributes to a better understanding of students' prior knowledge, and their errors and misconceptions. The lecturer is then able to place additional emphasis on the material that students did not grasp as well as correct misconceptions in prior knowledge.
Stage 4: Activate
The final stage (Activate) of the Iterative Learning Model involves the activation of prior knowledge. As students possess a shared base of declarative knowledge, less time is allocated to coverage of facts and definitions and more time is allowed for application and discussion. Thus the focus of lecture is shifted from lower-level learning activities to activities which encourage higher level of learning outcomes such as concept and use as defined by the IS 2002 Curriculum and Guidelines for undergraduate students (Gorgone et al 2003). Students are required to apply their prior knowledge to short scenarios or problem-solving exercises. Where the topic allows the same scenario will be used in different weeks by changing the discussion questions and problem solving exercises to correspond to the new topic. This shared context provides continuity and better demonstrates the relationships between topics.
5.  REFLECTIONS ON PRACTICE
This section reports on our experiences in applying the iterative learning model to teaching in this course.
Stage 1: Enrich
The main purpose of this stage is to build a knowledge base shared by all the students. Since the majority of the students submit each assessment, we have some evidence that the students prepare the recommended material prior to the lectures. In addition, some lecture sessions use opening questions that evaluate prior knowledge and in our experience, a majority of the students are able to respond correctly. However, feedback from students raises a number of concerns regarding this stage. Since the assessments are not invigilated, students can refer to their textbook while answering the questions. Most of the multi-choice questions used for the assessments are based on definitions of concepts. Therefore, a superficial scan of the chapter prior to the assessment will allow the student to note the main definitions without understanding or long term recall. If the new knowledge that the question was trying to create and elicit is consequently lost (or the student reverts to an inadequate prior knowledge base or misconceptions), then the purpose of this stage was not achieved. This failing of the use of the model can be minimized by changing the nature of the questions to encourage deeper approaches to learning.
Stage 2: Evaluate
The objective of this stage is to gain feedback on the level of prior knowledge and to encourage students to prepare the material. The method of assessment of prior knowledge promotes the students’ approach to preparing the material. Our experience confirms findings in the literature that multi-choice (MC) questions promote surface approaches to learning (Scouller, 1998). This was not a concern when the model was implemented using MC questions as literature suggests that even lower level knowledge can be activated during the learning event to transform the declarative knowledge into procedural knowledge (Yates & Chandler, 1991). However, as discussed above, this approach does not ensure even memorization without understanding.
The second objective of this stage is to enable teachers to assess student levels of prior knowledge. Since the majority of students submit the assessments this allows the teachers to gain feedback on student knowledge as a whole. Again, it is a limitation of this assessment method, that it does not ensure that the feedback is a true representation of student prior knowledge.
Stage 3: Reflect
The main objective of this stage is to interpret the feedback on assessment and use this to prepare the lecture material. Since the assessment is quantitative, it is difficult to interpret the results to identify misconceptions. Unless the majority of students choose incorrect answers, it is hard to determine by using MC questions where misconceptions may lie. Even if students have consistently selected the same incorrect answer, it is not always possible to interpret the reasons behind the mistakes. Furthermore, if the students have entered the correct answer by looking it up and have not learnt the concept, the teacher is likely to incorrectly determine that students have understood that concept. Consequently, the topic may not receive the attention required.
Stage 4: Activate
The purpose of this stage is to activate the prior knowledge by setting contextual exercises. In our experience we are able to spend less time during lectures covering factual material and definitions. This allows us to allocate more time to case based activities and problem-solving exercises. However, the lecture provides limited opportunity for knowledge activation due to time constraints and class size.
Finally, it is important to note the additional resources required to support the implementation of this learning model. A substantive investment of time and effort is required to prepare the questions, administer the tests, respond to related student queries, analyse the responses, and amend the lectures within a very limited timeframe.
6.  CONCLUSIONS
The knowledge that students bring to the lecture is one of the most important factors influencing their learning. Since effective teaching is that which makes learning possible it is of great importance that lecturers understand the level of the students' prior knowledge and target their teaching accordingly. This paper has proposed an iterative learning model that aims to improve teaching effectiveness by building and leveraging the prior knowledge of the learner. The model has been adopted in the context of teaching and learning in IS education.  Based on the iterative learning model, this paper describes a technique (through on-line testing) that can help determine and leverage the prior knowledge of students, for informing teaching. Future research could undertake an analysis of the impact of this approach on teaching effectiveness as measured by student understanding, motivation and performance.
7.  REFERENCES
Ausubel, D. P. (1968). Educational Psychology: A Cognitive View. New York, Holt, Reinhart & Winston.
Beckwith, J. B. (1991), "Approaches to learning, their context and relationship to assessment performance", Higher Education, 22, 17-30
Biggs, J. (2003). Teaching for Quality Learning at University. What the Student Does. Berkshire, The Society for Research into Higher Education (SRHE) and Open University Press.
Christen, W and T. Murphy (1991), "Increasing comprehension by activating prior knowledge", ERIC Clearinghouse on Reading and Communication Skills, Bloomington, IN. 
Dochy, F. and P. A. Alexander (1995). "Mapping Prior Knowledge - a Framework for Discussion among Researchers." European Journal of Psychology of Education 10(3): 225-242.
Entwistle N. (1998).  Improving Teaching Through Research on Student Learning.  In Forest J.J.F. (ed.) University Teaching: International Perspectives, Garland Publishing: London, 73-112.
Glaser, R. (1976). Components of a psychology of instruction: towards a science of design, Review of Educational Research, vol.46(1), 1-24.
Gorgone, J. T., G. B. Davis, et al. (2003). IS 2002: Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems. The DATA BASE for Advances in Information Systems 34(1), 1-52.
Hadwin, A., Kirby, J. and R. Woodhouse (1999), "Individual Differences in Notetaking, Summarization, and Learning from Lectures", The Alberta Journal of Educational Research, 35, 1-17.
Hmelo, C. E., A. Nagarajan, et al. (2000). "Effects of high and low prior knowledge on construction of a joint problem space." The Journal of Experimental Education 69(1): 36-46.
Jenkins, A. (1994) "Active Learning in Structured Lectures" in G. Gibbs and A. Jenkins (eds.) Teaching Large Classes in Higher Education, Kogan Page, London.
Laurillard, D. (1993).  Rethinking University Teaching: A Framework for the Effective Use of Educational Technology.  Routledge:London.
Marrs, K. A., R. E. Blake, et al. (2003). "Web-based warm up exercises in just-in-time teaching." Journal of College Science Teaching 33(1): 42-47.
Novak, G., M, E. T. Patterson, et al. (1999). Just-in-Tine Teaching: Blending Active Learning with Web Technology. New Jersey, Prentice-Hall.
O'Donnell, A. M. and D. F. Dansereau (2000). "Interactive effects of prior knowledge and material format on cooperative teaching." The Journal of Experimental Education 68(2): 101.
Palomba, C. A. and T. W. Banta (1999). Assessment Essentials.  Planning Implementing and Improving Assessment in Higher Education. San Francisco, Jossey-Bass.
Ramsden, Paul (1992).  Learning to Teach in Higher Education, Routledge:London.
Scouller, K. (1998), "The Influence of Assessment Method on Students' Learning Approaches: Multiple Choice Question Examination Versus Assignment Essay," Higher Education, Vol. 34, pp.105-122.
Willoughby, T., Waller, G, Wood, E and G. MacKinnon (1993), "The Effect of Prior Knowledge on an Immediate and Delayed Associative Learning Task Following Elaborative Interrogation", Contemporary Educational Psychology, 18, 36-46.
Yates, G. and M. Chandler (1991), "The Cognitive Psychology of Knowledge: Basic Research Findings and Educational Implications", Australian Journal of Education, 35, 131-153.
??

??

??

??