A Survey of Assessment Mechanisms for Continuous Process Improvement of IT Curriculums William Todd Smoak and Kevin C. Dittman Dept. of Computer Information Systems and Technology, Purdue University West Lafayette, IN, 47907-1421, USA Abstract A repeating trend within the Information Technology (IT) community is that every year a new technology approach or technique comes to the forefront. That may be a new programming language or a new method for developing information systems. At any rate, it is becoming increasingly more difficult to stay abreast of these technologies and approaches as they emerge in the workplace. An even more difficult challenge is to find and retain qualified people to teach these new state-of-the-art concepts and technologies. The foundation skills required for IT professionals have and will probably be the same for many years. These include skills such as math, problem solving, logic and interpersonal communications, but within IT there is also another skill set that is more dynamic. These are the demanded technology skills that seem to change every few years. Because of this phenomenon, it is important that the topics being taught by IT instructors be flexible and adaptable to current as well as future needs. Institutions of Higher Learning need to consistently re-evaluate their curriculum and assess whether the material being taught is in the best interest of the students as well as the IT industry. In other words they should resist preparing students to use specific technologies and instead prepare them to use any technology, which may assist the students in solving IT related problems. Institutions must be careful not to be swayed by the market demands of providing "training", rather prepare the students for a "lifetime of learning" and continuous career growth through education. This paper outlines in detail possible approaches and methods for tracking course relevancy information. Keywords: Curriculum quality assessment, evaluation techniques, continuous improvement, course content 1. ACCESSING IT CURRICULUM CONTENT AND QUALITY: IS OUR STUDENTS PREPARED TO SATISFY THE NEEDS OF THE INDUSTRY? In our continuing effort to be as successful as possible with respect to our universities and curriculum, we must look to improving the quality of what we teach in every way possible. By increasing the quality of our degree programs, we increase the potential value of our students and their chances to succeed in the workplace. A common theme of many quality models used within the IT industry is that in order to achieve the highest level of quality, a Continuous Process Improvement (CPI) philosophy needs to be implemented. Stating a vision, establishing objectives, reviewing outcomes, making assessments, and implementing improvement changes should be part of a continuous process improvement activity for any IT curriculum. Because the IT industry is changing and developing at an alarming rate and we must prepare our students for adapting to those changes. If our teaching forces "Tunnel Vision," our students may not be able to function in this environment of change, therefore placing them at a severe disadvantage in relation to the other professionals in the industry. 2. WHY CURRICULUM REVIEWS? An annual review of the curriculum must be performed for the following reasons: 1) Ensure that students are receiving a "fair" return on their investment. 2) Ensure the quality and content of the programs meets or exceeds the programs of other comparable institutions in order to not only attract high-quality prospective students and faculty but for retention purposes as well. 3) Ensure the program is striving to meet the current and future needs of industry, which not only hires the graduates of the program but provide opportunities for grants, gifts and partnerships. 3. ASSESSMENT MECHANISM FOR EVALUATING AND IMPROVING COURSE QUALITY AND CONTENT Outlined below are several different techniques and tools that can be used for assessing IT curriculum. No evaluation mechanism by itself or even used jointly can guarantee course quality, but the more evaluation mechanisms used, the more likely course quality will continue to improve. With any evaluation mechanism, they must be used on a reoccurring basis so baselines can be established and comparisons can be made to make improvements when necessary. The techniques and tools listed below are in rank order based on applicability and useful return. Bloom's Taxonomy (Bloom 1956) Bloom established a hierarchy of educational objectives, which is generally referred to as Bloom's Taxonomy, and which attempts to divide cognitive objectives into subdivisions ranging from the simplest behavior to the most complex. In order to evaluate an IT curriculum it must first be built based on skill areas and those areas can then be evaluated for quality. Bloom's Taxonomy helps to categorize and focus questions on the skills, as well as depth or those skills that should be taught within any curriculum. Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation are all part of Bloom's model. * Knowledge is defined as the remembering of previously learned material. This may involve the recall of a wide range of material, from specific facts to complete theories. * Comprehension is defined as the ability to grasp the meaning of material. This may be shown by translating material from one form to another (words to numbers), by interpreting material (explaining or summarizing), and by estimating future trends (predicting consequences or effects). * Application refers to the ability to use learned material in new and concrete situations. This may include the application of such things as rules, methods, concepts, principles, laws, and theories. * Analysis refers to the ability to break down material into its component parts so that its organizational structure may be understood. This may include the identification of parts, analysis of the relationship between parts, and recognition of the organizational principles involved. * Synthesis refers to the ability to put parts together to form a new whole. This may involve the production of a unique communication (theme or speech), a plan of operations (research proposal), or a set of abstract relations (scheme for classifying information). Learning outcomes in this area stress creative behaviors, with major emphasis on the formulation of new patterns or structure. * Evaluation is concerned with the ability to judge the value of material (statement, novel, poem, research report) for a given purpose. The judgments are to be based on definite criteria. These may be internal criteria (organization) or external criteria (relevance to the purpose) and the student may determine the criteria or be given them. (Carneson 1998) Pros Cons Bloom's Taxonomy is very detailed and thorough and can apply to all courses at every rank. Bloom's Taxonomy is generalized to creating and maintaining any curriculum and is in no way specific to IT. Using Bloom's Taxonomy to create an evaluation plan and then implement could take 6-12 months depending on resource availability. IS'97 Curriculum Model (ACM 1997) The IS'97 Model Curriculum and Guidelines was jointly developed by top IT associations. Adapted over the last 20 years the IS '97 curriculum encompasses two broad areas: (1) acquisition, deployment, and management of IT resources and services (the information systems function) and (2) development and evolution of technology infrastructures and systems for use in organization processes (system development). The Model Curriculum provides guidelines, a set of courses, course materials, curriculum design objectives, and knowledge elements. The model is based on common structures and degree programs, but grounded in a fundamental body of computing and information systems knowledge. Courses are laid out in three areas: General, design, and Development. Specific areas to be reviewed for quality laid out in the curriculum are Communication, Computer Application Systems, Information Technology and Tools, Interpersonal Relationships, Management, Problem Solving, Systems Development Methodologies, Systems Theory and Concepts, and Professionalism. By reviewing this curriculum model, abilities and knowledge can be tested and compared with the demands of various jobs or industries and allow education institutions to better focus on these skills. Ideally this model would be used as a basis for developing a new program, but it can also be applied to already existing programs that are looking for quality improvement. A revision (IS'00) is currently in development and should be a better model to follow in the near future. Pros Cons The IS'97 Curriculum goes into great detail to guide you in setting up an ideal curriculum. To review and change or to start from scratch would be quite a time and resource consuming process. Industrial Review Boards (IRBs) Industrial Review Boards bring together representatives from top companies of the IT industry and faculty to review the curriculum for future relevance. The more companies that are invited to interact and develop curriculum the more likely they are to hire students upon graduation. Pros Cons Program receives insight to industry trends and plans and can adjust curriculum accordingly. Company management may try to steer program to address short term need. Companies who participate are more likely to hire the program's graduates. Company management doesn't always know or want to admit to company problems and may place blame on curriculum. Program can take advantage of company management planning skills for strategic and tactical planning. Costs may be incurred for the travel, lodging, and meals of company representatives. Companies who participate are more likely to invest (gifts, grants, endowments, etc.) in the program. Alumni Review Boards Alumni Review Boards bring together the top graduates of the program to review the curriculum with faculty. Alumni offer interesting insights and can interact and develop curriculum that is more relevant or would have been more relevant. They often have a vested interest to see the program succeed and could be a potential donor. Pros Cons Alumni have the previous experience of knowing curriculum and how it impacted them. Alumni may be biased about devaluing their education if problems are found or may want to steer it in wrong direction. Alumni recently out of school will be able to tell you what they remember most and how they are using it. It may take a few years for alumni to realize the benefits and missed opportunities in their education. Costs may be incurred for the travel, lodging, and meals of Alumni representatives. Pre/Post Student Testing This evaluation technique allows you to test students before and after they take a course for knowledge. This helps you assess what they learned from the course. You can also use this along with course evaluations to determine if the course met the student's expectations. Pros Cons Can be good measure of knowledge growth. Students have different reasons for taking the same course, so course objectives need to be clear. Some students may not learn appropriate material based on teaching style of the instructor. Post Course Evaluations Post Course Evaluations offer the students the opportunity to evaluate course quality and whether it met their expectations. Results of evaluations can offer insight to course as well as instructor improvements. Pros Cons Most students will provide constructive feedback provided they are given the appropriate amount of time to do so. Encourage criticisms as well as compliments. Depending on whether a course is required versus an elective may have an impact on student answers and their motivation for being in the course. Can be easily tabulated and scored if an OCR form is used. Students are easily biased by professor and their grade without fully evaluating what they have learned. Students may not want to take the time to thoroughly complete the evaluation and may mark any response to just complete it. Inappropriate feedback may be received from less than mature or poor performing students. Placement Rates Job placement statistics are an indication of market demand and curriculum quality. Pros Cons High placement rates are a good marketing tool for students and prospective employers. Rates may reflect market demands and state of the economy rather than quality of the curriculum. High placement rates can be useful for soliciting additional budget or other resources from institution. Placement research takes time and resources. Placement research assists in tracking graduates for future fund-raising efforts. Salary Rates Starting salaries surveys can determine if you are attracting or producing the best and the brightest graduates. If your students' starting salaries are higher than those of other similar institutions or higher than available salary surveys it may be a reflection of the quality of the program and curriculum. Pros Cons High salary rates are a good marketing tool for students and prospective employers. Salary rates may reflect market demands and state of the economy rather than quality of the curriculum. High placement rates can be useful for soliciting additional budget for faculty salary adjustments. High student salary rates may cause concern among lower paid faculty and lead to retention problems. Salary research assists in tracking graduates for future fund-raising efforts. Must reply on student's honesty to report accurate starting salary. Starting Salary can be biased based on school recognition and not quality of programs. Salary research takes time and resources. Employee Rankings Path By tracking an employee's progress in industry you can determine if your program is producing the types of employees that you desire. Pros Cons Research assists in tracking graduates for future fund-raising efforts. Research takes time and resources. Successful graduates can be a source of guest lecturers and recruiters. Graduate success may be attributed to their individual skills and "luck" and not a reflection of the quality of the program. Extremely successful graduates can be used as a marketing tool for attracting potential students and bring notoriety to the institution. School Rankings The top School Ranking guides often look at all areas of the school, the specific programs, applicability, experience, industry ratings etc. The college guides and U.S. News rankings provide a variety of indicators of institutional quality, but nothing that directly measures what may be the most important: what students actually learn. Pros Cons Little or no effort required of institution. The rankings can be biased and look more at larger institutions. Positive rankings can be a great marketing and recruiting tool. Sources aren't always from a random group. 4. CONCLUSION Many assessment mechanisms were presented previously and not one stands out as clearly the best tool. By combining the use many of these tools, and acting on the results, institutions can accurately assess and adapt their curriculums to a higher level of quality. 5. REFERENCES Bloom, Benjamin, 1956, Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. Longmans Green, New York. Association for Computer Machinery (ACM) and Association for Information Systems (AIS) and Association of Information Technology Professionals (AITP), 1997, IS'97 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems. Carneson, J. and G. Delpierre and K. Masters, 1998, "Designing and Managing Multiple Choice Questions." Lee, D. and E. Trauth and D. Farwell, 1995, "Critical Skills and Knowledge Requirements of Professionals: A Joint Academic/Industry Investigation." MIS Quarterly, 19(3).