The Effect of Technology Integration on Critical Thinking Skills in a Graduate Introductory Information Systems Course Jean F. Coppola1 Information Technology, Pace University Briarcliff, NY 10510, USA and Barbara A. Thomas2 Lienhard School of Nursing, Pace University, Pleasantville, NY 10570, USA Jennifer D.E. Thomas3 School of Computer Science and Information Systems, Pace University, New York, NY 10038, USA Abstract This investigation was to determine whether, and to what extent, various hardware technologies (specially designed electronic classrooms) and software (Blackboard(, Healthlite(, Ginormous() support the acquisition of critical thinking skills. One instructor taught three different sections of the same graduate introductory information systems course during a single 14-week semester in this study. The preliminary results obtained from a validated critical thinking tool, the California Critical Thinking Skills Test( (CCTST), indicate that technology integration had a positive effect on students' acquisition of these skills. There were noted differences, however, on other higher-order learning skills, problem-solving, research skills, and creative idea generation. Keywords: Technology integration, critical thinking, learning skills, higher-order learning, problem solving, smart e-classroom, electronic classroom 1. INTRODUCTION Educators and industry agree that it is important for students, especially graduate students, to foster critical thinking skills through their program of study (Cook, et. al., 1996; Facione, P.A., Facione, N.C., Blohm, M.A., Howard, K., & Giancarlo, C.A., 1998; Facione, P.A., Facione, N.C., & Giancarlo, C.A, 1998). The South Carolina Higher Education Assessment Network Critical Thinking Task Force has provided a definition of critical thinking adapted from a number of authors. Critical thinking is defined as, "a reflective, systematic, rational, and skeptical use of cognitive representations, processes, and strategies to make decisions about beliefs, problems, and/or courses of action" (Cook, et. al., 1996; Facione, 1998). In the California Critical Thinking Skills Test( (CCTST), critical thinking skills are considered to include several important components, identified as Analysis, Inference, Explanation, Evaluation, Interpretation, and Self-Regulation (Facione, et. al., 1998). A recent study by Stoney and Oliver (1999) found that utilizing a well designed interactive multimedia software application, which focuses on the acquisition of higher-order thinking and problem-solving skills, fostered and developed cognitive engagement. Stoney and Oliver state that less emphasis should be placed on the assimilation of a large body of isolated facts and more on attracting and holding the students' attention. Furthermore, results of Stoney and Oliver's study were that basic skills were not learned in isolation, but while completing real world tasks that integrated a variety of skills relating to previously learned knowledge. There are currently few research studies that clearly indicate that technology contributes to critical thinking. A study by Duncan (2000) found that students, in sections where computer technology was integrated, scored the same or marginally lower on proficiency tests and homework, as compared to students in traditional sections. The classroom lab in Duncan's study was equipped with PC's, a projection screen, a whiteboard and printers. Findings from a study conducted by Herrington and Oliver (1999) suggest that "a multimedia program based on a situated learning approach can provide a learning environment capable of supporting and maintaining substantial levels of higher-order thinking." The computer in Herrington and Oliver's study, however, essentially served as a demonstration tool with minimal student interaction. For technology to have an impact on vital skills such as critical thinking, technology integration must go beyond simply using computers as a secondary tool, such as a projection or demonstration device, but as the primary instrument (Stoney & Oliver, 1999). According to Girod and Cavanaugh (2001), "Teachers must first begin to define contexts for learning differently and then treat technology resources as serious contexts for stimulating learning". This paper presents preliminary results of a pilot investigation, which examined the effects of two types of technology integration on the development of critical thinking skills in a graduate introductory information systems course. The purpose of the study, methodology, results, discussion, and conclusion are presented, as well as future work. 2. PURPOSE OF STUDY This study examined whether different technology integrations supported critical thinking, and if so, which type offered better support to students during the course of the semester. Two technology integrations were examined: 1) a smart electronic classroom, which used a broadcast-on-demand system, integrated with other various technologies, 2) an on-line class management tool (Blackboardt(). In addition to these technologies, case analysis tools, Healthlite( and Ginormous(, were also used throughout the course in all sections in the study. Research question: Will there be a significant difference in the increase of higher-order learning skills, and critical thinking, with students using a smart electronic classroomt vs. students using Blackboardt? 3. DEFINITION OF TERMS Blackboard( - Blackboard is a course management software tool that is server based and can be accessed via the World Wide Web. This course management tool includes features such as announcements, assessments, question pools, a student web site area, threaded discussions groups, an entire class email list, and course documents. This software also provides student access to an online chat room, and course documents. Broadcast-on-demand - A broadcast-on demand system is a network which operates over coax cable to desktop computers. The coax cable is connected from the teacher station of an electronic classroom to every student station. The system allows video, audio, keyboard, and mouse data signals to be controlled and directed by the teacher. Therefore, this allows a student or the teacher to type or use the mouse of any computer in the room from their own individual station. The most common feature is the ability to project video signals from one station in the room to any station or stations with the same signal quality as the original. CCTST - California Critical Thinking Skills Test is an exam that was purposely intended to gauge the skills dimension of critical thinking (Facione, Facione, & Blohm, et. al., 1998). The CCTST consists of 34 multiple-choice items. The test comes in two forms labeled as Form A and Form B which are intended to be equivalent but have questions and answer that vary slightly. Critical Thinking - Critical thinking is both a skill and a habit of mind (Facione, Facione, & Blohm, et. al., 1998). Facione et al. also refers to critical thinking as the practice of purposeful, self-regulatory judgement, which allots reasoned consideration to evidence, context, conceptualizations, methods, and criteria. Ginormous( - Ginormous is an information systems case study software tool. Healthlite( - Healthlite is an information systems case study software tool. Robotel( - Robotel is specific brand of a high quality coax cable based network, which links the teacher station of a computer classroom to all students in the room. The system includes a master teacher control interface that can at a touch of a button direct any computer screen in the room to any or all of the other computer screens. The teacher and students can share electronic documents, websites, spreadsheets, etc. Robotel includes additional interactive features such as surveying, quizzing, teamwork, scanning all monitors, and a game-like test. 4. METHODOLOGY The electronic classroom used was a specially designed facility that included two networks and access to a course management system. The electronic classroom included customized curved recessed furniture, a hardware-based broadcast-on-demand system, (e.g., Robotelt,), access to the Internet, an electronic chalkboard, multimedia capabilities for viewing and creating presentations, and student interactive and teamwork interfaces. The software that facilitated these functions was unobtrusive to the students. The electronic classroom included other functionalities such as VHS/Laserdisc, instant quizzing, and room scanning. Subjects: The population for this study consisted of first year MBA students taking a required introductory course on Information Systems Principles. Students registered randomly for one of the three possible sections. One section of the course employed face-to-face lectures and activities only. Another section used face-to-face lectures and activities plus Blackboardt. A third section had face-to-face lectures plus utilized electronic classroom technologiest. The same faculty member taught all three sections. All sections used a case analysis software package called Healthlite. Healthlite was employed in each class session throughout the term, in order to master the course material. Weekly analyses were used to build a student portfolio, and the knowledge gained supported answers to specific questions related to the case at the end of the process. The end of term "exam" for all sections was another case analysis, Ginormous. Ginormous required students to work in teams choosing from among three possible solutions to the case and to debate their respective solutions with other teams. "Proponents of collaborative learning claim that the active exchange of ideas within small groups not only increase interest among the participates but also promotes critical thinking", according to Gokhole (1995). Grades for the Healthlite and Ginormous cases were collected along with a survey of students' perceptions of the support to higher-order learning provided by technologies used. Healthlite and Ginormous grades furnished a good indication students acquisition of higher-order learning skills, one of which was critical thinking. Other higher-order learning skills tested included problem-solving skills, research skills, and creative idea generation skills. The California Critical Thinking Skills Tests (CCTST) was administered to the students along with pre- and post- satisfaction surveys at the beginning and end of a 14-week course. The CCTST instrument was chosen over others because of its validity and feasibility for measuring critical thinking and interdisciplinary functionality, as assessed by the South Carolina Higher Education Assessment Network Critical Thinking Task Force (Cook, et. al., 1996). The evaluators rated it as having moderately high validity and high feasibility. The American College Testing Instrument (Assessment of Reasoning and Communicating), the only instrument rated higher than the CCTST in validity, was rated low in feasibility. (Cook, et. al., 1996). 5. RESULTS Most students were in the 20-29-year age category, in all sections. Complete demographic data on students are presented in Table 1. (Note, the total number of students for age, gender, and previous computer experience varies slightly because information was voluntarily requested.) Males outnumbered females, except in the lecture only section where they were equally split as shown in Table 2. Table 3 indicates that most students had moderate computer experience. 1) Did CCTST scores differ according to the technology integration (Blackboard/electronic classroom), at the beginning of the course and at the end of the course? The results were compared and are shown in Table 2 and Figure 1. The mean of the critical thinking scores showed a positive increase at the end of the course compared to the beginning of the course (11.00 to 14.28) in the electronic classroom section. A decrease (11.28 to 10.32) was noted in the Blackboard section. The medians and modes follow the same pattern. (See Table 4 and Figure 1). 2) Did final class grades differ between the two groups (Blackboard/electronic classroom)? Grades on the final exam (Ginormous and Healthlite) for students in these sections are shown in Table 5. The mean grade on the Healthlite case analysis, for those in the Lecture + Blackboardt class, was 84%, compared to 82% for the Lecture + Electronic Classroomt. (See Table 5). The grades on the Ginormous case analysis were the same for both Electronic Classroomt and Blackboardt, 96%. Clearly students using either of the technologies were able to handle the ambiguous Ginormous and Healthlite cases. Those in the lecture only situation scored higher on the Healthlite case analysis, 94.5%, than either of the technology sections. A turn around seems to be emerging, however, with respect to the final "exam" grade for the Ginormous case. Students using both the technologies had a higher grade 96% compared to 93.5% in the lecture only class. There was a slightly greater difference between the Healthlite and the Ginormous cases for those in the Lecture + Electronic. These results are consistent with the test scores obtained from the CCTST. 3) Were there differences in the students' perceptions of the support that Healthlite offered in the acquisition of critical thinking and other higher-order learning skills? Was the students' perception dependent on the technology used, e.g., Blackboard or electronic classroom? Support was rated as "None", "Some, or "A lot". The figures of those indicating, "A lot", are reported in Table 6. Students were asked to rate their perceptions of how the Healthlite case study tool helped to foster an increase in their critical thinking skills, and other higher-order learning skills, whether they used technology or not. A higher percentage of students, who analyzed Healthlite without technology than students who used technology perceived the case to contribute to higher order learning skills, critical thinking (59%), problem-solving (56%), and creative idea generation (52%) (Electronic Classroom - 45%, 35%, 36%, respectively; Blackboard - 46%, 47%, 44%, respectively). Research skills were perceived to be supported by a smaller percentage in the lecture class than among those who used Lecture + Blackboard, (22% vs. 26%). However, they were more likely to perceive support for higher order skills than those who used the lecture + Electronic Classroom, (22% vs. 17%). Critical thinking was perceived to be equally supported by both Blackboardt (46%) and the Electronic Classroom (45%). But, Blackboardt (47%, 26%, 44%) was perceived as offering more support to acquiring problem-solving skills, research skills and creative idea generation, than the Electronic Classroomt (35%, 17%, 36%), when combined with the Healthlite case. Complete results are shown in Table 6. 6. DISCUSSION In monitoring discussions or group work activity faculty must engage in a line of questioning that will continue to drive an idea, thus helping students develop and apply critical thinking skills (MacKnight, 2000). The results of the study are quite interesting. As graduate computer science students, it was expected that computer literacy was high. These particular students did not have exposure to Healthlite or Ginormous prior to the start of the class activities. Although students felt that the integration of Blackboard, with analysis of the Healthlite case, resulted in better support for critical thinking and other learning skills than integration of the electronic classroom, the results did not support this. The grades, pre- and post- CCTST scores suggest that higher-level learning skills, including critical thinking, were acquired regardless of the technology integration employed. According to MacKnight (2000), "In any discipline, the level of questions asked influences the depth of thinking that occurs". "The instructor's role is not to transmit information, but to serve as a facilitator for learning. This involves creating and managing meaningful learning experiences and stimulating students' thinking through real world problems" according to Gokhale (1995). 7. CONCLUSION The results suggest that integrating technology helps with the acquisition of higher-level learning skills in graduate students taking an introduction to Information Systems Principles course. This appears evident by their final grades, the CCTST scores, and notwithstanding their own perceptions. The integrated functions included both the threaded discussion board and chat room in Blackboardt along with the other electronic classroom technologies. A more extensive study would have to be undertaken to generalize these findings to other students. Nonetheless, the findings are promising and interesting. 8. FUTURE WORK Further investigation of this process is needed. We would propose extending our research to include universities with similar smart electronic classrooms. The need for at least two sections of the same course taught by the same faculty member would assist in determining which functions, within this environment, contribute to the critical thinking and its perceptions for the students enrolled in these courses. A more rigorous instrument will also have to be developed to ascertain the pre- and post student perceptions of the contributions made by the various technologies. It may also be desirable to compare the different dispositions of critical thinking skills such as analysis, inference, explanation, evaluation, interpretation, and self-regulation, as provided by the CCTST instrument. 9. ACKNOWLEDGEMENTS The students and faculty who participated in this pilot should be acknowledged for their assistance in taking the time to complete the surveys and assessments. Pace University's President, Dr. David Caputo, should be acknowledged for awarding funds via a Learning Assessment Grant to continue this research. 10. REFERENCES Carl, Walter John, III, 1996, "Six Thinking Hats: Argumentativeness and Response to Thinking Model." Annual Meeting of the Southern States Commission Association, Memphis, TN. Chin, Jr., G. and J.M. Carroll, 2000, "Articulating Collaboration in a Learning Community." Behavior and Information Technology, 19(4), pp. 233-245. Cook, P., et. al., 1996, "Critical Thinking Assessment: Measuring a Moving Target." Report and Recommendations of the South Carolina Higher Education Assessment Network Critical Thinking Task Force, Rock Hill, SC. Coppola, J.F. and B.A. Thomas, 2000, "Creative Learning Technologies that Enhance Critical Thinking Skills in a Smart E-Classroom." Proceedings of the IEEE Frontiers in Education (FIE) Conference, Kansas City, MO. Duncan, D.G., 2000, "Using Computers in the Classroom: A Boon or Bust?" Journal of Computer Information Systems, 40(4), Summer, pp. 1-4. Facione, P.A., 1998, "Critical Thinking: What it is and Why it Counts." California Academic Press, Millbrae, CA. Facione, P.A., N.C. Facione, M.A Blohm, K. Howard and C.A. Giancarlo, 1998, Test Manual: California Critical Thinking Skills Test (CCTST). California Academic Press, Millbrae, CA. Facione, P.A., N.C. Facione and C.A. Giancarlo, 1998, Professional Judgement and the Disposition Toward Critical Thinking. Millbrae, CA: California Academic Press. Girod, M. and S. Cavanaugh, 2001, "Technology as an Agent of Change in Teacher Practice." T.H.E. Journal, April. Gokhale, A.A., 1995, "Collaborative Learning Enhances Critical Thinking." Journal of Technology Education, Fall, http://scholar.lib.vtedu/ejournals/ JTE/v7n1/pdf/gokhale.pdf Herrington, J. and R. Oliver, 2000, "Using Situated Learning and Multimedia to Investigate Higher-Order Thinking." Journal of Educational Multimedia and Hypermedia, 8(3), pp. 401-422. Iowa State Department of Education, Des Moines Division of Community Colleges, 1995, "The Iowa Adult Basic Skills Survey." Comprehensive Adult Student Assessment, Final Report, San Diego, CA. Jones, E.A., 1994, "A Plan for Validating Criteria and Measures to Monitor Progress Toward National Education Goal 5.5: Identifying College Graduates' Essential Skills in Writing, Speaking and Listening, and Critical Thinking." Contract No. R117G10037, University Park, PA: National Center on Post-Secondary Teaching, Learning and Assessment. Liao, Yuen-kuang Cliff, 2000, "Effects of Hypermedia on Students' Achievement: A Meta-Analysis." Journal of Educational Multimedia and Hypermedia, 8(3), pp. 255-278. Lewis, A. and D. Smith, 1993, "Defining Higher Order Thinking." Theory into Practice, 32(3), pp. 131-137. MacKnight, Carol B., 2000, "Teaching Critical Thinking Through Online Discussions." Educause Quarterly, 4, pp. 38-41. Mukherjee, Arup, 2000, "Effective Use of In-class Mini Cases Analyses for Discovery Learning in an Undergraduate MIS Course." Journal of Computer Information Systems, XXXX(3), Spring, pp. 15-23. Raths, David, 2001, "Measure of Success." Online Learning, 5(5), pp. 20-26. Schumacher, B., et. al., 1997, "Assessing Knowledge and Cognitive Skills." Annual Meeting of the National Communication Association, Chicago, IL. Stoney, Sue and Ron Oliver, 1999, "Can Higher Order Thinking and Cognitive Engagement Be Enhanced with Multimedia?" Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 2(7). Retrieved August 2001, from http://imej.wfu .edu/artiles/1999/2/07/index.asp 11. TABLES Lecture Only Lecture + Electronic Classroom Lecture + Blackboard <20 20-29 30-39 >39 <20 20-29 30-39 >39 <20 20-29 30-39 >39 2 18 6 1 0 19 7 3 0 27 11 1 Table 1 - Age Lecture Only Lecture + Electronic Classroom Lecture + Blackboard Male Female Male Female Male Female 13 13 20 9 22 13 Table 2 - Gender Lecture Only Lecture + Electronic Classroom Lecture + Blackboard Minimal Moderate Extensive Minimal Moderate Extensive Minimal Moderate Extensive 4 19 4 10 14 3 9 18 11 Table 3 - Previous Computer Experience Pre- Post- Pre- Post- Blackboard Blackboard Electronic Classroom Electronic Classroom N 29 31 29 25 Mode 11 7 11 14 Median 11 10 11 14 Mean 11.28 10.32 11 14.28 Standard Deviation 3.14 3.64 3.52 3.86 Table 4 - Pre- and Post- Critical Thinking Test Results Summary of Blackboard and Smart Electronic Classroom Lecture Only Lecture + Electronic Classroomt Lecture + Blackboardt Healthlite Ginormous 94.5 93.5 82 96 84 96 Table 5 - Mean Student Grades Lecture Only Lecture + Electronic Classroomt Lecture + Blackboardt Critical Thinking Skills Problem-Solving Skills Research Skills Creative Idea Generation 59 56 22 52 45 35 17 36 46 47 26 44 Table 6 - Student Perceptions - Healthlite % Support for Critical Thinking Skills - "A Lot" 1 coppola@pace.edu 2 bthomas@pace.edu 3 jthomas@pace.edu