A Comparison of Using CBT and Teaching Assistants to Teach Microsoft Office 2000 Peter Simis Ivan Hoong The Sid Craig School of Business California State University, Fresno 5245 N. Backer Fresno, CA 93740-0007 peters@cvip.net Abstract An experiment was conducted during the spring 2001 semester to determine the effectiveness of using a computer-based training (CBT) package to cover the basic concepts of Microsoft Office 2000. Two sections were conducted using CBT and 14 sections were conducted using a traditional lecture/lab format. The students in both groups were given the same tests using an on-line assessment package. No significant differences were found between the two delivery methods. Keywords: CBT, computer-based training, Microsoft Office 2000 Introduction Our university has a typical computer concepts course that covers the fundamentals of computer technology in a weekly 2-hour lecture session and the basic components of Microsoft Office 2000 in a 2-hour lab session. The lecture sections average over 100 students, while the labs are limited to 24. During the spring semester 2001, the lab portion of this class experimented with having two sections learn the material using a computer-based training (CBT) software package from Course Technology, while the other 14 sections were conducted in the traditional manner using teaching assistants (TA). This paper reports of the results of that experiment. Methodology During the first week of instruction the students in two sections of the class were notified that they were enrolled in a section that had been selected to try a self-paced software (CBT) package to cover the material. No notification was given them or other students that this option was available. These students were given the opportunity to change to a traditional section, but none did. Two sections were selected because they yielded sufficient numbers for experimental design purposes and it allowed us to provide the CBT disks at no cost to the participants, so their cost for the course would not differ from those taking the traditionally taught sections. The students in the CBT labs were given CBT disks. and informed their only requirement was to show up on the scheduled dates to take a test to prove their proficiency in Microsoft Office 2000. There were no homework assignments and no scheduled times when they could go to a lab for help. Their communication with the instructor was through email and an e-group. The students in the traditional sections had weekly homework assignments and periodic quizzes in a lab where a TA demonstrated some of the material and was available to help. Periodically throughout the semester, both sets of students were tested on the same material using a software package called Skills Assessment Manager, also from Course Technology. This software tests the student on the material by confronting them with a set of tasks to perform within the Microsoft Office product. The software compares the student's response with a stored template and scores the effort. During the first week of classes, and whenever new students were added, the students were given a questionnaire to collect demographic information regarding such items as major, year in school, and prior experience with Microsoft Office. Kolmogorov-Smirnoff tests were used to compare the data from the two sets of students. There were no significant differences between the two groups on demographic factors. Results Three tests were administered and analyzed for each set of students. The tests covered Microsoft Word, Excel, and Access. PowerPoint was also tested, but there were indications that the students in the traditional sections were not informed they would be tested on this package and therefore were not prepared. The average scores for Word, Excel and Access are shown in Table 1. Table 1 Comparison of Scores from CBT and Traditional Sections CBT Traditional Software N Avg. score Std. Dev N Avg. Score Std. Dev. Sig. .05 Word 42 77% .13 302 79% .16 No Excel 39 71% .16 285 75% .13 No Access 39 57% .19 271 61% .16 No Discussion It needs to be emphasized up front that any conclusions related to CBT must be restricted to the particular CBT package used in the study, the one from Course Technology. While this was perceived to be the best CBT package based on research conducted in Fall 2000, there may be other CBT packages that would have yielded different results. Students perceived the CBT package as being overly picky - it was process oriented rather than giving credit for achieving the correct answer. In each case shown in Table 1 the CBT students performed at a lower level than those in the traditional labs, although the results are not statistically significant. Clearly, the CBT package is not more effective than the traditional teaching method (lab TA's cheered), nor has it been proven to be less effective. Table 2 Percentage of Students Scoring Below 70% Software CBT Traditional Word 26.2% 15.3% Excel 44.4% 31.0% Access 69.2% 64.6% Table 2 provides some insight regarding the suitability of CBT to the student population. While there was no significant difference in the overall results, based on the mean, there was a clear distinction between those who did poorly in the lab and those who struggled in the traditional sections. For each software package, those using CBT performed much more poorly. The reasons for this have not been determined objectively, but one can speculate that many of the CBT students needed the time schedule structure provided by the traditional lab to keep up with the learning process. Further, our students seem to have a "will work for points" mentality, such that the provision of points for homework and quizzes in the traditional format encouraged their involvement in the learning process. The relative passing results between software packages are those that would be expected by ones who have been associated with this course. Word is the easiest of the subjects to master and Access is the hardest. This probably has as much to do with the prior experience of the students with the material as with the difficulty. As shown in Table 3, almost all the surveyed students had used Word, many had used Excel, but only half had used Access prior to the class. Table 3 Percentage of Students Reporting No Prior Experience with Software Software CBT Traditional Word 4.7% 3.8% Excel 20.9% 29.8% Access 48.8% 53.6% Conclusions and Recommendations While many students struggled with the CBT process, many also did well, including a majority of those in the Word and Excel portions. CBT is an effective tool for some students, but there is a need for the traditional method of covering the material. CBT requires more self-discipline and the ability to solve problems on one's own.. It appears there are many in our student population who lack these characteristics. The Course CBT package from Course Technology needs refinement, but it does provide a very good environment for students to be exposed to the material. Students become frustrated when they solve a task using one method and the package scores their effort as wrong because a different solution method was called for. Unlike the SAM package, the CBT package provides a simulated Office environment, therefore it is more process oriented than results oriented. A way is needed to pre-test students regarding their ability to be successful with a CBT tool. If we can reduce the numbers of students taking the traditional labs we can save money in the form of wages and resources tied up in labs. Clearly those students would benefit from not having to sit through the labs and more attention can be given to the more needy students. Further research is planned to provide a means for students to self-select an instructional mode for achieving competency in this material. The poor results from both teaching methods in Access point to the need for further examination of this area. If it is assumed that competency in a database package is beneficial to our students, we need to reevaluate how this material is being covered and determine the source of the learning problem. The extent of this problem was not realized until the results of the study were analyzed. It is recommended that schools consider adopting CBT to permit students to learn the basics of Microsoft Office needed for computer literacy. However, accommodations need to be made for those for who require a structured learning environment. Even if one does not adopt CBT, it is highly recommended that schools adopt an on-line assessment package such as SAM. This is a clearly superior way to have students prove competency in using a software package. Such a package can be used as an exit exam or to have students test out of course/program requirements. Side-notes The assessment package provided some interesting insights into the effectiveness of our computer concepts lab. Each of the tests was based on the material covered in an introductory text. We found that in most cases the lab TAs were not covering all the material at this introductory level. For example, they were not able to get to the topic of tables in Word. It is easier to require the students to cover this material in the CBT mode because they are focused on taking a competency test at that level., whereas the traditional lab students force a slower pace to accommodate the slower learners. We will be making the traditional labs more output oriented. Further, we found that the student scores were lower using the assessment package than with our traditional method of testing. The assessment package seems to be a better measure of their ability. We will continue to use some form of assessment software regardless of the decision made regarding CBT.