Course Technology & Online Education: A Study of the Impact on Student Learning Andrea S. Taylor1 Linda Jo Calloway2 School of Computer Science & Information Systems, Pace University 1 Pace Plaza, New York, NY 10038 USA Abstract This paper describes a quantitative study on the use of course technology/online education to enhance student learning. The objective was to study the effects of using course technology/online education upon the success and learning of undergraduate students in a particular course. The course, Fundamentals of Programming, was taught over the course of 4 semesters and 75 students were evaluated. The course takes place in a hands-on lab classroom. The Fall Term A and B semester courses did not use the technology and the Spring Term A and B semester courses did. All students in both courses were given the same in-class instruction and the same number of similar assignments. The two research questions are: (1) What is the comparison of student grades and course completion between the Fall semester course without course technology and the Spring semester course using course technology; and (2) Do student grades correlate with access and usage of course technology during the Spring semester course? Keywords: technology-enhanced instruction, student learning, retention, assessment 1. INTRODUCTION This paper describes a quantitative study on the use of course technology/online education to enhance student learning. The objective was to study the effects of using course technology/online education upon the success and learning of undergraduate students in a particular course. The course, Fundamentals of Programming, was taught over the course of 4 semesters and 75 students were evaluated. The course takes place in a hands-on lab classroom on the New York campus. The Fall 1999 Term A and B semester courses did not use the technology and the Spring 2000 Term A and B semester courses did. The Fall 1999 courses were taught with the use of a projector attached to the professor's workstation to visually demonstrate design and coding of projects. Student grade and course completion rates from the Fall 1999 semester were compared with those from the Spring 2000 semester. All students in both courses were given the same in-class instruction and the same number of similar assignments (6 homework assignments, 1 extra credit assignment and 1 final project). All courses were taught in a hands-on lab classroom setting. The two research questions are: (1) What is the comparison of student grades and course completion between the Fall semester course without course technology and the Spring semester course using course technology; and (2) Do student grades correlate to access and usage of course technology during the Spring semester course? The research was inspired by the overwhelming increase in course technology/online education in higher education and the need to analyze its impact, either positive or negative, on students. This summer, I plan to teach another course using this technology, whereas I had not used it to teach the same course previously. I would like to continue my observations with these students and compare their retention, learning and involvement with previous students in the same course taught without the online education. Qualitative evidence collection included a survey of faculty and online assessment/tracking. The analysis of students' success and learning in the course focused on their personal feedback, the level of programming achieved in their assignments, and their access and usage of the website. The problems students experienced in not being able to access the site from the beginning to midway through the course was an important stressor. The course technology/online education piece was introduced as a website to supplement the in-class instruction. Blackboard Inc.'s CourseInfo technology provided the online course website and Pace employees served as its administrators. All students in the course were given access to the website with usernames and passwords. The benefits of having access to course technology included enabling students to keep up with the course even if absent, to spend less class time coding, to have design and coding demonstrated via pc and projector, and to have convenient access to course information. Summary of the Paper This introduction concludes with the description of the technology used in the courses and a definition of terms. Section 2 is a review of the key literature used in designing this study. Section 3 is a description of the study and Section 4 describes the results of the study. The paper concludes with a brief discussion in Section 5. About the Technology Blackboard Inc. claims that "nearly half of higher education institutions engage in online distance learning ..." and "penetration of campus networks has reached 83% for higher education as a whole and half of all college students own personal computers, making online education as much an on-campus phenomena as distance learning." (Blackboard.com) Blackboard CourseInfo is server software developed for institutions and/or the individual departments for a common look and feel for all of their course websites. The software powers online teaching and learning environments at more than 1600 leading colleges, universities and K-12 schools in each US state and more than 70 countries. Other companies offer similar services, including Course Technology's MyCourse.com, CyberClass and WebCT. Pace University also offers another service called WebBoard. These online education options are encouraged by Pace University, as we have formed a TLTR (Teaching, Learning, and Technology Roundtable) Group, which represents diverse parts of the university, has regular discussions on how to improve teaching and learning with technology, and provides recommendations to the CIO (Chief Information Officer) and other academic leaders. Terms Defined Online education - the use of a website on a network to supplement instructional teaching Distance learning - the use of applications and networks to provide instruction without (or with very limited) face-to-face instruction Course technology - applications developed to assist with online course syllabi and instruction for online education or distance learning Multimedia applications - video, graphics, and text combined in a presentation, either online or in class 2. REVIEW OF THE LITERATURE In reviewing literature, preparatory to designing this study, many types of technologies are used in many different ways. For example, the Rickman & Grudzinski (2000) study reported results of the use of seven different technologies used in the classroom. This study justifies the use of technology in the classroom because the study results show that students perceive that the technology is helpful and useful. The assumption that technology enhances student learning and improves performance is based on the fact that students perceive technology use as beneficial. However, the effects of the technology on student learning and grades are not clear. With the use of classroom assessments, students are afforded the opportunity to express their needs, goals, concerns, etc., as they relate to their learning, and faculty are able to give anonymous feedback to the class as a whole or on an individual basis. Although the usage of classroom assessment techniques were reported to be beneficial to both students and faculty in both the teaching and learning aspects, Dr. Diana Kelly's study showed no significant change in students' grades with or without the use of assessments (Kelly, 1993). Surveys taken by students and faculty at Northwest Missouri State University on the use of technology in the classroom revealed that students did not want the technology used at all times, but did expect its use in all subject areas as an enhancement to, not replacement of, in-class instruction. Both faculty and students agreed that faculty training and technical support are important aspects of a university's commitment to information technology (Rickman & Grudzinski, 2000). The percentage of time Northwest Missouri State University students felt technology should be used in class equaled the percentage of time technology is actually used by faculty; and the average percent of time students felt technology should be used was 45% of the class time. The student and faculty comments of the survey revealed that students felt faculty should be trained or supported in the use of the equipment so as not to lose valuable class time due to a professor not being familiar with using the equipment; both faculty and students felt that technology should enhance not replace the instructional experience; and students prefer clear, concise Powerpoint presentations for in-class use, as well as Web downloading. While many students and faculty are comfortable with the use of technology in the classroom, there are certain concerns that need to be considered. Courses taught via video-conferencing tend to have communication issues associated with them. For example, as evidenced by a study done at Ball State University in Indiana in 1997, those courses that rely on Internet access can be frustrating if the server is down or the network is inaccessible (Saunders, et al., 1997). Furthermore, the study showed that there are also computer novices or cyberphobic students that see coping with the computer environment as another challenge to learning. The use of an online text-based medium can also present a challenge to many students to stay focused on the course material. In the same vain, the results concluded that some instructors may not be comfortable with course technology or online education and require technical support and training. In the Ball State study, most students, those that are computer savvy as well as those with less experience, expressed a benefit in gaining a sense of empowerment and satisfaction, derived from the learning independence online education requires, but they also appreciated the value of the face-to-face encounter with the professor. For each of the studies, education was the discipline, therefore the comparison groups consisted of undergraduate, graduate and adult evening students. The use of comparison groups in all 3 studies encouraged the use of comparison groups in this study. 3. DESCRIPTION OF STUDY The results of the literature analysis revealed no conclusive relationship between the use of course technology/online education and its effects on students' grades, learning involvement, and course completion. There were two schools of thought evidenced by the literature analysis. The first school encourages the use of technology as an enhancement to in-class instruction and the improvement of student learning (e.g., see Gilbert, Steven, Feb. 14 & 22, 2000). The second school encourages its use but with no belief that it serves to improve student learning. This study explores the relationship between course technology and its impact on student learning, grades and course completion. The assumption is that those students who accessed the CourseInfo website for in-class coding demonstrations, hints to complete the homework assignments, descriptions of projects, course requirements, and assignment deadlines had a better chance of learning and understanding the course material, getting a higher grade and completing the course. It was expected that in comparing the same course over 2 semesters, one semester with the use of course technology and one without, the findings would reveal a better grasp of the course concepts and higher grades for the semester with the use of the course technology. The accessible areas of the CourseInfo website included Content, Communication, Group, and Student Areas. The Content Areas consisted of assignments, course outlines, staff information, and course documents. The Communication Areas enabled the students to email the professor, email each other, and participate in a discussion board. The Group Areas included areas for students to communicate as a group, such as virtual chat and discussion board forums. The Student Areas allowed the students to check their grades throughout the course, drop their assignments into a digital dropbox accessible by the professor, create home pages, and change their contact and password information. 4. RESULTS OF THE STUDY There are 3 sub-sections - the first describes the results of the Spring 2000 semesters, where course technology was used; the second describes the results of the Fall 1999 semesters, where course technology was not used; and the last highlights the relevant comparisons between the 2 semesters. Spring 2000 IS 224 Term A and B Classes Within the Spring 2000 semester, we analyzed those students with access to the CourseInfo website for the course and how they fared in relation to the access and usage of the site. Term A - 23 students Term B - 7 students 27 students completed the course Of the 3 that did not complete the course, 2 students dropped the Term B course and another did not complete any of the requirements of the Term A course and will audit the course this summer in an effort to improve his grade. Areas Accessed: (see charts below) Of the available areas to access, students preferred the course-relevant areas and did not bother with the Group Areas at all and did very little communicating. Access by Day of the Week: Most students accessed the site on Tuesdays and Thursdays, which were also the days of the week the course was offered. It is our observation that they waited until the last minute to download the needed in-class project for that day's class meeting or also, they needed to read the homework hints in order to do the homework due for that day. The next highest amount of hits occurred on Mondays and Wednesdays, especially during Term A and with more students in that term. Access by Hour of the Day: The number of hits concentrated in one large section of time, from 10am - 6pm, were evident in both terms. The highest percentage of usage occurred at 6pm in Term A and 5pm in Term B. Total Access by Users: Most of the students in the Term A course had access to the website. Two students, Student DS and Student JS, did not have access to the website due to site administration problems. For the Spring 2000, Term B course, one student (Student MI) dropped the course midway through the semester. One student (Student MR), had trouble accessing the site, but did not inform me until the end of the semester. Fall 1999, IS 224, Term A and B classes This course was only taught as in-class instruction, with no use of course technology or online education. The only technology used in the classroom was a projector attached to the professor's workstation. Term A - 25 students Term B - 23 students All 48 students completed the course. Comparison of both semesters The following table reveals the students' grades for Fall 1999 (without course technology): Term A A 18 A- 2 B+ 3 B- 1 D 1 Term B A 10 A- 4 B+ 3 B- 3 D 1 F 2 The following table reveals the results for Spring 2000, Term B (with course technology): # of Hits Percent Grade 157 22.1 A 132 18.6 A 130 18.3 A 117 16.5 A- 16 2.25 A 12 1.69 A 1 0.14 B This particular class was so small that the results are not readily able to be included in this study. Of note, the A students averaged 89.4 hits at 12.59%. Both these averages and individual results reveal that this course of students fared better with less usage of the site. One possibility for these results could be the individual attention afforded with only 7 students. Based on the number of hits and percentage of usage of the site during the Spring 2000 Term A, the following table reveals a slightly significant difference between the grades of those students that accessed the site, those that accessed it frequently, those that accessed it a few times, and those that did not access it at all: Spring 2000 Term A (with course technology) Final Grade # of Hits Percent A 363 11.3 356 11.0 317 9.88 245 7.63 221 6.88 156 4.86 144 4.48 120 3.74 112 3.49 105 3.27 63 1.96 60 1.87 46 1.43 3 0.09 A- 164 5.11 B+ 103 3.21 0 0 B 221 6.88 B- 72 2.24 C+ 25 .77 D 21 .65 F 43 1.34 0 0 The average hits for grades of A were 165 hits at 5.11%; for A-, the average hits were 164 hits at 5.11%. For grades of B+, students averaged 51.5 hits at 1.61%; B, 221 hits at 6.88%; and B-, 72 hits at 2.24%. As the grades got lower, C+ to F, the number of hits and percentage decreased. The students with grades of B averaged more hits and higher percentage than those students with higher grades. One of the students without any access to the site fared pretty well with a grade of B+. * Of the students in the Spring 2000 courses (out of 27 students), 63% received A's (with course technology) * Of the students in the Fall 1999 courses (out of 48 students), 58% received A's (without course technology) * Of the students in the Spring 2000 courses, 7% received F's (course incompletion) (with course technology) * Of the students in the Fall 1999 courses, 4% received F's (course incompletion) (without course technology) All Grades for Students in Fall 1999 IS 224 Term A & B Classes All Grades for Students in Spring 2000 IS 224 Term A & B Classes 5. DISCUSSION Course technology is an enhancement and supplement to the instruction. In this study, it was also perceived by the students as being beneficial. However, the results of the study do not show a dramatic correlation between grade distribution and usage of the course technology site. Nevertheless, there is a direct correlation between individual student grades and site access in the courses that were supported with course technology. These results could indicate that better students chose to access the site more often. It is also possible that some poorer students improved their grades by having access to the course material via the course technology site. However, these results could also indicate that a small percentage of students may have become disenfranchised by the inclusion of the technology. It appears, throughout the research, that the objective, practical consensus of opinion among the higher education community is that distance learning will peak, but the use of course technology to enhance and supplement instruction, communication and information management will sustain a long life. Steven Gilbert's Connected Education and Collaborative Change papers support this view. There is an increase in universities that are providing technology to students, faculty and staff. Along with this increase in technology, must come an increase in valued support for both the technology and those who use it and those who access it. We must be careful in making it mandatory to access course information online may create a disadvantage for students who do not have access to technology in their homes and may not have the time to stay at the school and use the technology provided there. College recruitment competition is focused on how well the institution supports the use of technology for teaching, learning and research. Currently, there is not enough research available on the benefits of technology investments to encourage some institutions to make the commitment. The jury is still out on the best fit for technology applications and traditional educational instruction. The evaluations and assessments of the impacts associated with technology and education must continue in order for institutional goals to change with the times. The growing use of word processing, presentation graphics, electronic mail and the World Wide Web in conjunction with traditionally scheduled and structured courses proves that we are on the right track. Half of all courses in US colleges and universities involve some email communication among students and faculty. Faculty have reported that the volume of email has increased and their workload has increased. Many younger (ages 18-25) students are more visually oriented and therefore seem to appreciate and are more comfortable with TV-like screens, pictures, diagrams, animation, and video clips in the classroom. Traditional-age students are more receptive to sound (more in the form of recorded music) and faculty need to pay attention to developments that include human speech being transferred to the Web. The increased use of information technology in the classroom has spurred much debate and very little concrete study as to student expectation, faculty preparation, and institutional technological support. Many universities are pushing for state-of-the-art, networked, multimedia facilities as a way to attract students and faculty. The mixture of online and face-to-face education will become more common than programs that offer either one alone, as more studies, assessments, and evaluations are done that focus on the effective use of such technology in achieving important educational goals. 6. REFERENCES Burgstahler, Sheryl & Swift, Catherine, 1996, "Enhanced Learning through Electronic Communities: A Research Review", University of Washington. CAST (Center for Applied Special Technology), "The Role of Online Communication in Schools: A National Study" Eaton, Judith S., 2000, "Online Teaching and the Issue of Quality", AGB Journal, March/April. Ehrmann, Stephen, 2000, "Selecting Web-Based Course Management Systems for a College or University, Segment 4 - Evaluation", February 15. ERIC (Educational Resources Information Center) Clearinghouse, Educational Technology, http://ericir.syr.edu/ithome/edutech.htm Garton, Edwin M.; Holloway, Ken C.; Wegner, Scott B., 2000, "Student Achievement in a Distance Learning Environment", EduCause Quarterly, Number 1. Gilbert, Steven W., 2000, "Building a Curriculum for Change: Connected Education and Collaborative Change" February 22. Gilbert, Steven W., 2000, "A New Vision Worth Working Toward - Connected Education and Collaborative Change", February 14. Green, Ph.D., Kenneth C., 1997, "Drawn to the Light, Burned by the Flame? Money, Technology, and Distance Education", The ED Journal, May, Volume 11 #5. Grudzinski, Mike & Rickman, Jon, 2000, "Student Expectations of Information Technology Use in the Classroom", EduCause Quarterly, Number 1. Kelly, Ph.D., Diana K., "Using Classroom Assessment Strategies to Improve Student Learning" Saunders, Nancy G.; Malm, Loren D.; Malone, Bobby G.; Nay, Fred W.; Oliver, Brad; Thompson, Jay C. Jr., 1997, "Student Perspectives: Responses to Internet Opportunities in a Distance Learning Environment", Ball State University, November. 1 ataylor@pace.edu 2 lcalloway@pace.edu