Surveying Students about Computing: Results of a Two Year Study Jeanine Meyer & Stuart Varden Information Systems Pace University 1 Pace Plaza New York, NY 10038 jmeyer@pace.edu, svarden@pace.edu Abstract As part of a continuing effort to understand our students in order to inform our teaching, we have surveyed the students in our core computing class at the start and the conclusion of the fall term. The surveys solicit information on prior experience with computing, attitudes towards computers and technology and other academic matters, hopes for the course, knowledge of current events and general demographic information. We present here our analysis of two sets of surveys given in fall 1997 and fall 1998. The findings confirm some subjective impressions on the part of the faculty, indicate opportunities and also reveal challenges. This is a follow-up study of a previous published study. Keywords: student attitudes, computer literacy, introductory computer course 1. BACKGROUND ON PACE UNIVERSITY AND CSIS 101 Pace University is a multi-campus institution serving a diverse population in New York City and Westchester County. Many of the students are the first in their families to pursue higher education. Immigrants, included so-called ESL (English as a Second Language) also represent a significant portion of the student population, especially on the New York City campus. Pace began as a business school and is now made up of the Lubin School of Business, the School of Computer Science and Information Systems, the School of Education, the Lienhard School of Nursing and the Dyson College of Arts and Sciences. The University also includes graduate programs and a Law School. Students tend to be focused on careers and jobs. The Pace co-op program is one of the attractions of the school. An Honors program also exists on both campuses in which students take Honors sections of standard courses, participate in special cultural and social activities, and complete an Honors project. All students take a core set of courses. One of these courses, the only one outside the traditional liberal arts, is Computer Information Systems 101 (CIS101). This course has at least three identities. It is a 'first course' in computing for people going on to major in this field and can also serve to recruit students to majors or minors or certificate programs. The course is also a computer literacy course, with hands-on experience in a computer classroom using Visual Basic, Excel, optionally PowerPoint, Web browsing and construction of Web pages using HTML. Lastly, the course is an introduction to the fundamental concepts of computer information systems similar to a 'Physics for Poets' course. The course is structured as a two-hour lecture/discussion (large size sections, capped at 72 for day classes) and a separate two-hour closed lab (capped at 24). Two different people generally teach these, with the lecturer teacher in charge and the lab teacher an adjunct or graduate student. Students going on to other computing courses may waive CIS 101 if they have had experience with computing, most specifically programming. This once was nearly universal for the computer science majors but is slightly less so now. A very small number of students opt to take a challenge exam, which can result in credit for the course without any requirement to take more computing courses. Requiring all students to take a computing course is not common, even in institutions with core requirements. 2. OBJECTIVES OF SURVEY CIS 101 is a substantial undertaking, involving around 2000 students per year, many sections and many different instructors across two distinct campuses. It also has this unique standing as the only non-liberal arts core course, and is subject to constant oversight and attention on the part of faculty, administration and students. The faculty continues to debate what should be in the course. The presence of a programming component is the most controversial. Faculty and others tend to form hypotheses about what incoming students are like and what they want to learn. Some people speculate about the presence of a "computer phobia." Theories abound about male versus female likes and interests. Other speculation is that students just want to learn how to use spreadsheets, based perhaps on the significant presence of business majors. We decided to do surveys to confirm or reject these theories in order to better inform our curriculum design and our teaching. After doing one set of surveys at the start of the 1997-1998 academic year, we felt obliged to repeat the survey process to confirm the original findings and to discover any trends. 3. STRUCTURE AND CONTENT OF SURVEY Surveys were administered during the first class meeting (the pre-test) and at the close of the semester (the post-test) over the two-year period. Altogether four surveys were conducted: a pre and post-test in fall, 1997 and a pre and post-test in fall 1998. The surveys contain questions on experiences with different types of computers and software by asking students to indicate if they have no experience, some experience or working knowledge. The surveys continue with questions relating to attitudes, starting off with "I get nervous when using computers." The format here is a 5-point Likert scale. The next questions ask students to describe specific current events. The first time we gave the survey, students were asked to describe "Deep Blue" and "the Year 2000 problem." It is important to point out that the story of Deep Blue, the chess playing computer system built by a team of IBM researchers, was heavily featured in our area of the country. The second year, we replaced this story with "Microsoft versus the Department of Justice." In both years, students were asked to identify "Year 2000 problem". Students were also asked to describe any other news event involving computing. The pre-course surveys asked what students hoped to get out of the course as an open-ended question. Then, the post-course survey took several of the answers given in the pre-survey and asked about them in the form of a Likert scale. The survey continued with demographic type questions: gender, age (within age ranges), credits accumulated, major, first language and language of schooling. We also asked if they expected to use computers in their careers: choosing among extensive use, some use, or little or no use. The current events questions were omitted from the post-survey (given the last day of class) since current events are covered explicitly in the curriculum and faculty agree to include such topics on the final examination. Surveys were given to sections in Westchester and New York City, and both day and evening. The suggested protocol was to tell students that we are "doing this survey in order to improve our teaching. It is to be anonymous. We appreciate their efforts." Surveys are a common phenomenon at Pace 4. FINDINGS Pre-test to Pre-test A total of 404 students participated in the 1997 survey, while 617 participated in 1998. These large sample sizes mean that even very small differences will prove statistically significant. Thus, statistical significance itself is not very revealing. We begin with a comparison of IT skills from 1997 to 1998. It should be emphasized that the student knowledge levels were self-reported. There was no attempt to verify whether or not students actual possessed the knowledge that they claimed to have. IT Skill 1: Familiarity with Windows Year Never used Some use Working knowledge 1997 6 41.1 52.9 1998 3.7 34 62 Comment: The surveys confirm that computer use is becoming pervasive. For example, the use of Windows, already substantial in 1997, increased in 1998: IT Skill 2: Knowledge of word processing Year Never used Some use Working knowledge 1997 10.6 37.3 52.1 1998 7.3 35.9 56.9 Comment: This familiarity is with the mechanics of word processing. It is a frequent complaint of faculty that students claim skills at word processing but do not apply these skills to proofread and polish their work. IT Skill 3: Programming Skills Students who already know programming generally do not take CIS 101. This is borne out by the survey results. Interestingly enough, however, whereas few students reported working knowledge of Basic/Qbasic or C/C++/Pascal, Logo, or Visual Basic, there was increased reports of 'some use' in Logo and Visual Basic. IT Skill: Logo Year Never used Some use Working knowledge 1997 90.7 7 2.3 1998 74.4 19.4 6.2 Comment: Though there was once some use of Logo in the early years of K-12, we had thought this had faded but either there is still some activity or we have a confounding result because of terminology. IT Skill: Visual Basic Year Never used Some use Working knowledge 1997 89.4 9.3 1.3 1998 78.1 19.7 2.2 Comment: The increase here is more plausible due to the increased importance of Visual Basic. Nevertheless, few students report a working knowledge. IT Skill 4: Email Use Year Never used Some use Working knowledge 1997 24.1 36.7 39.2 1998 13.1 30.5 56.5 Comment: The use of email grew substantially. IT Skill 5: Web/Internet Use Year Never used Some use Working knowledge 1997 16.4 42.1 41.6 1998 7.6 35.8 56.6 Comment: Similarly, Web/Internet use grew. We ascribe these last two results to the AOL marketing effect and the wiring of schools. Now we turn to questions regarding attitudes toward computing and IT. In general, the differences between 1997 and 1998 were not as much as was found in the IT skill questions. This confirms for us that the population is essentially the same as far as underlying attitudes forward computing goes. Attitude 1: "I get nervous when using computers" Year Strongly disagree Disagree Neutral Agree Strongly agree 1997 32 29.2 24.4 9.2 4.7 1998 43.6 31.8 14.8 7.2 2.6 Comment: This shows a shift towards greater comfort with computers from a base that was already fairly comfortable. Attitude 2: "I get nervous learning new software" Year Strongly disagree Disagree Neutral Agree Strongly agree 1997 15.7 21.9 33.8 22.6 6.0 1998 22.7 27.7 25.8 17.1 6.6 Comment: The findings also show a shift but, we were struck by the "strongly agree" column. Attitude 3: "Computers don't like me" Year Strongly disagree Disagree Neutral Agree Strongly agree 1997 37.8 27.3 21.8 9.5 3.5 1998 43.6 30.4 16.0 6.6 3.4 Comment: Similarly, this showed a shift while leaving the most extreme column more or less along. Attitude 4: "I get nervous when learning new software" Year Strongly disagree Disagree Neutral Agree Strongly agree 1997 2.7 5.5 25.9 44.6 21.2 1998 1.4 5.1 19.4 44.1 29.9 Comment: When a similar question was worded with the opposite polarity, the results were consistent. All of these results indicate a high general comfort level with computing and also a positive trend. However, a core group appears to exist that admits to being nervous. Attitude 5: "I enjoy computer games" Year Strongly disagree Disagree Neutral Agree Strongly agree 1997 4.3 6.3 23.5 39.8 26.3 1998 3.8 4.3 21.5 40.9 29.3 Comment: Again, there was no striking year-to-year change. The current events questions showed some increased awareness. However, the cause for this is the events themselves, not increased reading of newspapers and watching the news by this cohort of mostly young adults. Our standard for an acceptable description of the event was quite lenient. The table indicates the keywords that resulted in a positive coding for these questions. Topic Keywords Deep Blue chess, computer/machine game playing Y2K any problem with computers involving date, New Year, Jan. 1, 2000 Microsoft versus DoJ Microsoft monopoly, forcing browser, operating system including browser/apps Current Event 1: Deep Blue & Microsoft versus the Department of Justice Year & Topic Don't know/couldn't describe Described event/issue 1997 Deep Blue 88.1 11.9 1998 Microsoft versus DoJ 74.5 25.5 Comment: In both cases, ignorance of the topic was quite high. It should be pointed that the "Deep Blue" topic was both a national and international story, but also a local one. The IBM team of researchers responsible for Deep Blue was located a few miles north of the Pace Westchester campus. Current Event 2: The Y2K Problem Year & Topic Don't know/couldn't describe Described event/issue 1997 Y2K 80.0 20.0 1998 Y2K 65.4 34.6 The findings show a heightened awareness, probably due to increased media coverage of the problem. When students were asked to estimate their expected future use of computers, only minimal year-to-year differences were observed. Estimated Use of Computers Year Use often Some use Little/no use 1997 62.9 34.3 2.8 1998 68.4 29.4 2.2 Finally, an open-ended question on the pre-test was to indicate what they expected to get out of the course. We coded the answers in the following categories. Note: the programming category was added in the second year. This may be because of the increased number of computing students who were directed to take CIS 101 because of a lack of experience with programming (Table 1). Campus Differences We next compare responses on the New York City versus the Pleasantville (suburban) campuses. The general finding was that in most respects there was not a great difference, and so a question-by-question breakdown does not seem warranted here. The one difference that did show up was that New York City campus students seemed to be somewhat some technically competent than their suburban counterparts. The authors believe that this is due mainly to a highly percentage of international students on the New York City campus, especially Russian, who tend to have a keener interest in computers and IT than American students. Gender The findings for self-reported experience and attitudes by gender are consistent with the stereotype, though on most cases there are no significant differences. For two of the programming categories, more boys than girls report experience. We repeat that students with strong programming backgrounds may opt out of the course. The attitude questions also revealed differences. The responses to the statement, "I get nervous when using a computer," produced results which demonstrate the stereotype of male confidence: Gender Strongly disagree Disagree Neutral Agree Strongly agree Female 40.7 30.5 15.7 10.0 3.1 Male 47.5 33.7 13.0 7.8 1.9 Notice, however, that most females also express confidence. In our informal and formal reporting on the earlier survey, audiences were surprised at the high levels of positive responses from females on the computer games statement. In the 1998 survey, the trend continues: many females claim to enjoy games, even more than before, but the proportion of males is still greater. Gender Strongly disagree Disagree Neutral Agree Strongly agree Female 5.7 4.8 23.6 40.6 25.3 Male 1.5 3.4 17.9 41.2 35.9 The general technology question, "I can program a VCR", also invoked a stronger response from males. Gender Strongly disagree Disagree Neutral Agree Strongly agree Female 9.1 8.6 7.1 36.2 38.9 Male 5.3 5.0 8.4 32.4 48.9 In contrast to the other findings, more females claimed to enjoy puzzles. Gender Strongly disagree Disagree Neutral Agree Strongly agree Female 4.3 5.4 29.5 39.8 21.0 Male 2.3 11.5 34.9 33.3 18 Pre-test to Post-test The results here were rather predictable in that students reported knowing more and having more positive attitudes about IT at the end of the course compared with the start of the course. One interesting finding was that many students reported knowing more about selected topics, such as Mac OS and C++, that were not covered in the course. We interpreted this to mean that students feel more confident in their knowledge of computers and IT, and so believe that they could learn it if they needed to. 5. IMPLICATIONS FOR CURRICULUM AND TEACHING Our second year of surveys confirms the findings of the first year and indicates expected trends. Many students come into the course with substantial experiences using computers, positive feelings and, in their words, open and optimistic attitudes towards the required course. This does put an obligation on us to keep up to date, as we have done by switching to Visual Basic and including construction of Web pages. We need to start thinking about the next additions and modifications of the curriculum. Possibilities include requiring more elaborate programming projects (using Visual Basic), introducing the creation of animations (using Paint Shop Pro and its companion Animation Shop) and formalizing the presentation unit (PowerPoint). Challenges arise from the fact that many students do report experiences with a variety of software. First, there is still a core of students who can be characterized as nervous about computers and technology. These students must function amidst students who appear skilled and confident. We need to make sure that the minority of inexperienced and nervous students is served. We do offer a Workshop for the Total Novice, but it is not well attended, so we need to recruit people more diligently to this event. Second, we have the situation that many students believe that they know all they need to know about computing simply because they do email, play games and surf the Web. It is our experience that their knowledge can be quite superficial. They do not do email reliably, their research skills using the Web are poor, and they do not proofread their papers. Their lack of accurate mental models on how things work may make it awkward for them when they need to learn new tools, but their over-confidence does not make them good students. It may be that we have moved from an era in which we were reassuring to one in which we need to have more nuanced responses. Knowledge of current events improved but remains limited. Our interpretation of the results is that students still do not read newspapers or focus on current events. As before, very few students attempted to give a response to the open question asking for any news story involving computing. What caused the difference in results was the saturation of these particular news stories. We, the faculty, still find it close to inconceivable that a majority of students could not identify these topics. It is our experience that current events, especially the two we featured in the second survey are excellent topics for instruction. We just need to accept the burden of introducing the topics in class. The results indicate that at some level students are open to the course as it is defined. Consistently over the two years, around 22% define their expectations for the course to be learning applications but the plurality claim general knowledge and if we combine the general knowledge responses with the comfort level responses, the results indicate a majority have general expectations and not narrow ones. Our incoming female students are still behind the males with respect to experiences and positive attitudes though both these areas are strong for both categories. We cannot claim that a population in which 70% disagree with the statement, "I get nervous when using computers", exhibits some gender stereotypes. Still, the results are troubling. However, what we may be able to work on is the relationship between doing puzzles and solving problems on computers. More generally, we need to help students distinguish between specific knowledge of keystrokes and general problem solving ability. This would be helpful for all students. As faculty members, we can report success with the course. Most students report satisfaction in terms of reporting 'working knowledge', modest improvements in the attitude questions, and specific objectives for the course. However, we can also be critical and more ambitious. Though there are some problems with comparing groups, we may not be doing better with the core of computer-adverse/phobic students that still exists. More generally, most of these students come to the course with positive attitudes towards technology. And yet teachers do not report high levels of achievement for these students, either in the lab in programming, spreadsheets and Web creation or in the lecture/discussion on concepts. It may be that students today view using computers as something independent of understanding them and this is not the paradox it appears to us. 6. REFERENCES Meyer, Jeanine and Stuart A. Varden, 1998. "A Profile of First Computing Course Students: New Insights and Their Implications for the Curriculum." Proceedings of the Fifteenth Information Systems Education Conference (ISECON'98), AITP Education Foundation, Antonio, TX, October 15-18, 1998, pp. 183-188. Table 1. Two-year Comparison on Expectation for Course. Year General Knowledge Applications Internet Career Increase comfort level Grade / Satisfy requirement Learn Programming Other 1997 57.5 19.9 2.3 4.0 8.1 6.6 1.4 1998 46.0 13.8 8.1 5.2 13.3 4.7 7.6 1.3