Student Expectations, Achievement and Feedback, A Study Of Student Self-Assessment In The General Education Computer Course Bruce White Computer Information Systems Department, Quinnipiac University Hamden, CT 06518 Abstract Many colleges and universities require a computer applications course as part of their general education curriculum. This computer application course varies in subject matter from campus to campus, from a concepts-oriented course to a hands-on applications course. This paper studies the difference in knowledge self-assessment and grade estimation between sections that are exclusively for freshman Business Administration majors and sections that are composed of upper-class non-business majors. The null hypothesis is that there is no difference in knowledge assessment or grade estimation between the two groups as measured by a self-assessment instrument. This paper presents the statistical analysis and summary of the findings, and discusses implications of the research. Keywords: Information Systems Self-Assessment, Statistical difference in student groups I. Overview The course assessed in this paper is CIS 101 Introduction to Information Technology. The course is a combination of Microsoft Office applications and general computer concepts (storage, processing, terminology, telecommunications, ethics, privacy, etc.). The course is part of the general education core at Quinnipiac University. Two diverse groups take the course: first semester freshman Business Administration majors and upper-class non-business majors. It is a required first semester course for all students in the School of Business and taught utilizing laptop computers. During the spring semester, the course is taught in a regular computer lab and is opened to students outside the School of Business. Spring sections quickly fill with students from the College of Liberal Arts, and the Schools of Health Sciences and Mass Communications. These students are sophomores or juniors who bring more academic experiences to the class as compared to the freshmen students. There is also more diversity in academic background as these students major in a variety of disciplines, including psychology, English, education, criminal justice, nursing, and mass communications. The major units in the CIS 101 course are: (1) Introduction to computers and computing; (2) Introduction to software and operating systems; (3) Input and output; (4) Microsoft Office Suite consisting of Word, Excel, Outlook, PowerPoint, and FrontPage; (5) networking concepts, telecommunications, Internet usage, computer systems, and (6) Issues of privacy, ethics and security. In regards to the subject matter, many students come to the course with substantial experience and skills in some of the topics but lack experience and skills in others. For example, most students have some experience in using Microsoft Word, but have limited exposure to Excel or networking. Almost no students have web page development experience, but all have surfed the web and used e-mail and search engines. II. The research model The author administered a questionnaire on the first day of the course asking the students to rate their skills in Microsoft Word, Excel, Outlook, PowerPoint and Front Page. He also asked the students to estimate the final grade that they expected in the course. On the last class day of the course, a similar questionnaire was administered. There were 68 students in the two sections during fall semester 2001 and 48 students in the two sections during spring semester 2002. The two sections from fall semester were aggregated as one group as were the two sections from spring semester. The questionnaire is illustrated below: Please select an answer from 1 (low) to 5 (high) for each of the questions. ____ 1) My knowledge / experience with Microsoft Word ____ 2) My knowledge / experience with Microsoft Excel ____ 3) My knowledge / experience with Microsoft PowerPoint (presentations) ____ 4) My knowledge / experience with a web page generator (we will use Microsoft FrontPage) ____ 5) My knowledge / experience with e-mail (we will use Microsoft Outlook) Give your best estimate as to the grade you would expect in this class: _________ The intent of the study The intent of the study was to find if there were significant differences in terms of the students' self assessed skill levels, both at the beginning and ending of the course, and in the estimated grades and actual grades. In addition, the author wanted to determine if there were significant differences in the skills assessment and grade estimation between freshman business majors and upper-class non-business majors. The null hypothesis was that there would be no difference in knowledge assessment or grade estimation between the two groups as measured by the self-assessment instrument. If statistically significant differences were found, it might imply that the professor adapt the course to meet the needs of the students. A final consideration was for input to the professor to possibly modify the course with less emphasis on certain of the topics and more emphasis on others. The author anticipated growth in the students' perception and evaluation of their skills from the beginning to the end of the class. For example, a student might rate his or her Microsoft Excel skills at a level 2 on the first day, and rate them at a level 4 on the last day of class - indicating that student felt that he/she did learn and grow in Excel skills. Student Grade Expectations As a freshman (100) level course, the Computer Information Systems department has set a goal of a 3.0 overall grade average in the course. This campus has a full plus/minus grading scale: A, A minus, B plus, B, B minus, C plus, C, C minus, D and F. These are equated to grade averages as follows: an A is 4.00, an A minus is 3.66; a B plus is 3.33; a B is 3.00, a B minus is 2.66, a C+ is 2.33, a C is 2.00 a C minus is 1.66, a D is 1.00 and an F is 0.00. III Analysis of Data Overall Average Grade Expectations at the beginning of the course Based on the questionnaire "Give your best estimate as to the grade you would expect in this class", the average student grade expectation at the beginning of the course was as follows: Start of Semester Dominate Student Make-up Average Student Grade Expectation (on a four point scale) Fall Freshman Business Administration majors 3.78 Spring Upper-class non-business majors 3.63 The overall expected grade for incoming freshman Business majors was statistically significantly higher than the overall expected grade for upper-class non-business majors as determined by a binomial distribution with two-tail analysis at the .05 level of significance. This would negate the null hypothesis that the groups were no different in terms of grade estimation as measured by the self-assessment instrument. This was on the first day of class - and for the freshmen it was within the first class days of their college career. Overall Average Grade Expectations during the last class session (prior to the comprehensive final test) Based on the questionnaire "Give your best estimate as to the grade you would expect in this class", the average student grade expectation at the end of the course was as follows: End of Semester Dominate Student Make-up Average Student Grade Expectation Fall Freshman Business Administration majors 3.48 Spring Upper-class non-business majors 3.44 Although the freshmen business majors' self-assessment was still higher at the end of the semester, it was not statistically significant using binomial distribution analysis, and thus the null hypothesis that there would be no difference between the groups holds. Overall Actual Grade averages When comparing the actual grade averages, the research indicates: Final overall grade Dominate Student Make-up Average Student Grade Fall Freshman Business Administration majors 3.05 Spring Upper-class non-business majors 3.16 The difference in final overall grade averages was not statistically significant. Thus we can accept the null hypothesis that there was no difference in actual achievement in the course in terms of final grade outcome. Correlation Analyses "Correlation analysis is used to measure the strength of the association between numerical variables." [Levine 1999 page 772] For example, there should be a high correlation between height and weight as taller people generally weigh more as compared to shorter people. A correlation coefficient of 1.0 would indicate a perfect match between the sets of data, while a correlation coefficient of -1.0 would indicate a perfect inverse relationship; while a correlation coefficient around zero would indicate no measurable relationship exists. Correlation of initial grade estimate to final course grade Semester Dominate Student Make-up Correlation coefficient Fall Freshman business majors 0.14 Spring Upper-class non-business majors 0.17 The correlation coefficient in this table indicates a low or non-existent relationship between what students estimated they would get for a grade in the course on the first day of the class and the final grade they received. Correlation of final grade estimate to final course grade Semester Dominate Student Make-up Correlation coefficient Fall Freshman business majors 0.81 Spring Upper-class non-business majors 0.82 The correlation coefficient here indicates a moderately high relationship between what students estimated they would get for a grade in the course on the final class day (prior to the final test) and actual final grade. This would indicate that as the semester was coming to an end, the students had a stronger grasp of the grade they would receive. Analysis of hypothesis that there was no difference between the two audiences The author analyzed the anticipated grades in the course twice - on the first day of the course with the whole course still ahead and on the last regular class day of the course (prior to the final test). There was no significant difference in the grade anticipation on the last day, but there was a significant difference in the anticipated grades at the .01 level of significance on the first day of the course, with the freshman business majors evaluating their anticipated grades statistically significantly higher than returning students. Analysis It seemed to the author that students were either unrealistic or overly optimistic! From the fall semester class with primarily incoming freshmen students, the overall expected grade was a 3.78 (where an A minus would be 3.66). Even with just the final test to be completed, the freshman still had expectations of a 3.48 overall GPA in the class. The final overall GPA was 3.05. An old adage is "hope springs eternal", and these students were hopeful of gaining higher scores. Since the grade expectation difference was not significant at the end of the semester - either in the actual grade received or in the expected final grade prior to the final test, it may seem that the incoming freshman had some unrealistic expectations about the nature of college work when they entered. This may have implications to information systems faculty that teach sections with freshmen students to be aware of overly optimistic expectations (and possibly with the deflated egos that can occur when the reality of college sets in). Course grade correlation There was a very low correlation or no significance between the initial final grade estimate and the actual final grade in both semesters, which would indicate there was little basis for the expectation of final grade on the students' part. For the final grade estimate (although the students estimated higher graded than the actual grades) there was a fairly high correlation of the students' final estimate to the actual grade. This would indicate that although they were optimistic, they were fairly accurate in terms of lining up with the final grade. For the more experienced college students that took the course in the spring, their expectations were only slightly less - with a initial estimate of 3.63 overall GPA, and a final estimate during the last class period prior to the final test of 3.44. They did achieve a higher overall GPA of 3.16 - possibly related to better study habits for a comprehensive final test. The final test used was the same and was not returned to students after the semester, so copies of the test were not available to the spring group. The overall average of 3.16 was higher than the instructor desired, but was basically comparable between the semesters in terms of assignments and evaluation activities (tests, quizzes and papers). The author notes that the students did learn and achieve, but feels he can add more rigor to the course and to expect more from the students and that might impact the overall GPA. Achievement and perception in Microsoft Office areas Most students have widely used computers through their entire academic career. They most likely have taken a computer course in high school, as well as used office productive tools. The incoming freshmen students in the School of Business were required to purchase a standard laptop computer to be used in the program. This computer was specified to the students prior to arriving on campus. For the spring sections of this course, the instruction took place in a computer lab, although the overwhelming majority (approximately 80%) had their own computers in the student residences. In the Microsoft Office area, students were expected to learn Word, Excel, PowerPoint, Outlook and web page development with FrontPage. Microsoft Word Initial self-evaluation: 3.71; Ending self-evaluation 4.70 (on 5 points scale) The first unit in the applications section was Microsoft Word. The author expected that many students had some kind of background in word processing and many had a background in using Word. This is borne out by the research. Of the five office applications areas studied, overall Word had the highest initial student self-evaluation at 3.71 on a five-point scale. But, the concluding self-evaluation by the students came in at 4.70. On the self-evaluation sheet the students filled out, "5" was indicated as "mastery", and "4" as excellent, and "3" as very good. To the author, the growth from a strong 3.71 very good to excellent initial self-evaluation to a 4.70 close to mastery evaluation indicates that the students felt their skills in Word had improved. Overall 73.5% of the students evaluated their skills as higher at the end of the class; and if you consider those that evaluated their skills as the same, 95.9% of the students evaluated their skills as higher or the same. There were four students (all in the fall freshman sections) that indicated at the end of the class that their skills had decreased!!! Also as happened with the grade expectation analysis, the incoming freshman self-evaluated their skills as significantly higher (at the .01 level) as compared to the upper-class non-business students, but that this significant difference had evaporated by the end of the semester. Analysis Word was familiar to the students, but they still gained skills and knowledge. It is of interest to note that about 4% of the students ranked their ending skills as less than their beginning skills. This might be attributed to the concept that at the end of the course, with exposure to many advanced concepts in Word, that they may have realized that they didn't really know all that much about Word. In the comparison between fall (incoming freshmen) students and spring students (returning students), there were some changes as illustrated by this table: WORD Fall (freshmen / SOB) Spring (other majors) Increased skills 31 41 Skills the same 16 2 Decrease in skills 4 0 Initial skill assessment - on a five point scale - average 3.95 ( ** significant at .01 level) 3.35 Ending skill assessment - on five point scale - average 4.68 4.72 The freshmen had a much larger number of students (20 as compared to 2) who felt their skill level either remained the same or decreased. Although no specific question was asked that might illustrate the reasons for this, the author might attribute this to a stronger incoming skill set as compared to the returning students. It might also be that as incoming business majors, they had taken more courses where computer skills were emphasized in high school. It is also interesting to note the change in the average skill assessment was much greater in the spring classes with older students, but outside the school of business. Word was the only skill area where the spring class of non-business returning students had a higher ending evaluation as compared to the fall class of freshmen business majors. The implication to the author is that the time spend on Word might be lessen as students indicate both a significant beginning understanding as well as an ending mastery of the skills involved. Microsoft Excel: Initial self-evaluation: 2.13; Ending self-evaluation: 4.10 Excel was the second Microsoft Office topic taught. As compared to Word, it might be expected that students had less skills in Excel as compared to Word when starting the class. It might also be reasoned that business students might need more skills and have more interest in Excel because of some of the quantitative aspects and business aspects of Excel. Overall, 91.8% of the students indicated that their skills indicated that their skill level increased with Excel, and no students indicated a decreased in skills. The data in this table illustrate the self-assessment of skills as reported by the students. The spring sections with non-business, sophomores and junior students reported lower initial and ending skills, but also reported the largest growth among the two groups. EXCEL Fall (freshman SOB) Spring (other majors) Increased skills 44 45 Skills about the same 6 2 Decreased skills 0 0 Initial self-evaluation (5 point scale) - average 2.39 **significant at the .01 level 1.78 Ending self-evaluation 4.20 4.00 Overall skills grew from 2.13 to 4.10 - or from limited knowledge to excellence. The spring group with non-business majors who were also returning students showed the highest growth - possibly since they were not as experienced with Excel coming into the course. Microsoft PowerPoint. Initial evaluation: 2.21; Ending Evaluation: 4.50 PowerPoint is a handy program for making presentations. It is fairly easy to learn and do most common tasks with. The overall ending evaluation indicated a strong excellent to mastery self-assessment of skills. Analysis PowerPoint had the second overall highest growth in skills, (web development was first). In addition to standard presentations, the students had to also make a continuous slide show with timings, animations, and transitions. Again, the largest grown was by the non-business returning students, with a change of 2.36 from a limited knowledge skill level to an excellence to mastery level. There were no students that indicated a drop in skills. There was also no significant difference in self-evaluation either at the beginning or ending of the course. Here is the table corresponding to PowerPoint skills assessments: POWERPOINT Fall (freshman SOB) Spring (other majors) Increased skills 45 42 Skills about the same 5 5 Decreased skills 0 0 Initial self-evaluation - avg. 2.23 (not significant) 2.07 Ending self-evaluation - avg. 4.63 (not significant) 4.43 Microsoft Outlook E-mail: Initial evaluation: 3.51; Ending Evaluation: 4.76 Students tend to be familiar with e-mail. They use to correspond with friends and family. In this class, we used e-mail extensively with assignments, reminders, and more. Students had to attach and send files as well as open attachments. They used distribution lists and did other advanced features associated with electronic mail. While the initial self-assessment was slightly lower than Word (initial of 3.51 for E-mail as compared to 3.71 for Word overall), the ending self-assessment indicated that students had a stronger mastery of e-mail (4.76 for e-mail as compared to 4.70 for Word). There was no significant difference in assessment either at the beginning or ending of the course. Analysis The table for Outlook e-mail is below: Outlook - E-mail Fall (freshman SOB) Spring (other majors) Increased skills 37 35 Skills about the same 13 11 Decreased skills 1 1 Initial self evaluation - avg. 3.65 not significant 3.37 Ending self evaluation - avg. 4.88 not significant 4.68 Outlook e-mail had the lowest number indicating increased skills and the highest number of students indicating no growth (approximately 24%). Many students used at least two e-mail systems on a regular basis - the campus e-mail and an external e-mail like AOL e-mail or Hotmail (or similar Internet based e-mail system). There were two individuals that indicated their skills in e-mail had decreased over the length of the course. As a skill that has lifetime usage, e-mail mastery is important, and the assessment of the students indicated that they were approaching mastery. The author did not spend much time with Outlook instruction with most the growth occurring through frequently use of the product. It might be such that the author can delete any formal instruction in the use of Outlook and use that time savings for other material. Web page development with FrontPage: Initial evaluation: 1.65; Ending evaluation 4.07. There was statistically a significant difference in initial self-assessment (at .05 level), but no significant difference at the end of the course. Many students were unfamiliar with web pages and in particular with a web page development tool. The students were required to develop one web site with at least three pages. The main page was to be the students "home page" and the second page was an academically related page and the third was a fun page. The pages were all saved onto the campus student web server. Some students really got into the web page assignment, particularly with the fun page, including many pictures of family and friends as well as links to their favorite music, sports teams, TV's and movies. Others just did the minimum for the assignment. This was the last topic from the Microsoft Office applications. Some students came for extra assistance on this topic to develop significant pages and asking for additional information related to web pages for student clubs and groups or for their personal activities. One student developed a web page for his band and was researching web hosting services for the band by the end of the class. The table for Web Development with Front page is below: FrontPage Fall (freshman SOB) Spring (other majors) Increased skills 47 43 Skills about the same 2 3 Decreased skills 2 1 Initial self evaluation avg. 1.90 (** significant at .05 level) 1.39 Ending self evaluation - avg. 4.14 not significant 4.03 Web page creation with FrontPage had the highest skill growth of all the skill concepts studied. Looking at the spring class of non-business majors, their initial assessment was at the little or no knowledge to limited knowledge, while their ending assessment was at the excellent level (level "4", where level "5" was noted as "mastery"). The author sensed through the instruction and through the quantitative analysis that students did acquire new skills in this subject area. IV Hypothesis testing analysis summary To test the null hypothesis that there was no significant difference between the two groups, a binomial distribution was used. The results are shown below: Initial comparison Word - the groups were significantly different at the .01 level of significance in their initial self assessment of their skills and knowledge of Word, with the incoming freshmen business majors indicating a stronger grasp of Word Excel - again the groups were significantly different at the .01 level of significance, with the freshman showing the stronger skills PowerPoint - not significantly different Web Development - significantly different at the .05 level (freshmen stronger) E-Mail - not significantly different Grades - there was a significant difference in their perceptions of anticipated grades - with the freshmen again thinking they had stronger skills and a stronger self-assessment. Ending Comparison At the end of the semester, there were no significant differences. Analysis There were statistically significant differences between the two groups in initial self-assessment in four of the six areas: (1) grade expectation; (2) Word; (3) Excel and (4) Web Development knowledge. By the end of the semester, there were no statistically significance differences and any differences could be considered as minor or not statistically significant. The author interprets this as indicating that incoming freshmen coming to campus may have higher expectations and assessment of their skills, but by the end of the semester, the freshmen have adapted to college life and the differences are not significant. In terms of the other skills assessments, the author felt that, in general, the conclusions were consistent with initial expectations. As a course, the ending assessment of skills should indicate that students had grown in their skills and knowledge. In the five skills area evaluated on this questionnaire, all areas experienced significant growth in terms of student expressed self-assessment. In instructor assessment, students also experienced growth as measured by tests, assignments, quizzes and other evaluative activities. Of note to the instructor was that in all five areas assessed, the overall final average was over "4" on a five point scale, indicating a good sense of mastery of the subject material by the students. This study also implies to professors that freshmen students do come to campus with false expectations and that as professors, we need to be aware of these expectations and help students adapt to college level academics. The area that was the most interesting to the author was in the student estimate of grades. Overall, both the freshmen and the upperclassmen overestimated their grade expectations in the class. The first estimate on the first day might be attributed to optimism at starting a new class. The estimates of grades on the final class period (even after grades had been posted and e-mailed to students throughout the course and pre-final test grades had been posted and discussed as a motivating method prior to the final test) are surprising high to the author. The author also notes that in two of the areas (Word and Outlook) that the students indicated a strong initial self-assessment of their skills. This implies that less time should be spent on these topics. The study also gave the author a stronger sense of accomplishment. While as professors we think students have learned and that test scores seem to indicate that learning has occurred, this study with student self-assessment of skills indicated that the students did learn and in most of the software packages developed mastery of the skills. Areas for further study As the author concludes this study, he suggests that there might be additional studies into student self-assessment and achievement in the information systems discipline. He also notes the initial statistically significant differences between first semester freshmen and upper-class students. This study suggested that initial statistically significant differences in assessment and self-evaluation disappeared by the end of the first semester. Does this have implications for professors that teach freshmen level students to be more understanding of expectations at the start of their first academic year? Is this a normal part of the acclimation to college life? Is there a correlation between initial grade estimation and actual achievement? Would these results be similar with other instructors? Are these results (indicating overall final self-assessment above the "4" level) reasonably? Are these results higher than average or lower than average and are the results consistent with other instructors? Would students have similar grade estimations and expectations in a course in another discipline - say English or the required college algebra course or in other Information Systems courses? Bibliography Levine, David M. Mark L. Berenson, and David Stephan, Statistics for Managers using Microsoft Excel Prentice-Hall, 1999