PEERS: Peer Evaluation and Effective Review Samples Cindy Meyer Hanchey, Ph.D. Associate Professor of Computer Science, Oklahoma Baptist University Shawnee, Oklahoma 74804 United States of America Abstract Peer review is common not only in education but also in most professional areas. Many faculty shy away from using peer evaluations by students due to a lack of expertise in the creation and administration of evaluative materials. There does not appear to be a single, comprehensive collection of sample evaluations to which faculty can go to peruse materials. This paper addresses that issue and provides bibliographic references, samples of evaluation materials, and web sites for additional resources. Keywords: peer grading , peer review, peer evaluation, rubric, self-assessment 1. LITERATURE REVIEW Peer assessment and self-assessment by students should be an important component of any course in which team projects comprise a component. As more and more courses incorporate team projects, there is a need to provide for peer evaluations to assist in the determination of both individual and group grades. According to Trytten (Nov. 1999) the peer evaluation component must be a significant factor in the final grade in order to be meaningful. Reif and Kruck (2001) describe a process that incorporates the use of an anonymous web-based form for data collection. An alternative to the anonymous evaluation is used by Vaidyanathan (Spring 2002). It is apparent that peer assessment is becoming important in other, more traditional, computer science courses as well (Null, L., et. al. Feb.-Mar. 2002) (Michael, M., March 2002). The consistent theme in using peer evaluations of programs, for example, is to provide the criteria by which the program and the evaluations will be judged. Another area in which peer assessment can be beneficial is that of oral presentations. Business courses in particular have increased the emphasis on oral and written communication skills. Specific rubrics can be applied for oral presentations (Moon, J., May 29, 2001). One author proposes that student self and peer assessment is a means for teaching professional responsibility (Zariski, A., March 8, 1996). While the appendix (Criteria and Standards for Assessment of Individual and Group Projects) included in the article is targeted toward the legal profession, it provides an excellent rubric to show how to evaluate and distinguish different levels of student performance. This same theme of professionalism appears in the ITiCSE '99 working group report. In particular, the professional areas included Interpersonal Relationships with oral and written communications and group work as two of the sub-areas which the computing and information technology curriculum must address. For peer assessment to work effectively, students must be provided guidelines (Peer Review Groups: A Quick Introduction). There were some course-specific forms available in Sytsma. A well-designed evaluation form can be found in Heiman (Spring 2002). Sample rubrics for the grading and evaluation processes can be viewed at Twiss (Spring 1997) and at the Center for Teaching Excellence web site. It is important for a faculty member to determine what guidelines students will use prior to the evaluation process and to inform students of those guidelines. 2. OVERVIEW Most courses have multiple components for which grades are, or could be, assigned. For example, some typical components might include the following: --homework --quizzes --exams --group project --group presentation of final project The Visual Programming course at Oklahoma Baptist University has all of the items described above as well as a peer teaching component-all of which contribute to the student's final grade: While the exams and quizzes do not have a peer evaluation grade, all of the other pieces do. For example, the homework takes the form of a Visual Basic assignment. Students grade each other's homework. They receive a score based on the peer evaluation and another score based on how complete an evaluation they did of someone else. There is an interesting learning pattern that evolves from this process-students discover new and innovative methods for solving problems as well as discover what not to emulate in their solutions. Other courses at Oklahoma Baptist University have segments in which peer evaluations are included in the grading process. In the section that follows, sample evaluation forms are provided. To be effective, the faculty member must first determine what is important in the evaluation process-in other words, a rubric is created. Next, a form is designed to solicit the evaluation. Effective evaluation forms are a continuous work-in-progress. As a faculty member who has used peer evaluations over many years, it becomes apparent that change is inevitable in the evaluative process just as it is in the curriculum itself! 3. SAMPLES Each sample included as an appendix is currently in use by this author. Depending on the course being taught, the appropriate forms are used. Each is described below with the corresponding appendix referenced. In the majority of the processes described, the students performing the evaluations are given a grade based on how well they performed the evaluation. This tends to promote more honest and reliable evaluations. Whether the student is "teaching" a lesson or doing some other kind of presentation, the form found in Appendix A is used. A variation of this form might be employed if the intention is also to solicit information regarding material learned. One could add the question "What were the two most significant points?" This definitely provides focus by the audience on the material being presented. The students are told that any score of 3 or below must include specific comments as to why such a score was given and what the speaker might do to correct that score. Additionally, students are told that there is always room for improvement and, therefore, something must be written under the heading "suggestions for improvement". While this form results in a maximum of 25 points for a presentation, the numbers can be changed or multiplied by a percentage in order to fit with the instructor's grading scheme. In a course in which a web site is created as a final project, students critique each web site-even the one for which they served as a team member. Appendix B illustrates the form used for that type of evaluation. Teams pose interesting grading problems. Who did what? Peer evaluations help in determining some of that nebulous information. This author uses the form in Appendix C for courses in which team projects comprise a component. Additional evaluation forms will be provided at the presentation of this paper. These are forms used by faculty members in the School of Business at Oklahoma Baptist University. While the forms are designed for specific business courses, they can provide information for conference participants as to other considerations one might give to the peer evaluation process. 4. RECOMMENDATIONS It is the strong belief of this author that peer evaluation works. It does not, and should not, take the place of the faculty determination of final grades; however, it provides valuable input into that process. Peer evaluation also gives students insight into the evaluation process and experiences which will be useful in their professional careers. While none of the samples provided in this paper will fit everyone, it is the hope of this author that the bibliography and resource materials will lead readers to useful materials. 5. BIBLIOGRAPHY Bulman, T. (Sept. 1996). "Peer Assessment in Group Work." Faculty Focus, v1, n1, newsletter by Portland State University. [Online 5/15/02]. http://www.oaa.pdx.edu/CAE/FacultyFocus/spring96/bulman.html Burgess, H. "Self and Peer Assessment with material adapted from the SAPHE Project (Self Assessment in Professional and Higher Education Project)." Social Policy and Social Work Learning and Teaching Support Network, University of Southampton. [Online 5/15/02]. http://www.swap.ac.uk/approaches/Assessment2.asp Center for Teaching Excellence. St. Edward's University. Online resources (including Students' Self and Peer Evaluations which was accessed on 4/25/00). [Online 5/21/02] http://www.stedwards.edu/cte/resources/resourcesA-Z.htm Dyck, B. (May 10, 2002). "Peer Assessment Teaches Students How to Think." Education World. [Online 5/15/02]. http://www.education-world.com/a_issues/issues314.shtml Greenan, K. (Dec. 18, 2002). "Groupwork and Peer Assessment." Business Education Support Team: Scotland and Northern Ireland, Glasgow Workshop. [Online 5/15/02]. http://cbs1.gcal.ac.uk/lts/PP_Files/Dec2001/K_Greenan_PP.ppt Heiman, S. (Spring 2002). "Peer Evaluations." University of Missouri-Columbia School of Journalism course titled Strategic Communication Campaigns. [Online 5/15/02]. http://www.missouri.edu/~journsh/Peer_Evaluations/peer_evaluations.htm (Note-the forms used were taken from "Group personality and performance: A model for managing advertising student teams" by Bichard, S., Roberts, M. and Sutherland, J. contained in Fall 2000 Journal of Advertising Education) "ITiCSE '99 Working Group on Professionalism." (Dec. 1999). Integrating Professionalism and Workplace Issues into the Computing and Information Technology Curriculum. In ITiCSE '99 Working Group Reports, v31, n4. Kirkpatrick, D. and R. Fuller. (1995). "The Challenge of Peer Assessment." In Summers, L. (Ed), A Focus on Learning. Proceedings of the 4th Annual Teaching Learning Forum. [Online 4/25/00]. http://wwwasu.murdoch.edu.au/asu/pubs/tlf/tlf95/kirkp146.html McLeod, A. (June 1999). "In Lieu of Tests." In Assessing Learning by NTLF. [Online 4/25/00]. http://www.ntlf.com/html/lib/faq/al-ntlf.htm Michael, M. (March 2002). "Using Peer Evaluation of Programs to Develop Analytical and Communication Skills." In The Journal of Computing in Small Colleges, v17, n4. Moon, J. (May 29, 2001). "Peer and Self-assessment in Student Work." [Online 5/15/02]. http://www.ex.ac.uk/admin/academic/tls/tqa/peerassess1.htm Null, L., et. al. (Feb.-Mar. 2002). "Rewrite Cycles in CS Courses: Experience Reports." In SIGCSE Bulletin, v34, n1. Oral Presentations. Materials by Iowa State University. [Online 5/21/02]. http://www.agcomm.iastate.edu/strategies/oralpres.html "Peer Evaluations of Your Group Members." (Feb. 18, 2001). Materials for Software Requirements and Specification-3 course sequence at University of Waterloo. [Online 5/15/02]. http://www.student.math.uwaterloo.ca/~cs445/project/groupEvals.shtml "Peer Review Groups, A Quick Introduction." Prepared by the University Writing Program at Virginia Tech. Modification of original by Schell, E. [Online 5/21/02]. http://www.agcomm.iastate.edu/strategies/vtech1.html Reif, H. L. and S. E. Kruck. (2001). "Integrating Student Groupwork Ratings Into Student Course Grades." Journal of Information Systems Education, v12, n2. Sytsma, S. "Project and Team Evaluation Process (CISM 700)." Ferris State University. [Online 5/15/02]. http://www.sytsma.com/cism700/course-eval.html Telesford, E. (May 7, 2000). "Any Thoughts on Peer Evaluations?" Bulletin board posting for University of Iowa, College of Business, Managing People and Organizations. [Online 5/15/02]. http://www.biz.uiowa.edu/class/6n212_kristof-brown/messages/882.html Topping, K. (Fall 1998). "Peer Assessment Between Students in Colleges and Universities." Review of Educational Research, v68, n3. [Online 5/15/02]. http://www.aera.net/pubs/rer/abs/rer683-1.htm Trytten, D. (Nov. 1999). "Progressing from Small Group Work to Cooperative Learning: A Case Study from Computer Science." In Proceedings of 1999 Frontiers of Education Conference. [Online 5/21/02]. http://fie.engrng.pitt.edu/fie99/ Twiss, R. J. (Spring 1997). Materials for course titled Science and Pseudo-Science as part of Davis Honors Challenge, University of California Davis. [Online 5/15/02]. Notes on Team Management http://www-geology.ucdavis.edu/~hnr094/Team_Management.html Evaluation for Self and Team Members http://www-geology.ucdavis.edu/~hnr094/HNR_094_PeerEval.html Peer Evaluation of Team Presentations http://www-geology.ucdavis.edu/~hnr094/HNR_094_Presnt_Eval.html Vaidyanathan, R. (Spring 2002). Materials for courses taught at University of Minnesota Duluth. [Online 5/15/02]. Peer Evaluation Q&A. http://sbe.d.umn.edu/mgts4731/peer_ass.htm Peer Evaluation Worksheet. http://sbe.d.umn.edu/mgts4731/peereval.htm Zariski, A. (March 8,1996). "Self and Peer Assessment as a Means of Teaching Professional Responsibility." Sydney University Workshop titled "Are There Innovative Ways to Teach Professional Responsibility?". [Online 5/15/02]. http://carmen.murdoch.edu.au/~zariski/peer2.html Zariski, A. (1996). "Student Peer Assessment in Tertiary Education: Promise, Perils and Practice." Published in Abbott and Willcoxson, ed, Teaching and Learning Within and Across Disciplines, Perth: Murdoch University. [Online 5/15/02]. http://carmen.murdoch.edu.au/~zariski/peer1.html (Note-this one has an excellent bibliography) APPENDIX A ORAL PRESENTATION PEER EVALUATION Speaker _____________________________ Date __________ Topic _____________________________ Evaluate the items that follow using the scale below: 5 Excellent, Very Well Done 4 Good, Above Average 3 Average 2 Poor, Below Average 1 Terrible Speaker Poise (calm, not flustered, responded well to questions) __________ Mannerisms (speech, appearance, tone of voice) __________ Presentation Organization (prepared, logically presented) __________ Clarity (used examples and/or visuals, easy to understand) __________ Interest Level (maintained attention of audience) __________ Total Points __________ Suggestions for Improvement Evaluated by _______________________ APPENDIX B WEB SITE PROJECT PEER EVALUATION Project ____________________________ Date __________ Consider all of what you've learned and discussed during the course of the semester and evaluate the web site created as the final project for this course. Please comment on the positive features as well as the negatives (or missing) features. Be honest! Evaluated by __________________ APPENDIX C PROJECT PEER EVALUATION Congratulations! Your user is so pleased with your project being completed that she is awarding your team a bonus of $10,000. Your group leader is asking each team member to provide input before deciding on how to divide the bonus among team members. Your recommendation should be based on actual team member performance. Please provide commentary indicating why a team member should or should not receive any portion of the bonus. Things such as attended/did not attend team meetings, met/did not meet deadlines, etc. should be included in the commentary. Your total award does not have to total $10,000 but cannot go over that amount. Remember, this should reflect actual performance. Circle your name below. Project ____________________________ Members Bonus