Mechanism Design for Peer and Self Assessment
of a Group Project

Yanbin Tu
School of Business, University of Connecticut
Storrs, CT  06269, USA
yanbin.tu@business.uconn.edu

Min Lu
Department of Economics, University of British Columbia
Vancouver, BC V6T 1Z1, Canada
mlmlml@interchange.ubc.ca

Abstract

Teamwork is becoming more and more important in IS professions. A group project assignment is an effective method to train students’ skills in teamwork. To reduce the free-rider problem and treat each group member fairly, the instructor needs to distinguish each individual’s contribution to teamwork. In the paper, we analyze one commonly used peer-and-self assessment approach and point out its critical drawback: the deduced ranking might be wrong as some members do not tell the truth. Alternatively, we propose an effective mechanism to modify the peer-and-self assessment. The advantage of the revised peer-and-self assessment is that under the new mechanism, truth-telling is each individual’s dominant strategy. Therefore, by using the revised peer-and-self assessment, the instructor can effectively distinguish each member’s contribution to teamwork.

Keywords: peer-and-self assessment, teamwork, group project, game theory, mechanism design


1.  INTRODUCTION

Teamwork is becoming more and more important in IS professions (Pottert et al. 2000). Group project assignments have become an intrinsic part of coursework in information systems education (Steenkamp 2002). There are numerous benefits for students learning via team or group projects (Lejk et al. 1997; Lopez-Real et al. 1999). In information systems education, many courses such as introduction to information systems, system analysis and design, and software engineering require group projects as an important part of the course works (Turban et al. 2004).  A cooperative group project assignment provides a good experience for students to understand the relevant principles in the courses. As the important step for a group project, the instructor needs to evaluate it and assign final grades to each member in the group. The fair assessment of a group project is important because it helps to stimulate students to work hard in a group work (Leach et al. 2001). The usual practice is that the instructor reviews and evaluates a project, and assigns the same grade to all the members in a group. Although it is an easy way for the instructor to implement such an evaluation, there are some underlying drawbacks. First of all, there might be a free-rider problem in a group project (Bartlett 1995). Since all the students in a group will be assigned the same grade, the marginal efforts of one student will benefit all the people in the group. But for the contributor, his marginal gain is only part of the total gains derived from his efforts. On the other hand, the people without any efforts can get some gains from other contributor’s efforts. In this way, some members might have an incentive to be a free-rider, which leads to a low quality group work. Secondly, this evaluation method eliminates the difference among the people in a group. Admittedly, even though everyone in the team does his best, his contribution to the group project is still different from others’ because of his different background and intelligence. Conway et al. (1993) points out that students complain that awarding the same mark to all group members is often not a fair evaluation of individual effort. Thirdly, the instructor usually expects a “nice” distribution of grades in his class. Groups of students with the same grade can easily lead to grade clustering and might bring a “non-smooth” distribution.

Due to the above three drawbacks, the instructor needs to distinguish each individual’s different contributions in a group project. However, there is an asymmetric information problem. Although each member knows each other’s efforts in the group, for the instructor, each individual’s effort is private information. Generally, there are two approaches for the instructor to distinguish each individual’s contribution. The first approach is that the instructor conducts an investigation, such as reading the logbooks which show the sequential progresses of a group project and the detail descriptions of each member’s activities towards the project. However, he needs to spend extra time and energy for such investigation. The second approach is to ask the students to report the efforts of all people in a group, which we call a peer-and-self assessment. There are a number of studies about the peer assessment practices (Dochy et al. 1999; Sluijsmans et al. 1999; Falchikov et al. 2001; Sindre et al. 2003). Keaten et al. (1992) affirm that peer assessment fosters an appreciation for internal awards and interpersonal relationships in the classroom. So, if the peer-and-self assessment is valid, it is appealing to the instructor. One critical problem with this method is whether everyone tells the truth or not (Sindre et al. 2003). It is obvious that each group member has an incentive to exaggerate his own contribution to the teamwork during a peer-and-self assessment. Besides, does everyone have an incentive to tell the truth about other member’s contribution?

Therefore, we need a simple and reliable mechanism which allows the students to easily complete the peer-and-self assessment and also induces them to tell the truth. If we take the peer-and-self assessment as a game played by the people in a team, equivalently say, we need a simple game so that every player has “tell-the-truth” as his dominant strategy. Rafiq et al. (1996) trace one university's approaches of peer assessment to develop fair and reliable systems for a group project and use one model in the field of civil engineering. They show the relevance and drawbacks of the method of peer assessment devised by Goldfinch et al. (1990) and propose some new methods. Conway et al. (1993) examine ways in which students may be awarded individual marks, reflecting personal effort, for a group project. They also criticize the method by Goldfinch et al. (1990), and outline a simplified scheme for assessing the contribution of an individual to a group project. Reif et al. (2001) take advantage of IT and use a Web-based form to conduct student peer assessments for group works. This paper tries to investigate the same topic, but it is strikingly different from the existing general education and information systems education literature in three aspects. 1) Unlike the existing literature which assumes that everyone always tells the truth, this paper admits that the members in a group have an incentive to lie. 2) Unlike the existing literature using traditional research methodologies, this paper uses the game theoretic approach. 3) While the existing literature emphasizes the cardinal assessment, this paper focuses on the ordinal ranking assessment.

It is worthwhile to mention that although the peer-and-self assessment is rather “generic, in the sense that it can be widely used in many disciplines, we present it in the information systems field because teamwork is becoming more and more important in IS careers and group project assignments have become an intrinsic part of IS coursework (Pottert et al. 2000; Steenkamp 2002). We believe that the concept of teamwork is becoming especially relevant to IS educators and the IS field, and this study might help IS educators to conduct an effective group project assessment.

2.  PEER-AND-SELF ASSESSMENT

One peer-and-self assessment approach for a group project is conducted as follows. Each individual will be asked to give a ranking evaluation for the contributions of all the members in a group. The instructor collects the evaluation reports to deduce each individual’s contribution to the group project. This application can be illustrated in detail by the following example. Suppose three students, A, B and C, form a team to conduct a project. We assume that after finishing the project, these three students know very well the ranking of their contributions as A>B>C. The instructor asks them to give a peer-and-self assessment for the teamwork. Suppose each one exaggerates his contribution and always ranks himself as the highest contributor. After collecting the assessment reports, the instructor ignores the evaluator’s rank for himself, and deduces the implicit ranking for all the members in the group. It might happen that some students truly believe they do more than others but actually not. If this happens, ignoring the evaluator’s rank for himself does not matter. The reason is that in this peer-and-self assessment, if everyone tells the truth about others, the instructor can deduce the correct ranking for everyone. For example, from Report 1 in Table 1, A tells the true ranking; B and C exaggerate their rank respectively, but they tell the truth about others’ ranking. Now let us look at how the instructor deduces the group ranking. He eliminates A’s assessment for A, B’s assessment for B and C’s assessment for C. The following information remains: from A’s assessment, B>C, from B’s assessment, A>C and from C’s assessment, A>B. He can immediately construct the ranking for the group: A>B>C, which is the correct ranking. If the instructor uses this deduced ranking to evaluate each member’s contribution, every member is treated fairly.

However, Report 2 in Table 1 is a different story. In Report 2, A lies to tell others’ ranking, B and C exaggerate their rank respectively, but they tell the truth about others’ ranking. Now, let us check the deduced ranking that the instructor can get. As usual, he eliminates A’s assessment for A, B’s assessment for B and C’s assessment for C. The following information remains: from A’s assessment, C>B, from B’s assessment, A>C, and from C’s assessment, A>B. He can immediately construct the ranking for the group: A>C>B, which is a wrong ranking. It is easy to see that A is indifferent, but C benefits (gets promoted), and B costs (gets demoted) from Report 2. Note the instructor will not know who lied in Report 2 unless B complains about the wrong ranking.

Table 1. The Possible Peer-and-Self Assessment Reports
(Ranking is vertically arranged)
Report
A
B
C
People Lie
Deduced Ranking
People Benefit From Report 
People Cost
From the Report
1
A
B
C
B
A
C
C
A
B
None

A
B
C
None
None
2
A
C
B
B
A
C
C
A
B
A
A
C
B
C
B
3
A
B
C
B
C
A
C
A
B
B
No Ranking

B due to punishment
4
A
B
C
B
A
C
C
B
A
C
B
A
C
B
C
5
A
C
B
B
C
A
C
A
B
A,B
C
A
B
C
A,B
6
A
C
B
B
A
C
C
B
A
A,C
No Ranking

A,C due to punishment
7
A
B
C
B
C
A
C
B
A
B,C
B
C
A
B,C
A
8
A
C
B
B
C
A
C
B
A
A,B,C
C
B
A
C
A
Recall we assume that each member’s contribution to the project is strictly different as A>B>C, which means no ties exist. If the deduced ranking from any report is a cycle, i.e., no sensible ranking can be gained from the report, the instructor will realize someone must have lied in the report. Report 3 gives us such an example. After the instructor eliminates A’s assessment for A, B’s assessment for B and C’s assessment for C, the following information remains: from A’s assessment, B>C, from B’s assessment, C>A, and from C’s assessment, A>B, which lead to the cycle A>B>C>A, which is absolutely invalid. In this case, the instructor realizes someone have lied and investigates the logbooks to find and punish the liar (in this example the liar is B). We outline the eight possible peer-and-self assessment reports and the associated liar(s), deduced rankings, people benefit or cost from the reports in Table 1.

Now, let us look at A’s strategy under different assessment reports. That is, which strategy is dominant for A: tell the truth or lie? We extract A’s payoffs under the eight reports in Table 1 and create Table 2 as the payoff matrix for A. From the table, we see “to lie” is weakly dominated by “tell the truth”. Therefore, we can say that A will choose to tell the truth in the assessment process. This choice is reasonable to A because he is the top contributor in the group and has no incentive to tell a lie about himself or about others as he cannot get any extra benefits.

Table 2. The Payoff Matrix for A

B lies
C lies
B and C lie
Neither B or C lies
A lies
Cost
Cost
Cost
Indifferent
A is truthful
Indifferent
Cost
Cost
Indifferent
Therefore, we can delete the reports in which A lies, i.e., reports 2, 5, 6 and 8 in Table 1, only reports 1, 3, 4 and 7 remaining in Table 1. Next, let us look at what kind of strategies that B and C will take. In the same token, we focus on B’s and C’s payoffs in the remaining 4 reports and outline the payoff matrix for B and C in Table 3. The first element in parenthesis is B’s payoff and the second element is C’s payoff.

Table 3. The Payoff Matrix for B and C

C is truthful
C lies
B is truthful
(Indifferent, Indifferent)
(Benefit, Indifferent)
B lies
(Cost, Indifferent)
(Benefit, Benefit)
It is obvious that neither B nor C has a strictly dominant strategy. For B, given that C tells the truth, he also chooses to tell the truth, but given that C lies, B is indifferent between truth-telling and lying. For C, given that B tells the truth, C is indifferent between truth-telling and lying, but given that B lies, C will also lie. We can see there are three Nash equilibriums in B’s and C’s strategies. That is, both B and C are either truthful, or lie at the same time, or C lies and B tells the truth. Strikingly, the strategy that both lie leads to the Pareto improvement compared to the other two sets of strategies. As we can see, under the conditions that both B and C lie and A tells the truth, the ranking becomes B>C>A, where both B and C are promoted. So, there are some incentives for both B and C to collude with each other.

So far, we can see that this simple peer-and-self assessment application has one critical drawback, that is, the deduced ranking of group members’ contributions might be wrong. Under the Nash equilibrium where both B and C are truthful, that is, everyone tells the truth, the instructor can easily obtain the correct contribution ranking of A>B>C. Under the Nash equilibrium where both B and C lie, he obtains the wrong ranking of B>C>A. Under the Nash equilibrium that C lies and B tells the truth, he obtains the misleading ranking of B>A>C. We need to modify the game mechanism so that everyone chooses to tell the truth. Theoretically, we can either reward the truth-teller (called the “optimistic” approach) or punish the liar (called the “pessimistic” approach) allowing a liar to switch to telling the truth, and letting the truth-teller remains truthful. Here, for simplicity, we modify the mechanism by using threat of potential penalty. Let us put two additional rules to the game. 1) After the peer-and-self assessment, the instructor will declare the final grades to all the members in a group, and allow them to complain about the final grade ranking. That is, everyone knows not only his grade but also all the others’ grades in the group. This rule is reasonable because if the instructor does not distinguish each individual’s contribution, everyone gets the same grade, and this also means that everyone in a group knows each other’s grades. It is obvious that if the ranking of final grades for the group members is A>B>C, everyone receives fair treatment. Under this result, there might be no one to complain about the final result, but we cannot exclude that B or C will purposely complain about it. If the ranking of final grades is B>C>A, or B>A>C, A will not be satisfied with the result. He will definitely complain that someone lies or there is collusion between B and C. So, complaint of A is a signal of a wrong ranking to the instructor. If we set up the penalty rule at advance, and let everyone know it before they submit their assessment reports, we can correct their misbehavior of either be lying or purposely complaining about the correct ranking. Say we add another rule. 2) If someone complains about the final grade ranking, the instructor will investigate the group project. If he finds the ranking from the peer-and-self assessment is wrong, he will punish anyone who lies by demoting his final grade. If he finds that the ranking from the peer-and-self assessment is correct, he will punish the complainer in the same way. For example, if both B and C collude and A complains for the final grade, the instructor investigates the teamwork, and he will assign the final grade like A>>B>C, where >> means that B’s and C’s final grades are largely behind of A’s. This rule also prevents B or C from complaining when they are treated fairly under the correct ranking A>B>C.

Under these two additional rules, we can change the payoff matrix for B and C. We adjust their payoffs so that both B and C deviate from the strategy of “lie”.  The new payoff matrix is listed in Table 4.

Table 4. The Payoff Matrix for B and C

C is truthful
C lies
B is truthful
(Indifferent, Indifferent)
(Benefit, Cost)
B lies
(Cost, Indifferent)
(Cost, Cost)
The new payoff matrix brings the two significant changes. Firstly, both B and C have a strict dominant strategy now. For B, no matter what kind of strategy C will take, he always chooses to tell the truth. For C, no matter what kind of strategy B will take, he always chooses to tell the truth also. Secondly, there is only one Nash equilibrium, that is, both B and C tell the truth. So, the modified mechanism eliminates the possibility of assessment reports where someone lies. Under the new game, only report 1 in Table 1 is the outcome of the peer-and-self assessment. The instructor does not need to spend time to investigate the group project, but he can distinguish the contribution ranking of group members, because the participants will tell the truth and the deduced ranking is the true ranking, that is, A>B>C.

3.  CONCLUSIONS

Group work assignments play an important role to train students’ teamwork skills in information systems education. To reduce the free-rider problem and treat each group member fairly, the instructor needs to distinguish each individual’s contribution to a group project. However, there is an asymmetric information problem to the instructor. In this paper, we analyze one commonly used peer-and-self assessment application and point out its critical drawback: the deduced ranking might be wrong as some members do not tell the truth. Alternatively, we offer an effective mechanism to modify the peer-and-self assessment. The advantage of the revised peer-and-self assessment is that under the new mechanism, truth-telling is each individual’s dominant strategy. Therefore, by using the revised peer-and-self assessment, the instructor can effectively distinguish each member’s contribution to a group work.

4.  ACKNOWLEDGEMENTS

This paper is based on the theoretical analysis part of our peer-and-self assessment research project which will appear in Journal of Information Systems Education (JISE). We are grateful to both ISECON 2004 and JISE anonymous referees for their constructive comments. Particularly, we are grateful to one JISE anonymous referee who introduced ISECON to us and encouraged us to present this paper in ISECON 2004.

5.  REFERENCES

Bartlett, R. L., 1995, “A flip of the Coin – A Roll of the Die: An Answer to the Free-Rider Problem in Economic Instruction”, Journal of Economic Education, Vol. 26, No. 2, pp. 131-139

Conway, R. and D. Kember, 1993, “Peer Assessment of an Individual's Contribution to a Group Project”, Assessment & Evaluation in Higher Education, Vol. 18, No. 1, pp. 45-57

Dochy, F., M. Segers and D. Sluijsmans, 1999, “The Use of Self-, Peer-, and Co-Assessment in Higher Education: a Review”, Studies in Higher Education, Vol. 24, No. 3, pp. 331–350.

Falchikov, N. and J. Goldfinch, 2001, “Student Peer Assessment in Higher Education: a Meta-analysis Comparing Peer and Teacher Marks”, Review of Educational Research, Vol. 70, No. 3, pp. 287–322.

Goldfinch, J. M. and R. Raeside, 1990, “Development of Peer Assessment Technique for Obtaining Individual Marks on a Group Project”, Assessment and Evaluation in Higher Education, Vol. 15, No. 3, pp. 210-225. 

Keaten, J. A. and M. E. Richardson, 1992, “A Field Investigation of Peer Assessment as Part of the Student Group Grading Process”, Paper presented at the Western Speech Communication Association Convention, Albuquerque, NM

Leach, L., G. Neutze and N. Zepke, 2001, “Assessment and Empowerment: Some Critical Questions”, Assessment & Evaluation in Higher Education, Vol. 26, No. 4, pp. 293-305

Lejk, M. and M. Wyvill, 1997, “Group Learning and Group Assessment on Undergraduate Computing Courses in Higher Education in the UK: Results of a Survey”, Assessment & Evaluation in Higher Education, Vol. 22, No. 1, pp. 81-92

Lopez-Real, F. and Y. R. Chan, 1999, “Peer Assessment of a Group Project in a Primary Mathematics Education Course”, Assessment and Evaluation in Higher Education, Vol. 24, No. 1, pp. 67-79

Pottert, R.E., P. Balthazardtt and K. Lee Elder, 2000, “Delivering the Cross cultural Virtual Teamwork Experience: An Example with Mexican and American MBA Team Members”, Journal of Information Systems Education, Vol.11, No.3-4, pp. 111-220

Rafiq, Y. and H. Fullerton, 1996, “Peer Assessment of Group Projects in Civil Engineering”,  Assessment & Evaluation in Higher Education, Vol. 21, No. 1, pp. 83-91

Reif, H. and S. E. Kruck, 2002, “Integrating Student Groupwork Rating into Student Course Grades”, Journal of Information Systems Education, Vol.12, No.2, pp.57-63

Sindre, G., D. L. Moody, T. Brasethvik and A. Solvberg, 2003. “Introducing Peer Review in an IS Analysis Course”, Journal of Information Systems Education, Vol. 14, No.1, pp.101-119 

Sluijsmans, D., F. Dochy and  G. Moerkerke, 1999, “Creating a Learning Environment by Using Self-Peer -and Co-Assessment”, Learning Environments Research, Vol. 1, No. 3, pp. 293–319.

Steenkamp, A. L., 2002, “A standards-based Approach to Team-based Student Projects in an Information Technology Curriculum”, Proceedings of the 17th Annual Conference of the International Academy for Information Management, pp.54-62

Turban, E., E. McLean and J. Wetherbe, 2004, “Information Technology for Management”, John Wiley & Sons, Inc