Ambiguity Tolerance, Performance, Learning, and Satisfaction: A Research Direction William Owen1 and Robert Sweeney2 School of Computer and Information Sciences, University of South Alabama Mobile, AL 36688, USA Abstract This paper describes an effort to assess student tolerance for ambiguity in assignments and its effect on performance, learning, and satisfaction. It is part of a continuing research effort to discover and comprehend the relationship between ambiguity tolerance and learning. Many factors affect a student's ability to learn including those controlled by personality and cognitive characteristics. Tolerance for ambiguity has been previously explored as a possible mitigating variable in individual behavior. In this project, student's in an upper level course in Information Technology rated the ambiguity of projects completed during the class. Additionally, the student's tolerance for ambiguity was measured using two previously developed psychometric instruments and correlated with student ambiguity tolerance ratings for each project. An explanation is offered for the significant correlation found for one of the projects. Keywords: ambiguity tolerance project instructions satisfaction performance learning 1. INTRODUCTION Ambiguous situations are a fact of life in Information Technology. We encounter ambiguous specifications, problem statements, installation instructions and even technical documentation. Students studying computing also routinely encounter ambiguous situations. How students deal with or react to ambiguity can have a profound effect on their educational experience. This paper begins exploring the relationships between students, ambiguity, and learning. 2. LITERATURE REVIEW Definitions Webster's New Collegiate Dictionary defines ambiguous as "adj: 1. doubtful or uncertain, inexplicable; 2. Capable of being understood in two or more possible senses." Most often used to refer to situations or events, Budner (1962) offered three basic types: new situations, complex situations, and contradictory situations. Budner defined these types, respectively, as situations where cues are nonexistent or insufficient, where cues are too numerous, and where cues suggest contradictory structures. Norton (1975) found that psychologists have developed eight different categories that defined ambiguous. They include: 1) multiple meanings, 2) vagueness, incompleteness, or fragmented, 3) a probability, 4) unstructured, 5) lack of information, 6) uncertainty, 7) inconsistencies & contradictions, and 8) unclear. Many of these situations or categories are common in situations that occur in educational settings for computer and information science and technology. Ambiguity Tolerance Because ambiguity exists, and humans must cope with it; individuals display varying levels of tolerance to or intolerance of ambiguity or ambiguous situations. Frenkel-Brunswick indicated that intolerance for ambiguity was "a tendency to resort to black-and-white solutions, to arrive at premature closure, ..., often at the neglect of reality." (Frenkel-Brunswick 1949, p.115) Budner (1962) believed that intolerance for ambiguous situations are usually perceived as sources of threats. Jonassen and Grabowski (1993) conclude that tolerant individuals should perform well in new and complex learning situations. However, intolerant learners may tend to avoid or give up when encountering ambiguous situations. Many psychologists have attempted to explain or categorize individual ambiguity tolerance. Budner (1962), Norton (1975), Rydell & Rosen (1966), Macdonald (1970), Leavitt & Walton (1983), and McLain (1993) have attempted to study and develop instruments that quantify an individual's ambiguity tolerance. Numerous attempts have been made to examine the relationship between tolerance for ambiguity and other constructs including prejudicial attitudes, rational decision-making, perceptual psychology, and aptitude for second language acquisition. (Frenkel-Brunswik 1949; Elisberg 1961; Budner 1962; Chapelle & Roberts 1986) Relationship to computing education Situations where learning occurs often contain many opportunities for ambiguity. Beginning students routinely encounter novel and uncertain problems and explanations. Throughout the curriculum, the level of complexity increases while structure decreases. Consider a typical CS1 assignment: from the new student's perspective - everything is new and perhaps unexpected. In most cases, there is only one correct answer and the student seeks it. However, at the senior level, a student may be presented a problem with limited specifications and many possible acceptable solutions. These situations represent varying levels of ambiguity for the student. Each student will react differently according to his or her tolerance to ambiguity and learning style. Both current computing curricula (ACM 1991 & IS 2000) require that students be able to display mastery by completing a "real world" experience. The nature and scope of these projects should include some level of ambiguity as a challenge for the learner. In fact, some argue that without experiencing the negative effects of ambiguity, students have not adequately completed their education (Dawson 2000). Measuring Learner Characteristics There have been a number of attempts to develop a valid and reliable instrument to measure an individual's tolerance or intolerance for ambiguity. Early efforts to measure tolerance or intolerance for ambiguity included Frenkel-Brunswik (1949), Budner (1962), Ehrlich (1965), Rydell & Rosen (1966), Nutt (1988), MacDonald (1970), and McLain (1993). These instruments are self-reported measures developed from cognitive constructs. Most of them use either true/false or Likert scale responses containing both positive and negative items. Budner's Scale of Tolerance-Intolerance of Ambiguity provides the seminal work in this area (Budner, 1962). The 16-item Likert response instrument is not only reliable but the subject of much later work. Nutt's (1988) Scale of Tolerance/Intolerance Ambiguity is a modified version of Budner's (1962) instrument. The Nutt instrument is described by Daft and Marcic (2001) as follows: "This survey asks 15 questions about personal and work situations with ambiguity. You were asked to rate each situation on a scale of 1 to 7. A perfectly tolerant person would score 15 and a perfectly intolerant person 105. Scores ranging from 20 to 80 have been reported, with a mean of 45." Rydell & Rosen's 16-item, true-false instrument to measure ambiguity tolerance has been tested for and shown to have high construct validity. MacDonald (1970) modified Rydell & Rosen's instrument by adding four additional items: two from the California Personality Inventory and two from Barron's conformity scale to increase the reliability. Accordingly, MacDonald's AT-20 scale retains the high construct validity of its precursor while improving its reliability and internal consistency. McLain (1993) has developed the MSTAT-I scale by updating the cognitive constructs of prior scales. This 22-item Likert response instrument reports a .86 Alpha reliability and significant positive correlations with the Budner and MacDonald scales. 3. RESEARCH QUESTIONS The connection between ambiguity and learning raises many possible research questions. We have developed an initial set; however, the exploratory nature of this study may suggest other questions. Our initial questions focus on the relationships between the predictor variables of ambiguity tolerance levels and assignment ambiguity levels and the criterion variables of assignment performance, learning, and project satisfaction. Figure 1 below illustrates the hypothesized relationships between these variables. Ambiguity Tolerance Level and Assignment Ambiguity Level As discussed previously, there have been numerous attempts to measure an individual's tolerance or intolerance for ambiguity. We employed two separate measures of individual ambiguity tolerance in our study, the AT-20 and the Nutt Tolerance for Ambiguity scale. Two measures were used, as this was a preliminary research effort where one goal was the assessment of the applicability of a number of ambiguity scales. Assignment ambiguity is a topic that has not been extensively researched. In our study we assumed the level of assignment ambiguity was related to two factors. One factor was the length of each set of project instructions measured by the total number of words. The second factor was related to the readability of each of the project instructions as measured by the Flesch-Kincaid Grade Level score as calculated by Microsoft Word. The Flesch-Kincaid Grade Level score indicates the U.S. grade-level of a document where the lower the score, the easier it is to understand the document. The score is calculated from a formula that includes average sentence length and average number of syllables per word to rate documents. Assignment Performance When a learner is confronted with a situation that has a high level of ambiguity, his or her performance may be impaired. The impairment may manifest itself either as lower performance (grades) or as increased periods of time required to reach a desired performance level. Research Question: Is there a relationship between a student's ambiguity tolerance level, the assignment ambiguity level, and the performance on an assignment? Learning After completion of various learning activities (labs, homework, and projects), a student is assumed to have a deeper knowledge level because of the reinforcing nature of the activities. If the learner becomes frustrated by the ambiguity of a particular activity, learning may be impacted. An individual's tolerance to ambiguity may be related to learning. Research Question: Is there a relationship between a student's ambiguity tolerance level, the assignment ambiguity level, and learning? Project Satisfaction Students derive a level of satisfaction from completing assigned learning tasks. (Keller 1987, Bahlen & Ferrat 1993) Their satisfaction may be greater if the assignment had a lower perceived ambiguity level. Research Question: Is there a relationship between a student's ambiguity tolerance level, the assignment ambiguity level, and project satisfaction? 4. METHODOLOGY Subjects Our subjects were 16 students enrolled in a senior-level course in the Information Technology curriculum entitled Web Site Management. Students seeking either a Computer Science or Information Systems masters degree can take this course for graduate credit. Consequently, of the 16 subjects, 9 were graduate students. The course requirements include two group projects related to installation of operating system, web server, and related software on two different platforms. These projects will be referred to henceforth as 'NT project' and 'Linux project', respectively. The projects were required of all class members. Each project was worth 10% of the overall course grade for the class and each required approximately two weeks to complete. A group typically consisting of two to three members completed each project. The instructor attempted to assign both graduate and undergraduate students to each group and to have different group members for each project. The NT projects were all completed first, followed by the Linux projects. The Linux project instructions consisted of 8 pages and 3617 words, whereas the NT project instructions consisted of 4 pages and 1761 words. Furthermore, the Linux project instructions score for the Flesch-Kincaid Grade Level were a 7.7, while the NT project instructions were at a 10.0 grade level. The additional amount of instructions required for the Linux project as well as the fact that the Linux instructions were on a lower grade scale, indicating higher readability, as compared to the NT project instructions may have contributed to the fact that students with high ambiguity tolerance reported higher levels of NT project instruction ambiguity. Data Collection At the beginning of the Spring 2002 semester, we administered both the AT-20 and Nutt's Scale of Tolerance/Intolerance Ambiguity in an effort to explore possible instruments for ambiguity tolerance measurement. Additionally, after completing each project, students were asked to rate their perceived level of ambiguity for the project instructions using a 6-point Likert scale ranging from 'Very Unambiguous' to 'Very Ambiguous'. The students also rated their perceived level of satisfaction with the project and their own tolerance for ambiguity, using 6-point Likert scales. Correlations (Pearson's r), which are used to measure the relationship between variables, were then performed to compare the results from the AT-20 and of Nutt's Ambiguity Scale to the student's perceived level of ambiguity for each set of project instructions received. Students with a high tolerance for ambiguity, as indicated by a high score on the AT-20 or a low score on the Nutt Ambiguity Scale, were expected to perceive less ambiguity in the instructions they received for the projects. Furthermore, student's ambiguity scores were also correlated with the student's perceived level of satisfaction with each project performed. 5. RESULTS Pearson's correlation coefficient for the student's AT-20 score and the student's perceived NT project instructions level of ambiguity was calculated and the two variables were strongly correlated, r(14) = .658, p < .01. This result indicates that students with high tolerance for ambiguity perceived the instructions to be more ambiguous than did those students with lower tolerance for ambiguity. Comparing the student's AT-20 scores with the student's perceived level of ambiguity for the Linux project instructions indicated that the two variables were not strongly correlated, r(14) = .220, p > .05. From the student survey completed at the end of each project, the two variables relating to student satisfaction with the NT project and student satisfaction with the Linux project were strongly correlated, r(14) = .490, p < .05. Using a paired-sample comparison of means test, there was not a significant effect for student satisfaction scores, t(15) = -1.321, p > .05, between the NT and Linux projects. Consequently, as the two satisfaction scores were strongly correlated and not significantly different, it appears that students were equally satisfied with both projects. The relationship between ambiguity and student learning was not investigated in this study for several reasons. First, the final grades assigned to the projects did not exhibit a great deal of variability and tended to be high. The subjects for this class were about even divided between senior-level Information Science or Information Technology majors and graduate-level Information Science students. As a result, the quality of the projects and consequently the grades assigned to the projects were mostly As and a few Bs. This lack of variance in the project scores makes finding relationships with ambiguity problematic. Second, the learning attributed to the projects was not formally assessed as part of the class examinations. Consequently, using those scores, which did exhibit a great deal of variability, as a measure of student learning does not appear to be valid since they were not measures of the same learning concepts covered in the class projects. It may be possible to design assessment measures to assist in measuring this type of learning in the future which would be appropriate for a research project of this type. 6. CONCLUSIONS AND FURTHER RESEARCH As reported previously, the subject's tolerance for ambiguity, as measured using the AT-20, was strongly correlated with their perceived level of ambiguity of the NT project instructions but not with the perceived level of ambiguity of the Linux project instructions. Factors that may have impacted this result were the length and reading level of the project instructions, and the student's greater familiarity with one the operating systems, Windows NT, used in the project. A comparison of the two sets of instructions reveals that in terms of length the Linux project instructions were longer and more detailed than those for the NT project. Furthermore, the reading level of the Linux project instructions was lower than for the NT project. Additionally, students had less familiarity with the Linux operating systems and the software they installed in the Linux project. One aspect of ambiguity that has been previously explored is the individual's use of background knowledge to complete an assignment (Norton 1975). Consequently, we considered the Linux project instructions to have a lower level of ambiguity than the NT project due to the NT project's additional amount of instructional material included, its lower reading level, and the higher familiarity with background knowledge used to complete the assignment. However, we have been unable to find any instrument that quantifies the level of ambiguity in assignments. Further research plans include the plans to develop an instrument based on Budner (1962) and Norton's (1975) definitions of ambiguity. Our research was of an exploratory nature with several goals. One goal was to compare and evaluate different ambiguity tolerance instruments. Future research will include other instruments for measuring ambiguity tolerance levels. A second goal was to verify our experimental methodology in the classroom. The experiment we performed had little impact on normal classroom activity as the instruments we used were quick to administer and easy to score. The third goal of our research was to discover additional research variables, research questions, and assessment methodologies for possible inclusion in future research. Possible variables include student's learning styles, additional learner outcomes (i.e., performance on individual assignments and overall course performance), assignment ambiguity level, and improved measures of student satisfaction. While our findings are preliminary, this area of research is not only promising but may provide more insight into student learning. Controlling ambiguity can have a positive impact on learning if we better understand these relationships. One of the potential implications of this research includes that by reducing or eliminating ambiguity from certain learning situations we may be able to improve student learning and performance. Another possible implication is that an assessment of the ambiguity tolerance of individual students may be useful when assigning students to project groups. Depending on the learning objectives of the project, it may be useful to group students of similar ambiguity tolerance levels or to try and randomize the distribution of students of different tolerance levels. Finally, the controlled introduction of varying levels of ambiguity may improve each individual's ambiguity tolerance level or enhance the individual's ambiguity coping strategies. 7. REFERENCES The Association for Computing Machinery, 1991. ACM/IEEE-CS Joint Task Force, Computing Curriculum, 1991. Behlen, G. A., and T. W. Ferrat, 1993, "The effect of learning style and method of instruction on achievement, efficiency, and satisfaction of end-users learning computer software." Proceedings of Conference on Computer Personnel Research. Budner, S., 1962, "Intolerance of ambiguity as a personality variable." Journal of Personality. 30, 29-50. Daft, R. A. and D. Marcic, 2001, Understanding Management, 3rd Edition, Harcourt College Publishers. Web site: http://emerson.thomsonlearning.com/management/daft3um/student/guide.html, 5/15/02. Davis, G. B., J. T. Gorgone, J. D. Couger, D. L. Feinstein, and H. E. Longenecker, 1997, IS '97: Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems. Association of Information Technology Professionals. Dawson, R., 2000, "Twenty dirty tricks to train software engineers." ACM Proceedings of the 22nd International Conference on Software Engineering. Limerick, Ireland. Ehrlich, D., 1965. "Intolerance of ambiguity. Walk's A Scale: historical comment." Psychological Reports, 176, 591-594. Frenkel-Brunswik, E., 1949. "Intolerance of ambiguity as an emotional and perceptual personality variable." Journal of Personality. 18, 108-143. Jonassen, D. H. and B. L. Grabowski, 1993, Handbook of individual differences, learning, and instruction. Hillsdale, NJ: Lawrence Earlbaum Associates. Keller, J. M., 1987, "Strategies for stimulating the motivation to learn." Performance and Instruction, 26 (8), 1-7. Norton, R. , 1975, "Measurement of ambiguity tolerance." Journal of Personality Assessment. 39, 607-619. MacDonald, A. P. Jr., 1970, "Revised Scale for Ambiguity Tolerance: Reliability and Validity." Psychological Reports. 26, 791-798. McLain, D. L. , 1993, "The MSTAT-I: A new measure of an individual's tolerance for ambiguity." Educational and Psychological Measurement. 53, 183-189. Nutt, P. C., 1988, "The Tolerance for Ambiguity and Decision Making." The Ohio State University College of Business Working Paper Series, WP88-291, March 1988 Rydell, S. T. & E. Rosen, 1966, "Measurement and some correlates of need-cognition." Psychological Reports, 19, 1303-1312, Smith, D. M. & D. A. Kolb, 1986, User's guide for the learning style inventory: a manual for teachers and trainers. Boston, MA: McBer & Company. 1 owen@cis.usouthal.edu 2 sweeney@cis.usouthal.edu