Categorizing ambiguity in assignments: A pilot study Robert Sweeney, Assistant Professor1 & William Owen, Assistant Professor2 School of Computer and Information Sciences University of South Alabama Mobile, AL 36688 Abstract This paper reports the preliminary results of the development of a scale for rating the ambiguity level of computing assignments. Ambiguity is a constant factor in the educational landscape. Students are regularly exposed to ambiguous situations (instructions, test questions, problem specifications, etc.) and must deal with them according to their individual tolerance to ambiguity. The ability to rate assignments and produce a meaningful assessment of their ambiguity is helpful in understanding the relationships and impact of ambiguity on student learning. This research is part of an ongoing effort to understand these relationships, control them, and consequently improve student learning. Keywords: ambiguity, ambiguity tolerance, ambiguity evaluation, learning objectives 1. INTRODUCTION Ambiguity is common; humans deal with it, ignore it, or succumb to it. This paper outlines a method for classifying the level of ambiguity in an assignment and is part of an ongoing research project to explore and understand the relationship between tolerance to ambiguity and learning. Students react to many situations while learning. Many of their actions and perceptions are controlled by various personality and cognitive traits. A positive or negative reaction to a particular learning situation will be reflected in their performance and learning. 2. LITERATURE REVIEW Definitions Ambiguity or an ambiguous situation is generally considered to be a condition where there is insufficient information for the particular situation. Individuals perceive and react to these situations with varying degrees of tolerance or intolerance (Budner 1962; Norton 1975; McLain 1993). Ambiguity research has been conducted from the early 1950's. Early research centered upon psychological aspects and their understanding. The next focus of research was the development of instruments that could measure or define an individual=s tolerance to ambiguity. Budner (1962), Norton (1975), McLain (1993) and others have developed various self reported surveys to quantify this characteristic. Little work seems to have been done in classifying and measuring the ambiguity of situations. These situations arise normally in everyday life and humans apply their own individual coping strategies to them. However, in learning situations (homework assignments, lab instructions, problems specifications, etc.), it is possible to have varying levels of ambiguity. Budner pointed out three situations that may be associated with ambiguity. They are: 1) "A completely new situation in which there are no familiar cues," 2) "A complex situation in which there are a great number of cues to be taken into account," 3) "A contradictory situation in which different elements or cues suggest different structures." (Budner 1962, p.30). After reviewing the literature, Norton (1975) reported eight different categories that have been used to define the meaning of ambiguous. They include: 1) multiple meanings: there are at least two meanings whether the person is aware or unaware of them, or the meanings are clear or unclear, 2) vagueness, incompleteness, or fragmented: parts of the whole are missing, 3) a probability: the situation can be analyzed as a function of some probability, 4) unstructured: the situation has no apparent organization, 5) lack of information: the situation has little or no information, 6) uncertainty: a state of uncertainty is created in the mind of the person, 7) inconsistencies & contradictions: a situation in which portions of the information appear to disagree with each other, and 8) unclear: any situation perceived as unclear (Note: It is common for authors to use ambiguous and unclear as synonyms) . Many of Norton's categories can be found in common computing education situations. In studying the effect of ambiguity on learning, it would be helpful to have a classification measure that quantifies the level of ambiguity in learning activities. Learning environments Are ambiguous situations always bad? The simple answer is "it depends." Consider a learner's progression through a computing curriculum. In the beginning the assignments and tasks are simple and structured; at the end, they are complex and unstructured. Many educators believe that initial learning is increased by removing complexity and focusing upon the essential concepts. Simple assignments provide such an opportunity. A typical introductory course has many assignments each designed to: cover one or two objectives, be narrowly focused, be very structured, be completed in a short amount of time, and be clear. At the senior level, there are usually fewer but more complex assignments. These assignments may: extend for one or more academic terms, be highly unstructured, and have many acceptable solutions. Denning (1998) pointed out that computing education should strive to ensure that students experience "real world" situations and apply best IT practices to the solutions. These learning activities are considered to be ambiguous. Information technology practice is full of ambiguous situations. Practitioners deal with them daily. It seems reasonable that we should include similar situations in the more controlled learning environment. Opportunities for ambiguity arise in: project assignments and specifications, research projects, laboratory specifications, and even examination questions. 3. EVALUATING ASSIGNMENT AMBIGUITY This report outlines the methodology and process used to develop an instrument for classifying (evaluating) the ambiguity level in an assignment. The evaluation survey considers two dimensions of an assignment: the physical assignment content and the student knowledge level. When an assignment is developed, the learning objectives and student prerequisite knowledge are important factors in the design. The objectives provide the focus for the assignment and its requirements. The student knowledge level, both prerequisite and knowledge being learned are major factors in the design. The ideal instrument would produce a score that relates or measures the ambiguity level of an assignment. The score should be based upon the various constructs that have been used to define ambiguity in the literature. Additionally, the instrument should capture an assessment of the assignment's learning objectives and the student's knowledge level. The success of such an instrument requires that it be easy to use by the assignment author and produce consistent and repeatable scores Assignment Content The readability statistics of the assignment forms the basis for evaluating the content. Statistics such as: language, reading level, length, assignment complexity, and the rules and constraints of the problems can be rated by inspection or review. The assignment components may also be evaluated using the three Budner situations and the eight dimensions outlined by Norton. Rating these characteristics will require that judgments be made by the assignment designers or other experienced computing educators. Student Knowledge Level Assessing the ambiguity level for student knowledge level requires consideration and rating of several factors: the novelty of the task, new knowledge being learned, and prerequisite knowledge. Each of these factors must be rated to complete the assessment of an assignment's ambiguity. The first factor, the assignment language and learning objectives, represents a new challenge for the learner. This factor is adequately covered by using readability statistics and evaluating the Budner and Norton constructs. The second factor, new knowledge, is the concepts being practiced / exercised by the assignment. Usually, this knowledge has recently been covered by the course materials and should be more recent in the mind of the learner. Recent knowledge may be considered less ambiguous because of its currentness. However, the complexity of the concept may raise the ambiguity level. For example, pointers may be the focus of the assignment and recently covered in class but usually prove to be difficult to master. The third factor, prerequisite knowledge, may be considered less ambiguous because it has been mastered in the past. However, this knowledge must be recalled by the learner. The assignment narrative contains important cues to the exact nature of the needed information. The type and nature of these cues will cause the level of ambiguity to vary. Consider the different cues provided by the following wording examples for the same problem: Example 1. "Write a program to compute the area of a series of circles given a list of diameters." Example 2. "Write a program to compute the areas of a series of circles given a list of diameter values. Use a while loop structure and recall that the area of a circle is given by: A = pi * r2 where: pi = 3.1416, r is the radius of the circle, and r=D/2 where D is the diameter." The cues provided by Example 1 are limited for both new and prerequisite knowledge. Essential relationships and constant values are left to the learner to discern and provide. Example 2, however, provides more complete cues to not only the new and prerequisite knowledge but what is required to solve the problem. The ambiguity level of Example 1 should be rated higher than Example 2. Judging and rating ambiguity level for both knowledge types is a more difficult task because of the individual nature of each learner. Each individual reacts according to his or her own response preference, knowledge, and confidence. However, the assignment author routinely assumes a baseline knowledge level into the assignment design. Authors may provide hints to lessen the ambiguity impact of the assignment knowledge requirements. Assignment Ambiguity Level Instrument A preliminary evaluation instrument was developed to rate the ambiguity level of an assignment (Appendix A). The survey instrument includes areas for: 1) Readability Statistics, 2) Ambiguity Construct assessment, and 3) Student Knowledge assessment. The Readability Statistics section allows the evaluator to record the readability statistics produced by the word processor. Data included are: number of pages, number of paragraphs, number of sentences, words/sentence ratio, sentence/paragraph ratio, and the Flesch-Kincaid Reading Level score. The Ambiguity Construct area is composed of eight Likert scale questions derived from the Budner and Norton constructs. For each of these constructs, a brief description and a Likert five point rating scale has been developed that allows evaluators to produce a numeric score for each construct of the assignment. The evaluator considers each construct for the assignment and then rates each using the 1 to 5 scale. The Student Knowledge Level section consists of five Likert scale questions covering new and prerequisite knowledge and cues provided in the assignment narrative. The evaluator rates each area based upon an understanding of the assumed student knowledge levels. Scoring The assignment is scored by summing the readability score and the Likert rating of each of the construct and knowledge questions. The Likert questions are arranged so that a higher rating corresponds to higher ambiguity. This instrument will produce scores ranging from: the lowest score of 13 plus the reading level to a high score of 65 plus the reading level. For example, assuming a 12th grade reading level, an assignment would produce a range of possible scores from 25 to 77 (low to high ambiguity). To evaluate an assignment, the evaluator simply reviews the assignment, records readability data from the word processor, rates each individual component and tallies the score. Higher scores indicate higher levels of estimated ambiguity and vice versa. The assignment author is expected to be the first evaluator of any assignment. The author has the most complete understanding of the assignment, its objectives, requisite knowledge, and potential solutions. Additionally, the author has access to the assignment in electronic form and can provide the detailed readability statistics from the word processor. [COMMENT1]Method for Validating the Instrument For a survey to be valuable, it must be both valid and reliable. Validity implies that the survey measures what it is intended to measure and reliability ensures that scores and classifications are accurate and consistent. Additionally, a survey instrument must also be usable. The typical evaluator must be able to understand the questions, make a judgment, and produce a workable score. The focus of this pilot study is to refine the usability/readability of the evaluation instrument. Therefore, initial efforts are centered on the clarity of question wording and ease of use and scoring. If this instrument is "workable" (i.e., can be easily completed by faculty) then further work on the validity and reliability is warranted. 4. RESULTS This instrument was circulated among several of the faculty in the School of Computer and Information Science during the summer 2002 semester. Faculty members were asked to select a typical assignment and complete the survey. The chosen assignments ranged from introductory level to senior project courses. After completing the survey, the faculty members were asked to comment on the clarity of the survey instrument. Because this instrument assesses constructs that are new to the assignment authors, understanding the instructions, examples, and question wording is vital to smooth operation of the instrument. Feedback, both positive and negative, was welcome during this pilot test. Instrument Clarity & Operation The initial responses from the faculty evaluators pointed out several errors. Typing errors were easily found and corrected. However, the instrument instructions were found to be less than clear. The wording of some of the questions and the examples were also confusing to some of the respondents. One problem, uncovered during the initial testing, was that lack of knowledge about the writing and readability statistics that each word processor could provide. Many evaluators were unaware that the readability information did not have to be computed by hand. The instrument was modified to include specific instructions for obtaining the statistics from both Microsoft Word and Wordperfect. The balance of the data from the experiment was related to the numeric scores produced by the instrument. It was hoped that for a range of different assignments, a range of useful scores would be produced. Course Level Assignment Number Ambiguity Estimate Amb. Score Freshman 1 Low 30.2 Freshman 2 Low 43.2 Sophomore 3 Medium 34.1 Sophomore 4 Medium 35.4 Junior 5 High 48.9 Junior 6 Medium 34.0 Senior 7 High 41.0 Senior 8 Medium 35.7 Senior 9 High 58.6 Table 1. Preliminary Assignment Scoring Data Classification An ambiguity estimate was assigned by the researchers to each assignment using the rationale described earlier: initial courses should have low ambiguity, while senior level courses represent high levels of ambiguity. Sophomore and junior level assignments may range between medium to high levels depending upon the purpose of the assignment. These classification ranges represent scores in the following ranges: low (23 to 40), medium (40 to 57), and high (57 to 74); assuming a 10th grade reading level. These data produced average scores for the low, medium, and high ambiguity ranges of 36.7, 34.8, and 49.5 respectively (See Table 1). Score dispersion The range of values produced in this sample was 28.4 or 52.6% of the total possible range. This range is relatively small compared to the possible range of values. Two possible explanations are possible: small sample size or inconsistencies in the instrument. Samples from the estimated low and medium assignments had average scores of 36.7 and 34.8 respectively. It seems possible that the instrument does not produce enough dispersion in scores. One potential solution is increasing the range of the Likert scales. Sample size and assignment diversity Time constraints of this pilot study produced small sample (n=9). Statistical conclusions cannot be made from this small sample. The study results did accomplish the primary goal of refining an evaluation instrument that appears to be usable, to produce numeric scores, and to classify assignments. The assignments sampled covered the three level of assumed ambiguity and course levels. However, there was limited coverage for any one assignment type. More diverse data are needed to further improve the instrument's rating ability. 5. FUTURE DIRECTIONS The results from this pilot study suggest that further work is warranted. A re-design of the instrument itself may be able to provide a larger scoring range. Refining the instrument instructions and language may improve the usability for the evaluators. Repeating the pilot study with a larger number of evaluators and assignments should allow further refinement of the instrument. In this study, only the assignment authors evaluated the assignment. Increasing the number of evaluators for each assignment as well as increasing the diversity of assignments is the next step. Once a "workable" instrument is available, a formal inter-rater reliability and validity study should be conducted by correlating the ratings among several evaluators and assignments. While not difficult to accomplish, obtaining suitable assignment materials and willing evaluators may pose a logistical and time challenge. 6. REFERENCES Budner, S. (1962). "Intolerance of ambiguity as a personality variable." Journal of Personality 30:29-50. Denning, P. (1998). "Computing the profession." Educom Review 33(6):26-40. McLain, D.L. (1993). "The MSTAT-I: A new measure of an individual=s tolerance for ambiguity." Educational and Psychological Measurement 53: 183-189. Norton, R.W. (1975). "Measurement of ambiguity tolerance." Journal of Personality Assessment 39(6):607-619. Appendix A Ambiguity Evaluation Survey Assignment Name: ___________________________________ Course Number: _______ Evaluator Name: ____________________________________ Date: ___/___/___ Instructions Part A. Assignment Content Please consider the physical language and content of the assignment only. Do not add any assumptions or meanings to the assignment. Consider only the "physical assignment" as provided when you rate each of the following questions. Part B. Student Knowledge Level Please consider the knowledge level assumed to be held by the average student completing the assignment. Consider the "recentness" (when it was initially encountered in their educational experience) of the required knowledge. Circle the rating value that in your opinion most closely matches the true nature of the construct for this assignment.Part A. Assignment Content Using the analysis features of your word processor, report the following assignment characteristics: Number of pages: _______ Number of paragraphs: _______ Number of sentences: _______ Number of words: _______ Words / sentence: _______ Sentences / paragraph: _______ Flesch-Kincaid Grade Level: _______ (Word processor note at end of this document) Multiple meanings implies that the learner might read the assignment and conclude more than one meaning. Multiple meanings may arise from different word meaning. (multiple meaning) 1. The assignment contains language that can be understood in more that one way? one way multiple ways ? ? ? ? ? Evaluate the wording of the assignment for completeness. Has the author provided all of the required information to adequately describe the problem? (completeness) 2. The information provided in this assignment is... very complete very incomplete ? ? ? ? ? Evaluate the language of the assignment for the preciseness in stating the problem and other provided information. Fuzzy words are imprecise. (Vagueness) 3. The language of this assignment is ... very precise very imprecise ? ? ? ? ? Evaluate the organization of the assignment. Is the narrative logically ordered and in an appropriate sequence. Do first things come first? (Unstructured) 4. The organization of the assignment is ... very organized very unorganized ? ? ? ? ? Evaluate the assignment for any missing information. Remember that perquisite information maybe suggested or assumed; and therefore, it is not to be considered missing. (Lack of information) 5. To solve this assignment, rank the level that all information is provided. All information Minimum information provided provided ? ? ? ? ? Evaluate the assignment in relation to your understanding about the current topics that should be being covered in the class at the time of the assignment. This ranking requires your best estimate from your experience. (inconsistencies) 6. When considering current course topics, this assignment is consistent with inconsistent with current topics current topics ? ? ? ? ? Evaluate the language of the problem for its clarity. Will the learner understand on the first reading or will they have to reflect to completely grasp the problem? (unclear) 7. The problem statement for this assignment is ... clear unclear ? ? ? ? ? Evaluate the objectives of the assignment for their complexity. Is there a simple one-to-one match with course concepts or is higher levels of knowledge required. (complexity) 9. This assignment is ... very simple very complex ? ? ? ? ? Part B. Student Knowledge Level Consider the knowledge about the main (primary) objectives needed to complete this assignment. 10. Primary knowledge needed to complete this assignment was acquired ... recently in the distant past ? ? ? ? ? Consider the prerequisite knowledge needed to complete this assignment. 11. Background knowledge required to complete this assignment was acquired ... recently in the distant past. ? ? ? ? ? Does the assignment language clearly indicate the knowledge relating to the main (primary) objectives? 12. The cues to primary knowledge needed to complete this assignment are very strong very obscure ? ? ? ? ? Does the assignment language clearly indicate the prerequisite knowledge? 13. The cues to background knowledge needed to complete this assignment are ... very strong very obscure ? ? ? ? ? In your opinion, has the first-time student encountered this type of assignment before? 14. From a student perspective, this assignment is .... not new very new ? ? ? ? ? To obtain the readability statistics for the assignment use the following tips for your word processor. Microsoft Word: Click on: Tools B> Spelling & Grammar B> Options B>Show Readability Statistics. You may have to allow the Spelling and Grammar checker to complete checking the document (Clicking Ignore All will advance the checking.) When checking is complete the statistics will be displayed in a box. WordPerfect: Click on: Tools B> Grammatik B> Options B> Analysis B> Readability. Record the Flesch-Kincaid Grade level. The balance of the data is under the Basic Counts button. 1 sweeney@cis.usouthal.edu 2 owen@cis.usouthal.edu [COMMENT1]3. Method for validating the survey and classification a. Have faculty submit and rank assignments b. Have other faculty independently rank each assignment c. Statistically correlation among faculty rankings 1