A CLA Assessment in the Principles of Information Systems Course Bee Yew byew@uncfsu.edu Department of Management Fayetteville State University Fayetteville, NC 28301, USA Abstract This paper summarizes a Collegiate Learning Assessment (CLA) performance task developed for a core course in Information Systems Theory and Practice course for business majors. The CLA in classroom project is a university-wide initiative aimed at improving students’ critical thinking skills, which include analytic reasoning and writing skills and problem solving skills. Institutions participating in CLA assessment are expected to increase efforts to infuse critical thinking skills in class teaching. The expected outcomes for the CLA in class project include a better understanding of the structure of a CLA performance task. Assessment results gathered from CLA in class project are expected to increase instruction efforts in developing class activities and course assessment components that will result in improving student performance in CLA performance task categories. These five categories, based on the CLA grading rubric are: 1) evaluation of evidence, 2) analysis and synthesis of evidence, 3) drawing conclusions, 4) acknowledging alternative explanations and viewpoints, and 5) written communications. Keywords: Collegiate Learning Assessment (CLA), Critical-thinking skills, Performance task 1. INTRODUCTION The national Collegiate Learning Assessment (CLA) program offers an authentic assessment of students’ higher order thinking skills that include critical thinking, analytic reasoning, problem solving skills and written communications. These skills are regarded to be “necessary for personal and professional success in the 21st century” (Architecture of the cla Task, p. 30). According to Mueller (2008), authentic assessment is “a form of assessment where students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills”, and alternative names for authentic assessment are performance assessment, alternative assessment and direct assessment. The CLA uses a rubric for scoring students performances in two types of task: the performance and the analytic writing tasks. The CLA in class project is an initiative to increase instruction efforts geared towards enhancement of student learning about critical thinking skills though application. The performance task designed for this course is to be regarded as a trial for students, as well as an experience that would prepare them in their expectations in a real CLA performance task assessment. The Council for Aid to Education (CAE), a non-profit organization, administers CLA. The task developed for the CLA in class project for this course is therefore a reflection of student class learning experiences about a teaching case. This paper describes the process of preparing students for the performance task developed for the Principles in Information Systems course. The results for a performance task that was developed by the author teaching this course, is discussed in this paper. This task was administered to students in two sections of the course taught in Spring 2009. 2. TEACHING CASE The CLA task performance designed for the Principles of Information Systems course is based on the real estate case for the Affordable Home Real Estate Company. This is a computer case used by the instructor for teaching the application of an Excel spreadsheet program to analyze business information for decision-making. The teaching case also requires students to learn about the Pivot table tool and to implement the tool to create very useful summaries of a large dataset. These table summaries assist the analyst or decision maker in identifying associations among key variables. To prepare the students for the performance task, a class plan was devised to relate students’ class learning about the Pivot table tool and the use of Internet for obtaining relevant information to support data analysis and recommendation. Students were also expected to complete the reporting of their learning experiences according to a template before the performance task. 3. THE CLASS PLAN The CLA in class project encourages an instructor to find ways to engage students with learning activities that focus on critical thinking, analytic reasoning and problem solving skills. For students in the two sections of the Principles of Information Systems course, a performance task that was designed based on a teaching case was administered after students had 1) completed the class activities and reporting requirement for the case, and 2) taken their first test on basic IT concepts. The following table is a class plan for three weeks of the semester that ends with the assessment of the class CLA task: Table 1: Class Plan Week beginning on Class activities Jan 18th Case discussion and Excel Pivot table tool Jan 25th Analysis of dataset Feb 1st Discussion on recommendation based on analysis and extended analysis Feb 8th Exam 1 and CLA performance task 4. PROCESS DESCRIPTION The process is described as a two-phase process. The first phase is the discussion of the case with class activities. Student learning in these class activities are assessed with extra credit assignments that they have completed and submitted via Blackboard System assignment tool. These extra credit assignments questioned 1) the students’ understanding on key assumptions in the case, 2) their proficiencies in using pivot table tool to create 1, 2 and 3-dimensional pivot table summaries of the dataset provided for the case, and 3) use of the Internet resource to enhance their understanding of contextual information relevant to the case. The class policy on extra credit assignments is that the points earned from these assignments will be sued to curve their overall course grade at the end of the semester. These assignments were designed to reinforce their learning in class activities. The lack of detailed instructions for analysis of the teaching case for Affordable Home Real Estate Company provided an opportunity for in-class discussions associated with activities about identifying facts, assumptions, gathering external sources of information by “googling”, and synthesizing pivot table results with relevant information. Students were prompted to develop an understanding of the case situation with the facts provided about the scenario. The dataset provided for the case consisted of 600 sales records for houses in some development project. Another similar housing development project was about to be completed. In explaining the specific information in the dataset that might be important for analysis, students were instructed to think about the role of analysis – what conclusions can be derived from the analysis? The second phase is about the process for the administration of the CLA task in these classes. Students were informed in the 1st day of the semester class meeting that their participation of a CLA assessment will be treated as extra points that will be added to their first exam scores. The CLA assessment date was set on sometime before February 14th after the completion of the instructor-led analysis of a case that will be used as the backdrop for the CLA performance task. The time allowed for the CLA assessment would be 45 minutes. This time limit is half the allowed time for the institutional assessment of CLA task performance. The justification for the shorter time is the familiarity of the case to the students. Students in these two classes were informed that their performance in the CLA task would be evaluated as points added to their earned first exam scores. Since the highest score for the first exam is 46 out of 54, the points added would be computed as CLA score divided by CLA possible multiply by 8. In this particular setting, students were asked to form two to three members groups for the completion of the performance task, although the actual CLA performance task is an individual performance test or assessment. The rationale for grouping students for the task is based on the instructor’s perception that these students may not have sufficient learning experiences in critical thinking to do well in individual performance task. Additionally, this course is taken during the students’ 1st junior year, and the in-class learning activities for the teaching of the case were student group activities. Several students however, opted to work individually on the performance task. 5. CLA TASK DESCRIPTION The CLA performance task requires students to answer three open-ended questions based on a range of information sources. These sources appear as exhibit documents along with the description of the task. These documents could be letters or memos, summaries of research reports, newspaper articles, maps, photographs, diagrams charts, tables, and interview notes or transcripts. CLA performance tasks require students to gather evidence from these various sources by distinguishing relevant information from irrelevant information, thus identifying “red herring” in the documents presented. In weighing evidence in documents to support their positions, students must be able to distinguish rational from emotional arguments, and facts from opinions. Students must also be able to synthesize these evidences and present their ideas clearly; therefore their written communication skills are critical to their performance in this type of task. 6. CLA TASK DOCUMENTS The CLA performance task designed for the course follows the guidelines by CLA in Classroom program. Six documents labeled Exhibit A through G were developed for the task. Appendix A provides the document for the instructions and questions for the task. Document A is a memo addressing the decision situation written by the owner of the real estate company to the consultant who was hired to analyze data and to make some recommendations on the housing project decision. Document B is an article from a website on survival strategies for small real estate companies. Document C is another memo from the consultant to the owner with suggestions on what other criteria to consider in the selection decision. Document D consists of charts on market days and price difference criteria and avg. selling price per square feet. Document E contains the NAR and NAHB survey results on homeowners’ considerations in their home purchase decisions. Document F provides tables summarizing information about average selling price, square feet of homes, and count of homes in housing projects by number of bedrooms. Document G reports survey results published in Mortgage News Daily about home features that were considered to be important by homeowners. A, B, and C documents are narrative documents, E and G documents are a mix of narrative and quantitative documents, while D and F documents are quantitative. 7. CLA TASK GRADING RUBRIC The CLA performance task evaluates five items in assessing critical thinking, analytic reasoning, and problem solving skills. These five items are 1) evaluation of evidence, 2) analysis and synthesis of evidence, 3) drawing conclusions, 4) acknowledging alternative explanations and viewpoints, and 5) written communications (cla Common Grading Rubric). The grading rubric and scale for the three open-added questions for the task is presented in Appendix B. The table below summarizes the documents to be evaluated to support answers for the questions asked: Table 2 Question Documents question 1 G, D and F question 2 B, D, E, F and G question 3 A, C, D, F and G 8. ASSESSMENT RESULTS The assessment results for 30 students from two sections of the Principles in Information Systems course who have opted to participate in completing the performance task designed for the class are provided in a table in Appendix C. Groups that have higher total scores (at least 6 and above) in the “Evaluation of evidence” and “Analysis and synthesis of evidence” categories tend to have high scores for the “Written communications” category. With the exception of Groups 12 and 18, student groups on the average, scored significantly lower in the evaluated category of “Acknowledgement of alternative explanations and viewpoints”. This could be explained by the statement in bold that appears in the instruction set in Appendix A, which says, “While your personal values and experiences are important, you should base your response on the evidence provided in these documents” (see Appendix A). This statement was taken from a different CLA performance task provided in a handbook used in a CLA workshop for faculty in this institution. This part of the instruction could be interpreted as to avoid having different explanations or viewpoint from what have been presented in the documents for their responses. In designing the performance task for this paper, a retired CLA “Crime Reduction” performance task that presents evidences for the two opposing views on how rising crime in a city should be tackled: drug addiction treatment program versus increasing police officers, was referenced (Collegiate Learning Assessment: Sample Performance Task). In the summary of results in Appendix C, the low average score of 1.11 in the “Acknowledgement of alternative explanations and viewpoints” could be attributed to the ambiguity in the instruction. In decision-making, individual personal values and experiences are acknowledged to be intrinsic factors in interpreting evidence in a evaluating a situation. The high incidence of missing responses in this category indicates that students may be confused with this part of the instruction, and therefore avoided providing a response for that would refer to an argument against perspectives presented in the documents A and C. A modification of the instruction to read as “Your responses to the questions asked should not be based on opinions, but on explanations about evidences in documents that lead to your conclusions”, may alleviate ambiguity in the instruction. 9. CONCLUSIONS The recent emphasis on student critical thinking problem solving and written communications competencies by accreditation associations, requires institutions to assess these skills in their academic programs. CLA offers institutions a way to assess students on these competencies in the form of performance tasks and analytic writing tasks. The CLA model is in an education that focuses on the “link between assessment and teaching, and learning”. The adjusted scores in the institutional report for CLA results compensate for “schools with different admission standards or imperfectly representative samples” (cla University College, pg. 4). According to this report, CLA is a way for institutions to benchmark student progress from freshmen to senior year in these competencies. A study by Klein, Kuh, Chun, Hamilton and Shavelson (2005) revealed that student performance on open-ended tasks was related to college grades, and mean scores for these tasks increased consistently from freshmen to senior class after controlling for SAT scores. Developing CLA performance tasks in classes will help to achieve these student-learning outcomes by creating experiences in applying critical thinking in classroom environments. 10. REFERENCES “Architecture of the cla Task”, Retrieved from http://www.collegiatelearningassessment.org/files/Architecture_of_the_CLA_Tasks.pdf “cla Common Grading Rubric”, Retrieved from http://www.cae.org/content/pdf/CLA_Scoring_Criteria_(Jan%202008).pdf “cla University College”, Retrieved from http://www.collegiatelearningassessment.org/files/CLA_0809_Institutional_Report_University_College.pdf. Klein, S., Kuh, G., Chun, M., Hamilton, L. and Shavelson, R. (2005). “An Approach to measuring Cognitive Outcomes Across Higher Education Institutions”. Research in Higher Education, Vol. 46. No. 3, pp. 17-26. Mueller, J. (2008). “Authentic Assessment Toolbox: What is Authentic Assessment”. Retrieved from http://jonathan.mueller.faculty.noctrl.edu/toolbox/whatisit.htm. Yew, Bee (2009). “CLA Instructor’s Assessment Report”. FSU Digital Commons. Retrieved from http://digitalcommons.uncfsu.edu/div_aa_wp/1/. . APPENDIX A Instructions You are about to take an assessment that was designed to measure your skills in critical thinking, analytic reasoning, problem solving and written communications. You will be answering a series of open-ended questions about a hypothetical but realistic situation. This assessment contains a series of documents that includes a range of information sources. While your personal values and experiences are important, you should base your response on the evidence provided in these documents. Scenario: Which one Housing Project? In the 3rd quarter of 2005, a national study announced that Eau Claire, Wisconsin was the safest place to live. Since then housing development projects have been springing up all around Eau Claire. Six housing development projects are currently dominating the Eau Claire market: Woodland Hills, Granite Mound, Creek Side Huntington, East River Community, Forest Green, and Eau Claire South. These 6 projects each started with 100 homes, have sold all of them, and are currently developing phase 2. Project Number Project Name 23 Woodland Hills 47 Granite Mound 61 Creek Side Huntington 78 East River Community 92 Forest green 97 Eau Claire South The owner of Affordable Real Estate Company (AHRE) is requesting the services of a real estate consultant, Alice Garner to analyze information from the past 600 home sales in Phase I. As the AHRE is a small business operation, the home builder company (Adobe Builder) has suggested that AHRE should focus on selling homes in only one of the six housing development projects. Question 1 Mark Fisher (owner of AHRE) suggested that price difference between selling price and asking price is a strong indicator of home buyers’ perceptions of value for the homes they have bought. His reasoning is that if the home buyers were willing to purchase homes at selling prices that were closer to the asking prices, these homes are perceived by buyers to be more desirable homes (Document A). Alice Garner, the consultant hired for analysis of new homes in Phase II thinks that although price difference does indicate the extent of fair pricing of homes, it does not take into account the buyers’ preferences for number of bedrooms and size of the homes. Your answer with supporting statements on whether you agree or disagree with her: 1a. Do you agree or disagree with Alice Garner? 1b. What are your supporting statements for your answer above? Question 2 The dataset available to Alice Garner for analysis has the following information about homes sold in Phase I: From the documents provided, can you derive factors that are important to home buyers which could be analyzed with the data provided? Write your answers in the following table: Document Answer: either Yes or No to factors that can be derived from documents Answer: What are these factors? B D E F G Question 3 3a. If you were to recommend a housing project, which one would you recommend to Mr. Mark Fisher (the owner of AHRE Company)? In your recommendation for that one project, indicate which one project: #23, #47, #61, #78, #92, or #97 3b. What are your supporting statements for your project recommendation? 3c. Would you recommend more than one housing projects? Your answer: Yes or No? 3d. What are your supporting statements for your answer above? APPENDIX B Grading Rubric Meaning of scale -> 1: emerging, 2: developing, 3: mastering Question Evaluation of evidence Analysis & synthesis of evidence Drawing conclusions Acknowledging alternative explanations or viewpoints Written communication Total Pts 1. Do you agree /disagree with the consultant on why price difference is not the only criterion to evaluate housing projects 2. What factors are important to buyers from documents B, D, E, F & G? 3. Which one project or projects would you recommend to the owner? Total Pts APPENDIX C Group # Individuals Total Pts Evaluation of evidence Analysis and synthesis of evidence Drawing conclusions Acknowledg. alternative explanations or viewpts Written comm. Group 1 1 6 2 2 1 0 1 Group 2 1 30 8 8 8 0 6 Group 3 1 31 9 9 7 0 6 Group 4 2 25 6 6 6 0 7 Group 5 2 16 4 4 4 0 4 Group 6 2 13 3 3 3 0 4 Group 7 2 23 6 5 6 0 6 Group 8 2 12 4 2 2 0 4 Group 9 2 3 0 0 0 0 3 Group 10 2 23 6 6 6 0 5 Group 11 2 17 3 2 4 2 6 Group 12 1 38 7 9 8 7 7 Group 13 1 19 4 4 5 2 4 Group 14 3 28 7 6 6 1 8 Group 15 2 16 4 3 3 2 4 Group 16 1 14 4 4 3 0 3 Group 17 2 23 7 5 5 0 6 Group 18 1 42 9 9 9 6 9   Average 5.17 4.83 4.78 1.11 5.17 Maximum pts possible is 45 points, 0 to 3 pts for each of the following categories for the 3 questions: Evaluation of evidence Analysis and synthesis of evidence Drawing conclusions Acknowledging alternative explanations/viewpoints Written communications Points for each category 0 for non existing 1 for emerging 2 for developing 3 for mastering APPENDIX D Description of Performance Task Which one Housing Project? In the 3rd quarter of 2005, a national study announced that Eau Claire, Wisconsin was the safest place to live. Since then housing development projects have been springing up all around Eau Claire. Six housing development projects are currently dominating the Eau Claire market: Woodland Hills, Granite Mound, Creek Side Huntington, East River Community, Forest Green, and Eau Claire South. These 6 projects each started with 100 homes, have sold all of them, and are currently developing phase 2. Project Number Project Name 23 Woodland Hills 47 Granite Mound 61 Creek Side Huntington 78 East River Community 92 Forest green 97 Eau Claire South The owner of Affordable Real Estate Company is requesting the services of a real estate consultant, Alice Garner to analyze information from the past 600 home sales in Phase I. As the real estate company is a small business operation, the home builder company has suggested that the company should focus on selling homes in only one of the six housing development projects. Adapted from Management Information Systems for the Information Age by Haag, Stephen and Cummings, Maeve, 7th edition, McGraw-Hill, page 465. Document A Document B Source Realty Times: http://realtytimes.com/rtpages/19981104_survival.htm, Published: November 4, 1998 The Top 10 Survival Strategies for Small Real Estate Offices by Steve Paris These strategies were originally compiled in a document by Don Taylor, entitled: Up Against the Wal-Marts. However, it applies to all small businesses regardless of who or how big your competitors are: 1. Focus completely on satisfying your clients. Develop a true client focus. Understand that clients go where they get good value, they go where they are treated well, and when the value isn't obvious or when the level of service slips, the customer slips away. Identify your target market - the more you know about the client, the easier it is to serve them well. 2. Study the success of others. If you want to be a smart Realtor, study smart Realtors. Study the competition, learn from your peers, and consider every situation you encounter as an opportunity to gather ideas to use in your business. 3. Gather and analyze management information regularly. You need strategic information on your own business in order to know what's really going on and to make wise decisions based on accurate, timely information. Use whatever technology you have, even if it's manual, to collect information in the following four areas: financial, client, industry and market trends. 4. Sharpen marketing skills. Marketing must consider your service offerings, your place or location relative to the client and their daily routines, your advertising efforts (print media as well as internet), your people (working in the office), and your positioning within the market. All of these are important, but you have to decide what you will focus on and what clients are most sensitive about 5. Increase the client's perception of value. Value is not the same as price. It also includes considerations for quality (is your service better?) and quantity (do you offer more than others?). But most clients are somewhat price sensitive. Therefore, the perceived value must be greater than your competition. 6. Position the business uniquely. You aren't one of the big guys, so don't try to be just like them. Position yourself in areas where big business cannot compete with you, like friendliness and customer bonding. Where can you say, "We're better because...?". 7. Eliminate waste. Plan your spending. Keep a budget. Constantly compare your budgeted and actual figures to learn where costs are going. Every dollar saved from current operating costs goes directly to your bottom line. Time is money, and eliminating time wasters saves you money. 8. Find something to improve every day. Make it a formal goal to improve one aspect of your business every day, no matter how small. 9. Embrace change with a positive attitude. Having a positive attitude is consistently one of the common factors of success. You must not fear change; change is your friend. As you continually improve, you'll be embracing change. Enjoy the ride! 10. Pull the trigger and start the battle. Some people stay on "Ready, aim, aim, aim..." You must learn to take action quickly and also cut losses when you're wrong. Go ahead and pull the trigger rather than wait for the perfect shop - it never comes. Document C Document D Document E NAR and NAHB survey results for 2000 households derived from national panel of respondents who purchased a primary residence within the last 48 months Document F Summary information of 2, 3 4 and 5 bedroom houses for housing projects PROJECT # Description 2 bedroom 3 bedroom 4 bedroom 5 bedroom Description summary for each project 23 Average SELL PRICE $89,445 $81,348 $83,163 $76,000 $82,420 Count of Homes 2 40 56 2 100 Average SQ. FT. 1,650 1,548 1,568 1,400 1,559               47 Average SELL PRICE $80,195 $80,238 $80,352 $86,000 $80,374 Count of Homes 2 28 69 1 100 Average SQ. FT. 1,500 1,502 1,508 1,600 1,507               61 Average SELL PRICE $74,290 $81,779 $82,957 NA $82,183 Count of Homes 2 51 47 NA 100 Average SQ. FT. 1,400 1,552 1,569 NA 1,557               78 Average of SELL PRICE $82,250 $81,866 $79,966 $73,267 $80,438 Count of Homes 2 33 62 3 100 Average SQ. FT. 1,525 1,528 1,505 1,383 1,510               92 Average SELL PRICE $82,570 $82,338 $82,255 NA $82,303 Count of Homes 3 46 51 NA 100 Average SQ. FT. 1,550 1,551 1,559 NA 1,555               97 Average SELL PRICE $73,933 $86,111 $86,203 NA $85,678 Count of Homes 4 37 59 NA 100 Average SQ. FT. 1,413 1,621 1,620 NA 1,612 FOR ALL PROJECTS Average of SELL PRICE by # of bedroom $79,720 $82,326 $82,382 $76,300 $82,232 Count of homes by # of bedroom 15 235 344 6 600 Average SQ. FT. by # of bedroom 1,497 1,553 1,552 1,425 1,550 Number of houses for projects by 3 possible classes of square feet range Count of homes SQ. FT. PROJECT # Between 950 to 1299 sq. ft. Between 1300 to 1649 sq. ft. Between 1650 to 2000 sq. ft. Total count of homes 23 1 64 35 100 47 4 66 30 100 61 10 52 38 100 78 8 64 28 100 92 5 51 44 100 97 6 41 53 100 Total count of homes 34 338 228 600 Document G Home Buyer Survey Ranks Features That Are Important To Buyers Story Information Title: Home Buyer Survey Ranks Features That Are Important To Buyers Date: Wed, 15 Aug 2007 12:02:54 EST Location: http://www.MortgageNewsDaily.com/8152007_Home_Features.asp Date Printed: 8/15/2007 All Content Copyright © 2003 - 2007 Brown House Media, Inc. All Rights Reserved. Reproduction in whole or in part in any form without the express written permission of MortgageNewsDaily.com is prohibited.