Assessment of a Systems Analysis Methods Course in a Small Liberal Arts College Joan E. Hoopes joan.hoopes@marist.edu Craig W. Fisher craig.fisher@marist.edu Information Systems, Marist College Poughkeepsie, NY 12601 Abstract Marist College is an accredited institution, having been granted accreditation by the Middle States Association of Colleges and Universities. The College is required to assess the overall institutional effectiveness with primary attention given to assessment of student learning outcomes. Each faculty member was expected to choose one of his or her fall courses to prepare for assessment. Two Information Systems faculty members chose to assess the Systems Analysis Methods course. A quasi-experiment using pre- and post-tests was employed to measure increase in knowledge during the semester. The "treatment" consisted of the course lectures, exercises, assignments, and materials. The pre- and post-tests were aimed at the broad categories of systems analysis and attempted to measure each student's ability to synthesize the concepts and ideas of systems analysis and each student's competence in one or more skills related to the objectives for learning. The pre- and post-tests were graded by two faculty members and analyzed. Statistical Package for the Social Sciences Version X (SPSSX) was used to measure the difference in knowledge as reflected by the tests. The post-test scores were significantly higher than the pre-test scores in all categories but one. This paper is a discussion and report of the assessment process and results, as well as of the experience gained. Keywords: Assessment, Accreditation, Systems Analysis 1. INTRODUCTION Marist College is a small liberal arts college in Poughkeepsie, New York, and is accredited by the Middle States Association of Colleges and Universities. "Accrediting agencies increasingly press schools to provide assessment models and to specify measurable outcomes for courses and curricula" (Diamond 1998). As such, the college is required to assess overall institutional effectiveness with primary attention given to assessment of student learning outcomes (Sherr & Teeter 1991). The college is required to identify the different components that comprise the assessment process. One of these components is the assessment of educational outcomes. One key educational outcome is the amount of knowledge gained over the semester. While there are a variety of opinions and theories related to establishing "outcomes," these authors decided that the most direct way to assess outcomes was to test students to see if they gained knowledge and skills related to the course objectives. The second reason for the type of assessment approach used was to assure that the students were well prepared for subsequent courses that rely on the knowledge and skills gained in the Systems Analysis Methods course. The two related courses are Systems Design Methods and Information Systems Project. Each faculty member was expected to choose one of his or her fall courses for assessment purposes. Two Information Systems faculty chose to assess the Systems Analysis Methods course. The faculty developed pre- and post-tests based upon final exams from prior years. The pre- and post- tests were aimed at the broad categories of systems analysis and attempted to measure each student's ability to synthesize and work with the various concepts and ideas of systems analysis. The pre-test was administered on the first day of class and the post-test was administered as the final exam. As expected, the actual teaching of the course, made a significant difference in the test results. The scores on the post-tests were significantly higher than the scores on the pre-tests. The purpose of this paper is to describe the assessment process used and the results obtained. 2. ASSESSMENT Educational assessment looks at teaching effectiveness, but there are many dimensions of assessment. Palomba (1999) states that "assessment is more than the collection of data." For a successful assessment, the information collected must have a purpose. Objectives for learning must be clearly defined as a basis for data gathering. The objectives must also be addressed in the curriculum. Faculty must present the students with a learning-centered syllabus that defines the responsibilities for successful completion of the course (Diamond 1998). There are problems with assessment, such as the need to measure effectiveness rather than efficiency and the degree of emphasis on tools and techniques versus specific outcomes (Hoopes 1993). For example, a teacher may be extremely aggressive in the use of multimedia techniques, but the students may miss the key points of the discussion. To address these problems, the reasons for the assessment must be clearly identified. The questions of what should be measured, why it should be measured, and how it should be measured must be answered. One can measure effectiveness using the course objectives. Assessment becomes much simpler with a clear set of objectives used for evaluation criteria (Angelo & Cross 1993; Hanchey 1995; MSA 1996). Assessment of learning outcomes attempts to answer the question: "How good is our teaching?" Therefore, the setting of objectives ensures that the faculty are assessing what they are teaching and teaching what they are assessing (Angelo 1993). Because the objectives for the Systems Analysis Methods course already existed, the authors decided to simply test against the objectives of the course, as described in the Course and Methodology sections below. Once it was decided that the assessment would focus on student outcomes, the method for assessment had to be determined. The three main strategies for assessment are proxy measures, portfolio measures, and value-added measures (MSA 1990; MSA 1996). One can use proxy measures such as retention, graduation rates, or job placement rates, but these factors tend to measure the overall Information Systems (IS) program and are heavily influenced by other moderator variables, such as economic climate and outlook. A second alternative is portfolio assessment, in which the students' work and projects are evaluated over time-perhaps many years (MSA 1990; MSA 1996). Alternatively, one can undertake value-added measures such as standardized tests or locally developed tests. Since standardized tests may not provide a good "fit" with local courses, locally developed pre- and post-tests are recommended (Sherr & Teeter 1991; MSA 1990; MSA 1996). The authors desired a direct measure of our teaching outcomes; therefore, we chose the locally developed pre- and post-test method. Objective tests are usually made up of a series of questions allowing students to demonstrate both the knowledge acquired and the ability to process and use that knowledge. Objective tests can be used in both pre- and post-test formats to collect data on the same group of students at different points in time (Palomba, 1999). These tests can contain multiple-choice, true/false, and matching questions, as well as questions that require problem-solving skills and that have a cognitive complexity component. Along with the assessment strategy, a college must also consider the costs and efforts associated with the assessment process (Richards 1995/96). The cost of conducting assessment programs depends on factors such as the assessment approach selected, the development of new assessment instruments or the purchase of existing tests, the reporting of the results, and class time that must be devoted to the testing (MSA 1996). Minimal resources were expended in the preparation of the objectives, the pre-test, and the post-test (final exam). However, the 90 minutes of class time dedicated to the pre-test was costly. 3. THE COURSE Two IS faculty members selected Systems Analysis Methods (IS 404) as the course to use for the assessment process. Twenty-nine students enrolled in the Systems Analysis Methods course. The students were primarily college juniors who had completed prior required technical courses in Computer Science and Information Systems, as well as core liberal studies courses and a variety of management studies courses. Due to the number of students and the high degree of student-teacher and student-student interaction required in the IS 404 course, the class was split into two sections. Both sections were taught on the same day and time, but by two different professors. Originally, equal numbers of students were assigned to each class. However, four students dropped one section and enrolled in the second section. One class finished with 10 students and the other finished with 19 students. This was not a concern to the authors, as we were interested in the overall totals, not the totals of individual sections. In addition, it was not our objective to attempt to compare teachers or techniques. Both professors were experienced teachers in the Information Systems field; however, one professor had previously taught the course several times at Marist College while the second professor had not taught it at Marist The course followed the first 12 chapters of Modern Systems Analysis and Design by Hoffer, George, and Valacich. The remaining chapters and appendices are covered in a follow up Systems Design course. 4. METHODOLOGY A pre-test and post-test with a single group of 29 students was used. The participants were the 29 students enrolled in both sections of the Systems Analysis Methods course. The overall group consisted of 20 males and 9 females: 22 juniors and 7 seniors. There were 17 Information Systems majors, 4 Computer Science majors, 4 Business majors, and 4 others. The criteria for the test were that the test should cover the main objectives of the course and be usable for the final exam at the end of the semester. The pre-test was given simultaneously to both sections during the first 90 minutes of the first class meetings. The post-test was administered during the final exam time period. The tests used for the pre- and post-tests were based upon the final exam from the prior year. Additional questions were developed to meet the objectives set for the course. Twenty-five multiple-choice questions were used. These questions covered analysis issues and concepts such as feasibility studies, system development life cycle, prototyping, Joint Application Development (JAD), Computer Assisted Software Engineering (CASE), functional decomposition, requirements specification, and the like. Three new problems were added that included analyzing a data flow diagram (DFD), developing a data gathering (DG) strategy, and critiquing an analyst-client interview (IV). Each problem had a degree of complexity associated with it. The DFD problem required the students to understand the concepts, syntax, and logic of a data flow diagram. The DG problem required the students to understand that there are multiple techniques used to gather data and that the techniques are dependent upon the situation at hand. Often a combination of a number of techniques should be used with varying priorities, depending upon the situation. Finally, the interview critique ensured that each student had a good grasp of appropriate and inappropriate interview techniques. Whereas the exam tested knowledge and some skills, it did not test interpersonal skills. We added a test on listening skills (Fisher 1993), but we did so without changing our course syllabus or adding any lessons to teach listening. The listening test consisted of listening to a reading comprehension question from a GMAT practice test. A volunteer from another course recorded the GMAT reading comprehension paragraphs onto an audio tape. Each professor played the tape to the students in a separate class and the students answered 10 multiple-choice questions about the material on the tape. These questions were the standard questions supplied with the GMAT practice book (Gruber & Gruber 1985). Three of the students did not take both the pre- and post-listening tests. The students knew that this exam did not count towards their final exam or their final grade in any way. It is the standard practice of the Information Systems faculty at Marist College to develop a syllabus for every course. The Systems Analysis Methods syllabus contained specific objectives, in addition to other materials such as schedule, grading, and attendance policies, as well as text and casebook requirements. As will be shown, the results of the listening test corresponded with the lack of emphasis placed upon the listening test. Hypotheses A series of hypotheses were specified for measurement purposes. The overall pre- and post-test included the 25 multiple-choice questions graded at two points each and the three problems graded at 10 points each, for a possible total of 80 points. The listening test was treated as a separate item and not added in to the totals. A face-value review of this measurement technique leads one to the "obvious" conclusions: students will probably fare poorly on pre-tests of the concepts in a course if the course material is new to them, and they should fare significantly better on post-tests after they have completed the course. Our hypotheses stress the improved test scores at the end of the semester as compared to the start of the semester. Hypothesis 1: The scores on the overall post-test will be higher than the scores on the overall pre-test. Hypothesis 2: The scores on the DFD post-test problem will be higher than the scores on the DFD pre-test problem. Hypothesis 3: The scores on the DG post-test problem will be higher than the scores on the DG pre-test problem. Hypothesis 4: The scores on the Interview (IV) post-test problem will be higher than the scores on the IV pre-test problem. Hypothesis 5: The scores on the Listening (LIS) post-test exercise will be higher than the scores on the LIS pre-test exercise. 5. RESULTS AND DISCUSSION Hypothesis 1 The first hypothesis was met with statistical significance t=11.9 and alpha equal to .000 (see Table 1). The average score on the pre-test was 34 out of a possible 80, whereas the average score on the post-test was 55. For N = 29 and 28 degrees of freedom, the t-value was nearly 12, indicating that the students were able to answer questions and problems at the end of the course that they were not able to answer at the beginning of the course. We concluded that learning occurred. The teaching, exercises, assignments, and class discussions were effective in communicating the objectives of the course. TABLE 1: Overall Pre- and Post-Test t-Scores Variable Mean Standard Deviation Standard Error t Value PRETST 33.7 9.1 1.7 11.9 PSTST 55.3 7.3 1.4 Hypothesis 2 The second hypothesis was met with statistical significance t=16 and alpha equal to .000, as can be seen in Table 2. The average score on the pre-DFD problem was less than one, whereas the average score on the post-DFD problem was close to 7.9 out of 10. The t-value of 16.4 was the highest t-score obtained. However, the raw score on the DFD problem was less than the raw score on the DG (8.4) and IV (8.2) problems. The high t-value is most likely due to the original low score on DFDs. The students clearly had no prior knowledge of DFDs but were able to grasp the key concepts of DFDs as taught in the course. TABLE 2: Data Flow Diagram Problem Pre- and Post-Test t-Scores Variable Mean Standard Deviation Standard Error t Value PREDFD 0.38 1.5 .28 16.4 PSTDFD 7.90 2.0 .38 Hypothesis 3 The third hypothesis, see Table 3, was met with statistical significance t = 8.5 and alpha equal to .000. The average score on the pre-DG problem was 4 and 8.4 on the post-DG problem. At the end of the course, the students were able to read a case problem involving a group of overworked people in a billing department and devise a strategy for collecting information that would minimize impact on the staff while maximizing the amount of information gained. The students had to specify at least three DG techniques in a logical sequence to receive full credit for this problem. TABLE 3: Data Gathering Problem Pre- and Post- Test t-Scores Variable Mean Standard Deviation Standard Error t Value PREDG 4.1 2.8 .53 8.49 POSTDG 8.4 1.5 .27 Hypothesis 4 The fourth hypothesis, depicted in Table 4, was met with statistical significance t = 5.22 and alpha = .000. The pre-test IV problem score of 5 out of 10 was the best percentage for the pre-test problems and the lowest overall t-value. Interviewing in general is a more common activity than writing data flow diagrams, and thus could easily lead to a higher initial score. For example, many of the students had been on more than one job interview, which is a very similar skill. The objectives of the course were met, as the scores on this problem increased significantly. TABLE 4: Interviewing Problem Pre- and Post-Test t-Scores Variable Mean Standard Deviation Standard Error t Value PREIV 5.0 2.9 .54 5.22 PSTIV 8.2 2.1 .38 Hypothesis 5 The fifth hypothesis was not met (see Table 5). The t value of -1.67 indicated that there were no differences on listening skills at the end of the course as compared to the beginning of the course. Many systems analysis texts stress the importance of interpersonal communications; however, little has been done to actually teach listening skills, and this course was no different. We did not explicitly teach listening skills, although we did have them listed as an objective of the course. It is also possible that because we did not include the listening test as part of the final exam, the students may not have felt an urgency to do their best. At the beginning of the course, the students did not know what to expect and may have tried very hard to do well on this test. In addition to "trying hard," there are several specific skills that can improve listening (Fisher, 1993). TABLE 5: Listening Problem Pre- and Post-Test t-Scores Variable Mean Standard Deviation Standard Error t Value PRELIS 42.3 19.0 3.7 -1.67 PSTLIS 37.3 17.5 3.4 Analysis The improved scores on the tests most likely indicate that our teaching was effective and give us a positive assessment of this course at this time. Alternative explanations or competing hypotheses are always possible, but not very likely in this case. The alternatives are possible because we did not maintain strict experimental control. The groups of students were not randomly chosen and there was no control group. This lack of control opens the possibility of threats to internal validity of the experiment. For example, one could say that the students may have learned DFDs on their own during this same time period, and we cannot compare our results to a control group to refute that claim. Perhaps there was something special about the 29 students in our classes that made them learn DFD and DG techniques on their own. However, on face value most would agree that the technical nature of the material would be too much for students to learn by "osmosis" during a busy semester. In a future assessment of this course, we plan to give the pre-tests and post-tests to students in other courses such as Data Management and Systems Architecture to provide for a control group. Another objection might be that the pre-test itself was an intervention that automatically raised the levels of the post-tests. On face value, this is not a likely effect, due to the 3.5 months between tests. The students did not receive any feedback on the pre-tests and the tests were likely forgotten by the end of the semester If there had been some influence, it would have probably been much smaller than the results obtained as indicated by the extremely high t-scores. 6. CONCLUSION Colleges and universities are now required by accrediting agencies to describe how they will assess learning outcomes. Assessment processes must begin with clearly defined objectives and methods to measure whether or not the objectives are being met. This paper has provided one method for assessment-pre- and post-tests, which are accepted by the Middle States Association (MSA 1990) and recommended by Palomba (1999) as one of several methods for assessment. Although questions have been raised about the assessment effort and its costs (Richards 1995), the cost of not engaging in assessment may be even higher. Lack of assessment could lead to the inability to examine student development because the attainment of objectives would be unclear. Assessment also provides information as to how well prepared students are for the more advanced courses that follow. The assessment results are intuitively obvious because most observers would expect the outcome to be positive (improved results after completing the course). However, if the course and/or professors do not adhere to the objectives (as in the listening exercise), then positive outcomes are definitely not a foregone conclusion. Therefore, professors must carefully define course objectives and teach to meet those objectives. The pre-test/ post-test method for assessing the Systems Analysis Methods course indicates that most objectives of the course were met. At the start of the course, the students were unable to answer questions or complete problems dealing with the concepts and methods of systems analysis. The average score on the pre-test was 34 out of 80, or 42%. At the end of the course, the students completed the questions and problems with an average score of 55 out of 80, or 70%. These post-test scores were higher with t-value equal to 11.9 and alpha equal to .000. Results indicate that a module on listening skills should be included in the course. Professors can state that listening is important, but we need to teach it if we want results. Also, to reinforce the importance of listening skills, the professors will include listening skills as part of the final grading process in future courses. The data collected during this semester will serve as a base for comparison for the listening training next semester. We will improve the internal validity of our testing by including other courses as control groups. For example, we may, with the cooperation of other professors, give the pre-tests and post-tests to students in other classes. Theoretically, the scores on those post-tests should not go up significantly. The only scores predicted to rise are the ones for the students receiving the "treatment," i.e., the Systems Analysis Methods course. 7. REFERENCES Angelo, Thomas, A. and Cross, Patricia K., 1993, Classroom Assessment Techniques. Jossey-Bass Publishers, San Francisco, CA. Diamond, Robert M., 1998, Designing & Assessing Courses & Curricula. Jossey-Bass Publishers, San Francisco, CA. Fisher, C.W., 1993, "An Empirically Based Model for Enhancing the Development and Retention of Listening Skills in Information Systems College Students." Proceedings, the International Association of Information Management (IAIM), pp 81 - 102 Dallas, TX. Gruber, G. R., and E. C. Gruber, 1995, Monarch's Preparation for The GMAT. Monarch, New York, NY. Hanchey, Cindy Myer, Winter 1995-96, "An Assessment Model for Surveying Graduates." Journal of Computer Information Systems, 36(2), pp. 67-75. Hoffer, J. A., George, Joey F., and Valacich, Joseph S 1996, Modern Systems Analysis and Design. The Benjamin Cummings Publishing Company, Inc., Menlo Park, CA. Hoopes, Joan E., 1993, "Information Center Assessment." Doctoral Dissertation, SUNY Binghamton. UMI Dissertations. Middle States Association of Colleges and Schools (MSA), 1990, Characteristics of Excellence in Higher Education Commission on Higher Education, Philadelphia, PA. Middle States Association of Colleges and Schools (MSA), 1996, Framework for Outcomes Assessment. Commission on Higher Education, Philadelphia, PA. Mohagen, Barta and Finklebinder, Brenda, Spring 1994, "A Pilot Study to Determine Student Learning Outcomes in Fundamentals of Management Information Systems Course." Journal of Computer Information Systems, pp. 13-18. Palomba, Catherine A. and Banta, Trudy W.1999, Assessment Essentials. Jossey-Bass Publishers, San Francisco, CA. Richards, R.M., Winter 1995/96, "Are Programs of Assessment and Continuous Improvement Really Worth It?" Journal of Information Systems Education, 7(4), pp. 131-134. Sherr, Lawrence A. and Teeter, Deborah J., Fall 1991.Total Quality Management in Higher Education. Jossey-Bass Publishers, New York, NY. Wiggins, Grant, 1998, Educative Assessment. Jossey-Bass Publishers, San Francisco, CA. 8. APPENDIX A Correlation Coefficients PRDFD PRDG PRIV PRTST PRLIS PSTDFD PSTDG PSTIV PSTLIS PSTST PRMC PSTMC PRDFD 1.0000 0.1581 0.1314 0.3545 -0.0665 0.0370 0.2274 -0.2406 -0.1592 -0.0805 0.1712 -0.0908 PRDG 0.1581 1.0000 0.3026 0.677 ** 0.1679 -0.1407 0.3452 -0.0228 0.3328 0.0530 0.4174 * 0.0379 PRIV 0.1314 0.3026 1.0000 0.705 ** 0.0518 0.1398 0.1273 0.1202 0.0570 0.2883 0.473 ** 0.2391 PRTST 0.3545 0.677 ** 0.705 ** 1.0000 0.1372 -0.0896 0.2631 0.0951 0.2282 0.3030 0.879 ** 0.3141 PRLIS -0.0665 0.1679 0.0518 0.1372 1.0000 0.1859 -0.0684 -0.0561 0.6533 ** 0.0061 0.1318 0.0036 PSTDFD 0.0370 -0.1407 0.1398 -0.0896 0.1859 1.0000 -0.1201 -0.2265 -0.0104 0.3576 -0.1480 0.0730 PSTDG 0.2274 0.3452 0.1273 0.2631 -0.0684 -0.1201 1.0000 0.1125 0.1111 0.2477 0.1099 0.2254 PSTIV -0.2406 -0.0228 0.1202 0.0951 -0.0561 -0.2265 0.1125 1.0000 0.1160 0.3733 * 0.1968 0.2162 PSTLIS -0.1592 0.3328 0.0570 0.2282 0.653 ** -0.0104 0.1111 0.1160 1.0000 0.4008 * 0.2310 0.3610 PSTST -0.0805 0.0530 0.2883 0.3030 0.0061 0.2477 0.3733 * 0.4008 * 0.3413 1.0000 0.3812 * 0.9294 ** PRMC 0.1712 0.4174 * 0.473 ** 0.879 ** 0.1318 -0.1480 0.1099 0.1968 0.2310 0.3812 1.0000 0.4376 PSTMC -0.0908 0.0379 0.2391 0.3141 0.0036 0.0730 0.2254 0.2162 0.3610 0.929 ** 0.4376 * 1.0000 PRDFD is PRe Data Flow Diagram PRDG is PRe Data Gathering Strategy PRIV is PRe InterViewing Problem PRTST is Overall PRe TeST Grade Score PRLIS is PRe LIStening Problem PSTDFD is PoST Data Flow Diagram PSTDG is PoST Data Gathering Strategy PSTIV is PoST InterViewing Problem PSTLIS is PoST LIStening Problem PSTST is Overall PoST TeST Grade Score PRMC is PRe Mutliple Choice Test PSTMC is PoST Mutliple Choice Test 2