Integrating a Web-Based Intelligent System into an Accounting Information System Course to Teach the Technique of Internal Control Evaluation Chuleeporn Changchit1 Department of Finance, Economics, & Decision Sciences, Texas A&M University Corpus Christi, Texas 78412, USA Abstract The issue of how to increase teaching effectiveness is a major concern to most educators. Understanding how to encourage a student to learn instead of memorize the presented material is one of the greatest problems faced in teaching. The nature of the topic taught contributes to this difficulty where a need for deep understanding may conflict with the need to understand the breadth of the topic. Evaluation of internal controls2 is part of an accounting course. For each accounting cycle in an organization, over 100 identifiable weaknesses may occur. Students may feel overwhelmed by this and try to memorize rather than understand the weaknesses and this affects their ability to effectively grasp the topic. This is a major issue in teaching internal control evaluation. This research identifies issues in using a Web-based intelligent system to teach students how to detect weaknesses in an internal control system. Such a system might prove to be highly beneficial to both students and educators. Keywords: Intelligent System, Expert System, Internal Control Evaluation, Knowledge Transfer, Web-Based Application 1. INTRODUCTION For many years, educators have been trying to find a way to effectively teach internal control evaluation. In order to increase the effectiveness of transferring the internal control evaluation knowledge, it is crucial to understand (a) the willingness of a student to learn, (b) how he or she learns, and (c) how he or she is able to improve his or her performance after such learning. Typically, there are two major issues: (1) how to make students actually learn and (2) how to improve their attitude and confidence toward the learning process. ( Transferring an auditor's internal control evaluation knowledge is very challenging. The nature of the topic contributes to this difficulty. For each accounting cycle, more than 100 weaknesses can occur in an organization. Students are often overwhelmed with examples of potential internal control weaknesses that are possible in one organization. In an attempt to address this issue, a Web-based intelligent system was developed. This system is aimed at teaching students how to evaluate an internal control system rather than simply memorizing all potential internal control weaknesses. 2. BACKGROUND AND RELATED LITERATURE The internal control evaluation process is a complex decision process that requires numerous qualitative judgments (Dorr et al 1988; Weber, 1999). Due to the large number of potential internal control weaknesses, it is not trivial and cannot be solved with common sense. Previous research findings have shown that intelligent systems could be used to transfer internal control evaluation knowledge from an expert to a non-expert (Changchit et al 2001 (a), 2001 (b); Dorr et al 1988; Gadh et al, 1993; Gal and Steinbart 1992; Graham et al 1991; Maureen 2001; Messier 1995; Odem and Dorr 1995; O'Leary and Watkins 1989). The results reported that subjects who practiced making decisions with the aid of a system were better and quicker at reaching decisions than subjects who practiced without the support of the system (Eining and Dorr 1991; Fedorowicz et al 1992; Meservy et al 1986). However, the literature contains no example of a system developed to facilitate the learning of the internal control evaluation technique via a Web-based technology. Unlike previously developed systems and other on-line teaching techniques, the system developed in this research provides an interactive learning experience. The integrated learning environment is designed to provide an effective process for teaching students how to evaluate internal control systems. Typically, students' performances are improved when interactivity is prominent. Therefore, a significant aspect of this system lies in the interactive nature of the system. The system does not only present students with text or other types of multi-media, but students are also required to interact with the system by entering the data into the system. The system will lead them step-by-step on how to detect internal control weaknesses. When the weakness is detected, it is reported immediately. The system also gives an open-enrollment atmosphere to students. The Web-based feature allows students to learn at their own pace. They can practice with the system at their convenience--during the workday, at night, or on weekends. In addition, special features were built into the system that allow the instructor to monitor students' performances. The instructor is able to keep track of how many times and how long each student has practiced with the system. All of this statistical data helps provide guidance to the instructor in relating students' efforts with the improvement in their performances. Table 1 below presents nine hypotheses tested to examine the value of the system. Ha Descriptions H1 There is no difference between participants in the Intelligent System (IS) Group and the Traditional Lecture (TL) Group on the accuracy of detecting internal control weaknesses. H2a There is no difference between participants in the IS Group and the TL Group on their perceptions about the difficulty of the task (i.e., detecting internal control weaknesses). H2b There is no difference between participants in the IS Group and the TL Group on the satisfaction with their accuracy in detecting internal control weaknesses. H2c There is no difference between participants in the IS Group and the TL Group on the perceptions on their internal control knowledge. H2d There is no difference between participants in the IS Group and the TL Group on their interest in detecting internal control weaknesses. H2e There is no difference between participants in the IS Group and the TL Group on their perceptions about the difficulty of learning internal control evaluation. H3a On the average, participants practicing with the Web-based intelligent system perceive that it is not difficult to use the system. H3b On the average, participants practicing with the Web-based intelligent system perceive that they can learn how to detect internal control weaknesses via the Web-based intelligent system. H3c On the average, participants prefer to learn how to detect internal control weaknesses via the Web-based intelligent system than via a traditional lecture (i.e., via white board and slide presentation). Table 1. Hypotheses Hypothesis H1 was investigated to examine if the learning occurs while users were practicing with the system. It examines whether a participant's accuracy in detecting internal control weaknesses improves after being practiced with a Web-based intelligent system. Accuracy of decision-making is examined as a measure of the system's effectiveness (Eining and Dorr, 1991). In order to examine the effectiveness of the system, participants' improvement in the accuracy scores between the subjects who practiced with the intelligent system (the IS Group) and the subjects who were taught with traditional lectures (the TL Group) was compared. Hypotheses H2a - H3c are based on the Technology Acceptance Model (TAM). The TAM provides a basis for explaining the determinants of computer acceptance and user behavior (Davis et al., 1989). The model suggests that computer usage is determined by two criteria: perceived usefulness and perceived ease of use. Perceived usefulness is defined as the prospective users' subjective probability that using a specific application system will increase his or her job performance within an organizational context. Perceived ease of use is defined as the degree to which the prospective user expects the target system to be free of effort (Davis et al., 1989). 3. RESEARCH METHODOLOGY An experiment was conducted to examine the impact of using the Web-based intelligent system to teach students how to evaluate internal control systems. Two teaching techniques were used as values of independent variables: traditional lecture and the Web-based intelligent system. Three response variables were (1) Effectiveness, (2) Participant Perception with the Task, and (3) Participant Satisfaction with the System. Independent Variables 1) Traditional Lecture - With the traditional lecture, an overview lecture is given on the importance of internal controls to an organization. Examples of weaknesses and how they affect the organization are also covered. Then, subjects are given a case study describing a scenario about a fictitious organization. The case study contains ten internal control weaknesses. The instructor leads the discussion on the internal control weaknesses in the case and why such events are considered internal control weaknesses (i.e., how such events could be harmful to the organization). 2) The Web-Based Intelligent System for Internal Control Evaluation - In order to study whether the Web-based intelligent system can be developed to enhance students' learning in internal control evaluation, such system was developed using the ASP technology and the Visual Basic Script language. Once the development of the system is completed, an experiment was conducted to examine its utility. Knowledge for the Web-based intelligent system was acquired via a six-month series of interviews with an expert. The expert is an auditor in an international firm and has more than ten year of experience in evaluating clients' internal control systems. The expert was asked to identify all potential weaknesses that can occur in the sales and collection cycle of a medium-size merchandising organization. The result was a list of 126 internal control weaknesses. The expert was then asked to describe, in detail, the techniques and processes he used to discover each of these weaknesses in a client's internal control system. The reasons for each decision-making heuristic were also acquired in an attempt to develop a Web-based intelligent system that would be able to emulate both the expert's knowledge and his reasoning behavior. Knowledge acquired from the expert was represented in sets of rules in the intelligent system. Upon processing these rules, an inference engine infers a potential internal control weakness recommendation. This recommendation identifies significant internal control weaknesses discovered in the situation being evaluated and indicates resulting exposures that could occur in a user's organization. In order to validate the system, often considered the cornerstone of intelligent system evaluation (Back, 1994; Jancura, 1990), the expert was asked to critique the expertise captured in the rule sets in order to ensure the validity of the system. In addition, four test cases were generated from the manipulation of several cues for detecting the potential weaknesses in an internal control system of the sales and collection cycle. The expert was asked to evaluate each test case and detect its potential internal control weaknesses. Reasons for each potential weakness were also requested. The prototype was used to detect the potential weaknesses and offer reasons of such weaknesses as well. The results were then compared. The human expert and the intelligent system identified similar weaknesses for each of the test cases. Where there were discrepancies, the expert reconsidered his responses and agreed that the intelligent system's responses were indeed correct. Interestingly, this illustrates that an intelligent system can sometimes be useful even to an expert (e.g., to double check the expert's reasoning). Response Variables 1) Effectiveness -- Accuracy of decision-making is examined as a measure of the system's effectiveness (Eining and Dorr 1991; Sharda et. al. 1988). 2) Participant Perception with the Task -- A post-experiment questionnaire was used to measure participants' perceptions with the task. Seven-point Likert scales were used in the questionnaire. The following questions were asked to measure their attitude toward the task. * On a scale of 1 (very difficult) to 7 (very easy), how difficult was it to detect potential weaknesses in the case study? * On a scale of 1 (very unsatisfactory) to 7 (very satisfactory), how satisfied were you with your accuracy in answering the case study? * Currently, on a scale of 1 (very poor) to 7 (very good), what is your knowledge about internal control? * On a scale of 1 (very boring) to 7 (very interesting), how interesting was the task you performed? * On a scale of 1 (very difficult) to 7 (very easy), how difficult was it to learn how to evaluate internal control? 3) Participant Satisfaction with the System -- A post-experiment questionnaire was used to measure participants' satisfaction with the system. Seven-point Likert scales were used in the questionnaire. The following questions were asked to measure satisfaction. * On a scale of 1 (very difficult) to 7 (very easy), how difficult was it to use the Web-based intelligent system? * On a scale of 1 (strongly disagree) to 7 (strong agree), I believe that practicing with the Web-based intelligent system can help me learn how to detect internal control weaknesses. * On a scale of 1 (strongly disagree) to 7 (strong agree), I prefer to learn how to detect internal control weaknesses via the Web-based intelligent system than via a traditional technique (i.e., via a white board and slide presentation). Subject Because the main purpose of this study is to investigate whether a Web-based intelligent system can be devised to facilitate the transfer of knowledge to novice subject, using students as the subjects in this study is deemed appropriate. Subjects are students enrolled in the Business College at a University in the Midwest region of the US. Table 2 presents their demographic information. Demographic Traditional Lecture Group Intelligent System Group No. of Subjects 16 161 Average GPA 3.12 3.08 Year 13 Senior, 3 Junior 12 Senior, 4 Junior Majors 13 Accounting, 2 MIS, 1 Other 14 Accounting, 1 MIS, 1 Other 1 Initially there were 17 subjects. One subject dropped before the experiment started. Table 2. Subjects' Demographic Internal Control Case Studies Four case studies (A, B, C, and D) were generated from the manipulation of several cues for detecting the potential internal control weaknesses. These cues were obtained from a review of auditing texts, accounting texts, and input from accounting professors and experienced auditors. The scenario in each case dealt with the adequacy of internal control over a company's sales and collection cycle. Also, each case included background information about the fictitious company and a partial organization chart. Each case contained ten potential internal control weaknesses. Three experienced auditors and three managers were asked to pilot test these cases to ensure their similarity with respect to the degree of difficulty in detecting the potential internal control weaknesses. Revisions to the cases were made based on feedback provided. Experiment Task One class before the experiment started, subjects in both groups were given a lecture of an overview of internal controls (e.g., the definition of internal controls, the importance of internal controls). This lecture did not cover anything about the internal control evaluation. In addition, in order to examine the homogeneity of the subjects on knowledge of internal controls, prior to the experiment, subjects in both groups were given a pre-test. The pre-test consisted of a case (Case A) describing a scenario in an organization on the policy and procedure of the sales and cash receipt cycle. Subjects were asked to identify ten weaknesses from the case. A questionnaire was also given to measure their perceptions about the task performed as well as their attitudes on their internal control knowledge. The experiment was conducted in an isolated, controlled environment in a college laboratory. Subjects in the Traditional Lecture (TL) Group were taught the technique of internal control evaluation via a traditional lecture, i.e., using white board and PowerPoint presentation. Subjects in the Intelligent System (IS) Group were instructed to practice with the Web-based intelligent system for internal control evaluation. After the experiment, subjects in both groups were given a post-test, also consisting of a case (Case D) describing a scenario in an organization on the policy and procedure of the sales and cash receipt cycle. As with the pre-test, subjects were asked to identify ten weaknesses from the case. Then, a questionnaire was given to measure their perceptions about the task performed as well as their attitudes on their internal control knowledge. In addition, additional questions were added to examine the subjects' satisfaction with the intelligent system. 4. EXPERIMENTAL FINDINGS Since there is no direct control on the assignment of subjects to each group, an ANOVA was performed on the pre-test's accuracy score to gauge the homogeneity of the subjects' internal control knowledge between the Traditional Lecture (TL) Group and the Intelligent System (IS) Group. Although the mean accuracy score in the TL Group was slightly higher that the mean of accuracy score in the IS Group ((TL Group = 58.59 vs. (IS Group = 52.78), at an alpha level of 0.05, no significant difference was found between the groups' initial levels of internal control knowledge (p-Value = 0.55). In order to examine the value of the system in facilitating the knowledge transfer, an improvement in participants' accuracy score and perceptions with the task between the TL Group and the IS Group was compared. T-tests were employed to test hypotheses about the means for the two groups for each response variable: (i) accuracy score (H1) and (ii) participants' perceptions with the task (H2a, H2b, H2c, H2d, and H2e). To investigate participants' satisfaction with the system, t-tests were performed to test hypotheses H3a, H3b, and H3c. The major findings for the performances and attitudes of participants in both groups are summarized in Table 3. Ha Response Variables TL Group System Group P-Value Pre-Test Post-Test Imp. Pre-Test Post-Test Imp. H1 Accuracy Score 58.59 60.16 1.57 52.78 79.86 27.08 0.045* H2a Task Difficulty 2.13 3.87 1.74 1.83 3.56 1.73 0.66 H2b Satisfaction with Accuracy 3.00 4.40 1.40 2.56 4.11 1.55 0.52 H2c Task Interesting 3.81 3.73 -.08 3.56 3.5 -.06 0.748 H2d Internal Control Knowledge 3.50 4.07 0.57 2.83 4.22 1.39 0.017* H2e Difficulty of Learning 3.67 3.39 0.49 H3a Difficulty of the System 5.28 0.00002** H3b Usefulness of the System 4.22 0.48 H3c System Preference 4.39 0.26 * represents the significant at alpha = 0.05 ** represents the significant at alpha = 0.01 Table 3. Experimental Finding Hypothesis H1 examined whether there was a difference between these two groups in detecting potential internal control weaknesses in the case provided. The results show that participants in the IS Group have a greater improvement in accuracy score the participants in the TL Group ((TL Group = 1.57 vs. (IS Group = 27.08). The t-Test confirms that (alpha level of 0.05), after the experiment, there was a significant difference in the improvement of the participants' accuracy in detecting an internal control weakness between participants in the TL Group versus participants in the IS Group (p-value = 0.045). Regarding hypotheses H2a, H2b, H2c, H2d, and H2e, participants' answers to the questionnaires reveal the following. * Task Difficulty - participants in both groups perceived that the task was a little bit easier in the post-test compared to the pre-test. However, there is no significant difference between the improvements on their perceptions about task difficulty between the two groups. The result implies that at least the use of the system to transfer the knowledge was as good as the knowledge transfer via a traditional lecture. * Satisfaction with Accuracy - participants in both groups were more satisfied with their accuracy in the post-test compared to the pre-test. However, there is no significant difference between the improvements in their satisfaction with their accuracy. * Task Interesting - participants in both groups perceived the task to be less interesting in the post-test compared to the pre-test. However, there is no significant difference between the two groups. * Internal Control Knowledge - participants in both groups felt more confident in their internal control knowledge after exposure to the teaching techniques. However, the t-test at an alpha of 0.05 indicates that the increase in confidence level for subjects in the IS Group are significantly higher than the increase in confidence level for subjects in the TL Group (p-value = 0.017). * Difficulty of Learning - although there is no significant difference in students' perceptions on the difficulty of learning internal control evaluation. It is interesting to note that the score in the TL group is slightly higher than the score in the IS group. This is not surprise because it is expected that the students learning the concept of internal control evaluation through practicing with the system will have to spend more effort (working with the system beyond the class time) compared to the students who learned the concept via the traditional lecture (listen to the lecture for two consecutive classes). Regarding hypotheses H3a, H3b, and H3c, these questions were asked only of the subjects in the IS Group. The score is compared to the neutral response (i.e., a score of 4 on the 7-point Likert scale). Participants' answers to the questionnaires reveal the following. * Difficulty of the System - participants in the IS Group perceive that it is not difficult to use the system (p-value = 0.00002). * Usefulness of the System - although participants in the IS Group slightly agreed that the system was useful, it is not significantly different from feeling neutral towards the system. * System Preference - although participants in the IS Group slightly preferred the intelligent system for learning the internal control evaluation technique, it is not significantly different from feeling neutral towards the system. 5. CONCLUSIONS, LIMITATIONS, AND DIRECTIONS OF FUTURE RESEARCH This research is only the initial investigation into the use of a Web-based intelligent system to facilitate the transfer of an auditor's internal control evaluation technique to students. The primary finding of this study is that it is feasible to build such a system. The research findings also demonstrate that students who practiced with the system (the IS Group) learned more and had more confidence about their internal control knowledge compared to students in the other section (the TL Group). The IS Group also expressed that the system was not difficult to work with. These findings indicate that it is feasible and beneficial to use the Web-based intelligent system as a substitute or supplement to traditional techniques for teaching internal control evaluation. Based on the research findings above, the benefits of this system may be divided into three different areas. First, benefits to students -- the system helps students not only learn the technique of internal control evaluation, but also make them feel more comfortable learning such a technique. The system addresses issues of flexibility, quality of learning, and teaching efficiency. Students will be able to work through problems as needed to familiarize themselves with the technique of internal control evaluation. Second, benefits to the instructor -- the system allows an instructor to expand on what is currently being done with traditional teaching methods. The instructor was able to keep track of how many times, and how long, each student practiced with the system. All of the statistical data gathered provides guidance to the instructor on the relationship between a student's effort and the improvement in his/her performance. The system also allows the instructor to make better use of class time to go over those areas students struggle with the most. Third, benefits to the university -- the system helps facilitate the circulation of expertise in evaluating internal control systems to a wide number of students with the least constraints (if any). The system will result in a new way to increase the efficiency and effectiveness of teaching internal control evaluation, saving time and resources The scope of this research study, however, may limit the generalizability of the results in several respects. First, this research concentrates only on the evaluation of controls commonly found in the sales and cash receipts cycle. Second, it investigates internal control systems commonly found in the merchandising industry, but is not expected to handle novel (uncommonly different) accounting systems. Third, the knowledge of the intelligent system developed for this study is based primarily on one auditor, a partner of an international accounting firm. The resulting system closely represents his reasoning about internal control evaluation. Thus, the system's knowledge may be firm-specific or expert-specific. These limitations point to directions in which the research presented here can be extended by future investigations. Future research might try to investigate the results of using the intelligent system over a longer period of time (e.g., conducting additional sessions one week or month after the first three sessions). Another research avenue might investigate, in more detail, the useful functions incorporated into the system. For instance, it would be interesting to examine any differences in users' performance if one group is allowed to see the explanation capability of the intelligent system, while another group is not. In addition, it might be interested to find out whether the use of Web features, such as sound and animation, can enhance learning. Other research may examine if the layout of the contents (i.e., grouping internal control weaknesses according to the nature of transactions versus grouping them according to the basic internal control concept) can lead to differences in users' performances. It would also be interesting to study the feasibility of integrating such a system into a company's database so that the data could be retrieved directly from the source instead of having the users' input such data. Along that same line of research, a study could examine if there is a way to integrate the system into an accounting information system so that any weaknesses could be detected automatically while an accountant is processing the data. Finally, researchers might investigate whether it is feasible to increase the usefulness of the system by developing it as a template for other types of knowledge. Then, other research could be conducted to examine the value of the system for other courses requiring interactive learning. 6. REFERENCES American Institute of Certified Public Accountants (AICPA). Codification of Statements on Auditing Standards. New York, NY: AICPA, 1996. Back Barbro, 1994, "Validating an Expert System for Financial Statement Planning," Journal of Management Information Systems, 10(3), pp. 157-177. Changchit, Chuleeporn, Clyde W. Holsapple, and Donald L. Madden, 2001(a), "Supporting Managers' Internal Control Evaluations: An Expert System and Experimental Results", Decision Support Systems, 30(4), pp. 437-449. Changchit, Chuleeporn, Clyde W. Holsapple, and Ralph E. Viator, 2001(b), "Transferring Auditors' Internal Control Evaluation Knowledge to Management", Expert Systems with Application, 20(3), pp. 275-291. Davis, Fred. D., Richard Bagozzi, Paul R. Warshaw, 1989, "User Acceptance Of Computer Technology: A Comparison Of Two Theoretical Models," Management Science, 35(8), pp. 982-1003. Dorr, Patrick. B., Martha M. Eining, James E. Groff, 1988, "Developing an Accounting Expert System Decision Aid for Classroom Use," Issues in Accounting Education, pp. 27-41. Eining, Martha. M., and Patrick. B. Dorr, 1991, " The Impact of Expert System Usage on Experiential Learning in an Auditing Setting," Journal of Information Systems, 5(1), pp. 1991. Fedorowicz, J., E. Oz, and P. D. Berger, 1992, "A Learning Curve Analysis of Expert System Use," Decision Sciences, July/August, pp. 797-818. Gadh, V. M., R. Krishnan, and J.M. Peters, 1993, "Modeling Internal Control and Their Evaluation," Auditing: A Journal of Practice & Theory. 12(Supplement), pp. 113-129. Gal, G. F. and Paul J. Steinbart, 1992, "Interface Style and Training Task Difficulty as Determinants of Effective Computer-Assisted Knowledge Transfer," Decision Sciences, 23, pp. 128-143. Jancura, E. G., 1990, "Expert Systems: An Important New Technology For Accountants," The Woman CPA, 52(2), pp. 25-28. Meservy, R. D., A. D. Bailey, and P. E. Johnson, 1986, "Internal Control Evaluation: A Computational Model of The Review Process," Auditing: A Journal of Practice & Theory. 6(3), pp. 44-74. Messier, W. F. Jr., 1995, "Research In and Development of Audit Decision Aids," In Judgment and Decision-Making Research in Accounting and Auditing. Edited by R. H. Ashton and A. H. Ashton. New York, NY: Cambridge University Press. Mascha, Maureen F., 2001, "The effect of task complexity and expert system type on the acquisition of procedural knowledge," International Journal of Accounting Information Systems, 2(2), pp. 103-124. Odem, M. D., and Patrick B. Dorr, 1995, "The Impact of Elaboration-Based Expert System Interfaces on De-Skilling: An Epistemological Issue, " Journal of Information Systems, 9(1), pp. 1-17. O'Leary, D. E. and P. R. Watkins, 1989, "Review of Expert Systems in Auditing," Expert Systems Review for Business & Accounting, 9(2), pp. 3-22. Weber, Ron, 1999, Information Systems Control and Audit. Upper Saddle River, NJ: Prentice-Hall. 1 cchangchit@cob.tamucc.edu 2 Internal control is defined by the American Institute of Certified Public Accountants (AICPA) as a process - effected by an entity's board of directors, management, and other personnel - designed to provide reasonable assurance regarding the achievement of objectives in the following categories: reliability of financial reporting, effectiveness and efficiency or operations, and compliance with applicable laws and regulation. 1