A Method of Measuring Fitness of Learning Tasks to Blackboard Technology Richard Orwig Sigmund Weis School of Business Susquehanna University 514 University Ave Selinsgrove, PA 17870 Abstract Research in Information Systems task/technology fitness has largely assumed that the business task is well-defined. This research demonstrates that even a simple task such as “Complete an IT Project Management Course” can be defined with great ambiguity among students. Thirty-three students were assigned to identify and define activities associated with completing a semester course. Their resulting models were merged to comprise a super-list of sixty-nine unique activities. Cumulatively, this defines the tasks associated with learning the material of the course. The functions of Blackboard Version 5, a technological learning environment, were identified and mapped into each task. The resulting many-to-many mapping demonstrates the complexity of attempting to determine the degree of fitness of a technology such as a Blackboard learning environment to a “simple” task as completing an IT Project Management course. We refer to the gap between the task definition and the system functionality as the “gulf of ambiguity.” Further research is needed to better come to consensus upon the task definition and identify a value measure to each mapping between the tasks and functions. Keywords: task-technology fitness, activity modeling, requirements engineering, computer learning environment 1. INTRODUCTION Motivation for this research is grounded in experience in attempting to implement a “theory-in-use” that was instituted by the Department of Defense (DoD) in 1991 called “Corporate Information Management.” Along the lines of business process reengineering (BPR), the DoD effort was a $5B project to better use information systems to improve logistic services and reduce costs. An unusual component of this effort was that business functional experts were directed to determine functionality through construction of business activity models rather than systems analysts directing the design and development. Two business activity models were to be constructed: an As Is model to clearly define the business task in terms of activities and then a To Be model based upon changes in business tasks resulting from streamlining ideas surfaced through the As Is modeling as well as implementation of technologies at appropriate places to enhance output of the tasks or reduce costs associated with those tasks. Unfortunately, the Corporate Information Management effort collapsed under its own weight and the transfer from the Bush to Clinton administrations (Taylor 1996). However, the “theory-in-use” that drove this effort is intuitively sound and worthy of further study. The paper applies this “theory-in-use” to a more simple managerial context: making a rational decision concerning using a learning environment such as Blackboard© for a course by attempting to determine how well the functions of the information system (Blackboard) fit the learning tasks associated with completing the requirements of a course. In this instance, as an exercise in business activity modeling, we had our students individually construct activity models of the tasks (activities) that he or she must do to complete the Information Technology Project Management course that they were in at the time. We merged all of the student models into one set of activities and mapped them with functions of the Blackboard learning environment. The goal of this research is two-fold. The first goal is to apply the theory in use to a well-known instance (taking an IS course). The second goal is to evaluate the modeling method used for its ability to model business activities and technological support within the same model. The next section describes the “theory-in-use” motivating this effort. The following section describes the process and data collected. Section 4 describes the results of the study and Section 5 concludes with recommendations for future study. 2. THEORY-IN-USE The U.S. Department of Defense (DoD) was very much aware through its own experience as well as well-known research reports such as produced by the Standish Group that successful implementation of information systems was rare. Consistent with reengineering ideas espoused during the late 1980’s and early 1990’s (Davenport and Short 1990; Hammer 1990), the DoD provided a viewpoint/theory-in-use, a method, and a $5B budget to reinvent the activities required to support our fighting forces. The effort was called “Corporate Information Management” (CIM) and was in implementation of the following theory-in-use: Figure 1 The CIM effort was to be driven by functional experts instead of technical efforts. Functional experts were defined to be the actual people who performed the DoD activities associated with supporting troops instead of DoD or consulting systems analysts. Thus, there were two major perspectives associated with the effort: business perspective and technical perspective. These two perspectives are represented in Figure 1 as the bottom two layers of the diagram. Using a top-down approach, the DoD used its Vision, Mission, Goals, and Objectives statements as the context for building a high-level activity model that defined separate activities required to support the complete DoD. This DoD corporate model was then used as a starting As Is model for constituent DoD organizations to further decompose the corporate model As Is activities. Each organization was to follow the initiative using guidelines provided in (DoD_8020.1-M 1992). These are the major steps: 1. Create a model of the current business activities (As Is), 2. Inventory existing legacy systems that support that business activity model, 3. Reengineer the current business activity model to create a To Be activity model, 4. From the set of legacy systems, select the best system to be used for DoD-wide implementation. This "migration system" was to be the only DoD-funded system to be supported during the time that it would take for a "To Be" system to be developed. An example of the facilitation of this effort can be found in (Dean, Lee et al. 1994; Orwig, Dean et al. 1998; Dean, Orwig et al. 2000). This theory-in-use driving the CIM effort suggests that consensus upon a model of activities or tasks associated with a given domain is feasible and useful. In particular, it should be useful as the determinant of degrees of fitness between activities associated with the business domain and any information system purporting to support that domain. Business systems analysts intuitively perform this analysis by mapping their understandings of the business domain with the functionalities of candidate information systems solutions. In short, the activity model defines the problem space and the functionality of an information system defines a possible solution space. A rational decision-making process would suggest that business analysts intuitively map activities with functions and select information systems that best fit together. Because of the complexity of this decision, other decision-making processes come into play (power and influence of superiors of the decision-maker, time constraints, relationships with vendors of the systems, etc.). While recognizing these other influences into the decision-making process of adopting information systems solutions, we wish to move the theory-in-use approach of the DoD into the research domain for perhaps making the analysis of the fitness of any information system to a business more rational. In short, we would like to put more “engineering” into the “requirements engineering” process. Also, while the BPR concept has developed a bad connotation due to the ambiguities and complexities of implementation, the philosophy and underlying theory is still useful (Taylor 1996; Stoica, Chawat et al. 2003). 3. ACTIVITY DATA ASSOCIATED WITH COMPLETING AN MIS COURSE One of the assignments of an Information Technology Project Management course required students to build an activity model to demonstrate their understanding of a business domain. Students were instructed to take a top-down, systems viewpoint and decompose a business activity to at least two levels of specificity. They were trained in basic IDEF0 method which provides a simple modeling syntax: activities are verb/noun labels with descriptions with associated noun-identified inputs, outputs, controls, and mechanisms. Students could select a business with which they were familiar taking the perspective of the business owner or they could model the activities of completing the IT Project Management course taking the perspective of a student. One benefit of using IDEF0 method is that, once a model is built, “what if” scenarios can be described by changing mechanisms to any activity in the model. For example, the “Discuss Questions with Fellow Student” activity may have e-mail and telephone mechanisms in the As Is model (mechanisms help transform a input, Unclear Point, into an output, Understood Point for this hypothetical “Discuss Questions” activity). Adding a discussion space that accompanies a learning environment such as Blackboard adds a new mechanism to that activity and may influence the manner in which that activity is decomposed in a manner that provides better “Understood Points” or lowers the cost of producing “Understood Points.” The basis for these types of scenarios is a good activity model. By making mechanisms just one attribute of an activity and associating technology as a mechanism, this reduces the myopia sometimes associated with the technological imperative associated with adopting new technology for technology’s sake. Using an IDEF0 representation, adopting new technology is explained in terms of what will happen to the business activities affected by its implementation either directly or through changes in the decomposition of any given activity. Thus, another reason for requiring IT project management students to build activity models is that the model can be an effective communication tool to business managers in that changes can be shown in business activity terms instead of the geek speak associated with data flow diagrams and/or low-level use cases. Over the course of three sections of IT project management, 33 students elected to build activity models using the student perspectives of completing the course. Students were required to decompose activities into at least three subactivities. Since they were also required to decompose to at least two levels, the minimum number of activities associated with their models was 13 (1 + 3 subactivities + 9 sub-subactivities). While students were required to fully populate each activity with inputs, outputs, controls, and mechanisms, we will focus only on the activities in this paper. Appendix A is the activity model of one of the better students. As can be seen from Appendix A, the models contain hierarchical arrangements of unique activities. Students frequently identified identical activities (e.g., “Study for Exam”). We went through all 33 models listing all activities and combining common ones to create a pool of 69 unique activities identified and associated with completing the IT project management course. Many of the students used the syllabus to identify and describe the activities that they performed to deliver the two exams, three assignments, and project required for the course. The list of 69 activities is given in the leftmost column of Appendix C. The center column of Appendix C lists the number of students who identified the activity in the left column in their models. A blank entry merely means that only one student identified that activity. Thus, for the purpose of analysis, these 69 activities comprise an activity model for completing the course. We then used the Blackboard© 5.0 student manual to identify the functions available to a student. The list of functions was numbered and a table of these numbers and functions is shown in Appendix B. We then attempted to map the functions to each of the 69 tasks using a hypothetical “can this function serve as a mechanism to this activity?”. As instructors of the course, we imagined using the functionality of Blackboard as a mechanism for each activity. If Blackboard could be added or replace an existing mechanism of an activity such that it would cause changes to occur in the children of the target parent activity, it was deemed to be possibly affecting that parent activity. 4. ANALYSIS RESULTS Consensus among students regarding the activities required to complete the course is weak. Only 31 activities of the 69 were identified by more than one student. Consensus does seem apparent regarding Taking Exams, Attending Class, Completing Homework, and Reading the Textbook. Earlier research in DoD activity modeling used groupware with as many as 25 people building one model (Dean, Lee et al. 1994). These earlier models benefited by the same-time-same-place collaboration afforded by the groupware. A better student model may result from using these 69 activities as input to a student groupware session. Of the 69 activities, 42 are possibly affected by using Blackboard functionality. This accounts for 61% of the activities. 39% of student activities are unaffected by using a learning environment from the students’ perspective. One dimension for a measure of fitness of the technology to the task may be the number of activities that are affected by the technology. Another dimension for a measure of fitness may lie in the distribution of functions across the list of activities. Some activities are potentially affected by many functions of Blackboard while others are potentially affected by only one. The degree of affect is not clear from this analysis but is hinted by the differences in the distribution. A hierarchical business model may suggest that the greater number of functions affecting the higher-leveled activities will be worth more in terms of potentially enhancing output and/or reducing costs. This suggests the greater risks involved in affecting higher-order activities in the model due to the number of changes that may be required in different decompositions of those activities to accommodate the new technology. 5. CONCLUSION AND RECOMMENDATION Any measure of fitness between a business task and technology supporting that task requires some consensus upon the definition of the business task. This research-in-progress indicates that consensus upon a hierarchically organized model is necessary for better determining fitness of learning environment software upon a course. Systems analysts intuitively map their understandings of the affects of technology upon their business domain. However, the business domain is complex with multiple layers of abstraction. This simple task of completing an information systems course identified 69 activities that need to be arranged into a hierarchical model. Many parameters affect our decisions for selecting a given information system from a set of candidate systems. We give lip service to a given selection in that it “fits” best of all candidates; yet we cannot provide a clear and rational measure of the fitness and let informal processes guide these decisions (Chan 2002). More research is needed in modeling business activities in a hierarchical manner to determine where in the As Is model technology may be inserted and what affects it will have on lower level (more detailed) activities. Support for greater research in this direction is provided by (Goodhue, Grazioli et al. 2002). 6. REFERENCES Chan, Y. E. (2002). "Why Haven't We Mastered Alignment? The Importance of the Informal Organization Structure"." MIS Quarterly Executive 1(2): 97-112. Davenport, T. H. and J. E. Short (1990). "The New Industrial Engineering: Information Technology and Business Process Redesign." Sloan Management Review. Dean, D., J. Lee, et al. (1994). "Technological Support for Group Process Modeling." Journal of Management Information Systems 11(3): 43-64. Dean, D., R. Orwig, et al. (2000). "Facilitation Methods for Collaborative Modeling Tools." Group Decision and Negotiation 9(2). DoD_8020.1-M (1992). Functional Process Improvement: Functional Management Process for Implementing the Information Management Program of the Department of Defense. Washington, DC, Office of the Secretary of Defense. Goodhue, D. L., S. Grazioli, et al. (2002). Beyond Task/Technology Fit: How Information Technology Affects Performance By Transforming the Task. Workshop for IS-CORE: Special Interest Group on Cognitive Research, Barcelona, Spain. Hammer, M. (1990). "Reengineering Work: Don't Automate, Obliterate." Harvard Business Review 68(4): 104-112. Orwig, R., D. Dean, et al. (1998). A Method for Evaluating Information Systems from WorkFlow Models: Results from a Case Study. 31st Annual Hawaii International Conference on Systems Sciences [HICSS-31], Maui, Hawaii, IEEE Press. Stoica, M., N. Chawat, et al. (2003). An Investigation of the Methodologies of Business Process Reengineering. Information Systems Education Conference (ISECON), San Diego, CA, EDSIG. Taylor, S. T. (1996). BPR Is Legacy of Pentagon's Troubled CIM Program, www.reengineering.com. 2002. APPENDIX A One student’s contribution: A0 Complete MIS 448 A1 Stay Organized A1.1 Keep a Calendar A1.2 Organize Work A1.3 Budget time A1.4 Visit Class Website Regularly A2 Seek Knowledge Proactively A2.1 Attend Class Regularly A2.2 Participate in Class A2.2.1 Listen in Class A2.2.2 Participate in Group Discussions A2.2.3 Ask Questions A2.3 Take Notes A2.3.1 Outline Material A2.3.2 Highlight Key Concepts A2.3.3 Draw Diagrams A2.4 Read Course Related Materials A2.4.1 Read Material for Future Lectures/Discussions A2.4.2 Read Material Not Required A2.4.3 Read Material Linked to the Class Website A2.5 Visit Related Websites A2.5.1 Download Programs/Files A2.5.2 Do Tutorials on the Web A2.5.3 Get Help/Answers From Internet Forums A2.6 Contact Someone if you are Absent A2.6.1 Contact Professor A2.6.2 Contact Classmate A3 Complete Assigned Tasks Adequately A3.1 Do Assignements/Projects A3.1.1 Get Aquainted with Task Instructions A3.1.2 Research the Task A3.1.3 Get Help A3.1.4 Complete Task A3.1.5 Review Task A3.1.6 Turn in Deliverable A3.2 Do Group Projects A3.2.1 Meet with Group Members A3.2.1.1 Meet with Group Introductory A3.2.1.2 Meet with Group During Project A3.2.1.3 Meet with Group to Combine and Review A3.2.2 Research Project Part A3.2.3 Complete Project Part A3.2.4 Turn in the Group Project A3.3 Study for Exams/Test/Quizes A3.3.1 ReRead Materials A3.3.2 Use Study Guide A3.3.3 Get Questions Clarified A3.4 Take Exams/Tests/Quizes A3.4.1 Put Your Identification on Test A3.4.2 Skim Over Exam/Test/Quiz A3.4.3 Answer Questions A3.4.4 Turn in Exam/Test/Quiz A4 Receive a Grade A4.1 Weight Grades of all Tasks A4.2 Calculate Grade of Weighted Tasks A4.3 Combine Weighted Task Grade with Other Grade-Based Criteria APPENDIX B: List of Blackboard 5.0 functions: No. Function 1 View Announcements 2 Maintain Calendar: manage course, organization, institution, and personal events 3 Tasks: post personal tasks and see tasks posted by supervisors for you 4 My Grades 5 Web Email 6 Send Email 7 User Directory: broadcast email to other users willing to post their e-mail address 8 Address Book: maintain contact information 9 Personal Information 10 Community (Discussion boards) 11 Services: external links/hand in homework 12 Academic Web Resources 13 The Web in general APPENDIX C: Mapping of tasks/functions: Activity No. Students citing task Blackboard Function budget time 1,2 dispose of class material when done download programs/files 1,3 drive to campus find place to read/study gather required tools 3 locate text reseller maintain calendar 2 note schedule changes 1,2 organize work 1,2,3,4 plan class requirements 1,2,3 plan class schedule 1,2,3 plan homework time 2 plan hw tasks reschedule work search for help/answers on web and via e-mail to prof 1,3,5,6,7,8,10 sleep visit class website All complete HW assignments 21 1,2,3,5,6,7,8,10,11,12,13 draw diagrams 3 read assignment materials 7 3 review HW assignment 4 3 deliver assignment 11 answer classroom questions ask questions 9 5,6,7,8 attend class 14 listen to lecture/pay attention 6 prepare for each class session 2 1,2,3,10,12,13 take class notes 12 compare problems w/ other students 5,6,7,8,10 contact classmate after missing class to see what was missed 5,6,7,8,10 exchange contact information 10 help others 1,3,5,6,7,8,10,12,13 introduce yourself to fellow students 10 participate in classroom discussions 10 complete tests 15 3 ensure comprehension 2 evaluate/outline material 2 plan study time 2 2 prepare for tests 10 1,2,3,4,5,6,7,8,10,11,12,13 read chapter summary review class notes 6 review ppt notes 4 1,3 review test/re-answer questions 2 1,2,3,4,5,6,7,8,10,11,12,13 review textbook notes study vocabulary study/review 8 1,2,3,4,5,6,7,8,10,11,12,13 complete discussion questions/perform chapter exercises 4 3,12,13 do web tutorials 3,12,13 read optional material 3 12,13 gain knowledge 1,2,3,4,5,6,7,8,10,11,12,13 deliver project 11 attend project team meetings 10 collect data for project 3,11,12,13 complete project 10 1,2,3,5,6,7,8,10,11,12,13 design project activity model 3,11 distribute tasks 3,5,6,7,8,10 form project team 2 integrate/finalize project 2 plan project work time 2 1,2,3,5,6,7,8,10,11,12,13 prepare status report 2 1,2,3,5,6,7,8,10,11,12,13 present project 7 review project guidelines 1,3,11 write final project report 5 write project proposal open textbook plan reading time 2 read textbook 14 take textbook notes 4