The Costs and Benefits of Learning Technologies Wendy Cukier1 Kenneth A. Grant Justyna Susla School of Information Technology Management, Ryerson University 350 Victoria Street Toronto, Ontario M5B 2K3 Abstract This paper examines the way in which the benefits and costs of learning technology have been addressed in recent years with a particular emphasis on identifying gaps between the discourse and the empirical data. The study suggests that while the benefits of learning technology have been discussed widely, there is limited empirical evaluation of the effects and very little discussion of the costs. It suggests that the promotion of learning technologies is, to some extent, driven by marketing rather than objective assessments of costs and benefits. It concludes that a more critical or questioning perspective on the claims and rigorous examination of the costs as well as the benefits would improve decision making. Rather than ignoring or marginalizing those who express concerns about learning technology we should address their concerns. Keywords: Information Technology, Education, Cost-benefit Analysis, Discourse 0. 1. INTRODUCTION Information and Communication Technology (ICT) decisions represent significant expenditures in every sector - often the largest single capital investment in a given organization. This level of expenditure has been growing over the last 20 years as ICT systems are applied in an ever-increasing range of applications within almost every industry sector. However, the results of these investments are less than clear. There have been extensive discussions of the "productivity paradox", which suggests that the impact of technology investments is not always evident at the broad societal level, or within organizations. This increased scrutiny of ICT investments has led to the development of an extensive body of knowledge on ICT investment management (for example, Weill & Broadbent, 1998). Industry studies have shown that an increasing proportion of private sector organizations are looking specifically at benefits measurement for ICT investments. The use of technology to support learning represents a major and growing investment for educational institutions as well as for students. While there is limited system-wide data available, expenditures on technology by institutions and students have been rising steadily at most institutions. More than half of the respondents in a US survey of academic institutions report increases in their institutions' academic computing budget for the academic years 2000/2001 and 2001/2002 (Green, 2001). For some, spending on IT as a percentage of academic expenditures has quadrupled in the past four years (Landry, 2000). Universities have been incurring this expenditure in a wide range of areas, such as administrative systems, course delivery software, campus-wide networks, etc. Over 200 North American universities and colleges have adopted "notebook computer" programs either for a specific program or for the entire university. In such programs, each student who participates in the program is expected to acquire a notebook computer. For the student, this represents a significant additional investment in their education: the annual cost to the student is likely to be in the order of US$1,000 per year. In relation to other fees charged to students this is a significant increase in direct cost to their education. In Canada, such a cost increase typically represents between 33% and 50% of an undergraduate student's tuition fee. In absolute terms, such programs represent a large capital investment, which may be largely "hidden" from the universities financial statements since it is spread across the student body. As an example, a university with 4,000 undergraduate students in four-year degree programs would likely require an annual student spending of well over US$1 million. "Rational" approaches to systems analysis and design theory propose rigorous and structured approaches to planning and implementing new technologies to meet user needs, yet there is evidence to suggest that there is a gap between these principles and the reality of technology decision making. Despite an emphasis on cost-benefit analysis in systems development theory, most of the academic literature on ICT management is silent on issues related to costs or other negative aspects. The issue of cost-justification is particularly challenging in universities, where "productivity" and "return on investment" are even less clearly defined than in industry. 2. RATIONAL DECISION MAKING: ASSESSING THE COSTS AND BENEFITS OF TECHNOLOGY Generally, it has been assumed that ICT decisions are rational. Prevailing notions hold that systems are implemented to support organizational objectives, to improve performance, etc. Theoretical models of "rational" systems development include step by step systems development life cycle (SDLC) models in which cost-benefit analysis is a critical step (Martin et al., 1994). The dominant view in the field is that organizations need to manage the realization of benefits as well as costs, "consequently enhancing the delivery of accurate, timely, and appropriate services within an organization, which in turn increases the economic vitality of the business" (Irani and Love, 2000:161). There is recognition that there remains a "technology management gap" within many businesses, which may result in a competitive advantage being jeopardized (Irani and Love, 2000:161). Emphasis is on the importance of a strategic alignment model, including both the fit with business strategy and functional integration (Henderson and Venkatraman, 1999). Recently, there has been specific attention paid to planning and implementing information technology in educational institutions which are inclined to simply adapt existing business models, such as Porter's value chain, and IT planning models to the university environment (Daniels, 1996; Katz and Rudy, 1999). While considerable literatures extol the virtues of technology, its actual effects are still the subject of significant debate. Some technologies for certain applications offer some benefits (Brynjolfsson, 1993; Barua, Kriebel and Mukhapadhyay, 1995). However, it does not follow that all technologies are either beneficial or cost effective for all applications. The question of the benefits versus the costs of technology at a system-wide level has also been discussed in the context of what is often called "the productivity paradox". Some wonder why output is not growing faster as the industry invests more and more in computers and attention is increasingly focused on huge investments and uncertainty. Recently, questions have been raised regarding the ways of assessing costs and benefits of learning technologies in universities (Bates, 2000; Finklestein et al., 2000). Certainly, some maintain that ICT has produced an economic revolution (Dewan and Kraemar, 1998), but others have raised questions about whether or not technology is improving productivity, or whether it is simply creating a "tyranny of the urgent, but unimportant" (Dewan and Kraemar, 1998). Still others have suggested that there is no "paradox", and that any apparent contradiction is merely a result of the problems of measurement and data, or of scale (Diewert and Fox, 1999). However, others have proposed that the problem really comes back to the management of ICT in education. It has also been suggested that "the inability to realize value from ICT investments is, in part, due to the lack of alignment between the business and ICT strategies of organizations" (Henderson and Venkatraman, 1999). Brynjolfsson and Hitt (1998) have suggested that there is a time lag in the payoff of ICT investments and that many companies have achieved a significant payback while other technology investments were "white elephants". This perspective highlights the importance of understanding the factors that may make some ICT investments a success and others a failure. While there is a great deal of literature on the benefits of ICT in education and even some that criticizes it, there is very little which addresses costs. Learning technology, particularly where the technology becomes the principal means of delivery, requires extraordinary investment not only for hardware, software, networking and support, but also for instructional design and content development. Designing and delivering a single course via multimedia and the Internet can cost as little as $15,000 (Boettcher, 2000) and as much as $500,000. Universities in Ontario currently spend approximately 10% as much on technology as they spend on salaries (CFOUO, 1998). This does not include the costs that are being carried by students who, at an increasing number of universities, are being required to purchase or lease computers. The Canadian public school sector spends a total of $24.3 billion per year, and it was estimated that providing state-of-the-art information technology would require an additional investment of $13 billion over four years (Fournier and MacKinnon, 1994). Although comparable numbers are not available for the whole university sector, surveys reveal significant growth in ICT expenditures at North American universities. One fifth of respondents indicated that ongoing financing of ICT on campus was a critical issue (Green, 2000). Often academic administrators have not anticipated the long-term costs associated with ICT and have been required to make cuts in other areas to support them (Lewington, 1998; Landry, 2000). Surveys of administrators have indicated that costs of ICT are a growing concern; however most institutions do not actually know how much they are spending on ICT, as the costs are highly decentralized (Green, 2000). Very recently there have been systematic efforts to assess the costs of learning technologies in post-secondary institutions (Bates, 2000; Finkelstein et al., 2000). 3. LEARNING TECHNOLOGY Technologies to support learning have evolved over the last two decades. They have been labeled in a variety of ways: instructional television, instructional technology, tele-education, tele-learning, distance education, open learning, computer-based education, computer-assisted instruction (CAI), computer-mediated learning, learning technology, on-line learning, virtual universities and, most recently, e-learning (Greco, 1999). E-learning is seen as a growth market (Goodridge, 2001). Learning technologies include e-mail, presentation systems, multimedia and computer-based applications, audio and video conferencing, and web-based applications (Bates, 2000). These technologies are used in traditional on-campus courses, to enhance and enrich classroom presentations. They are also used to support self-paced activities in labs, to support electronic discussions and access to materials. ICT also provides "distance" learning both in synchronous or asynchronous modes to off campus students. Figure 1 provides one way of understanding the range of applications. Figure 1: Learning Technology Time and Place (adapted from Johansen, 1992) TIME Same Different PLACE Same ENHANCED CLASSROOM Simulations Remote access to resources PC Projection Laptop Enabled Process Support SELF-DIRECTED LEARNING Computer Labs Computer-based Training CDRom/DVD Internet Different SYNCHRONOUS DISTANCE LEARNING Audio conferencing Audiographics Video Conferencing Internet Screen Sharing Broadcast TV/Radio ASYNCHRONOUS DISTANCE LEARNING Print Audio/Videotape CDRom/DVD Internet Most universities use email, and 43% of courses offered now use the web as part of the syllabus. More than half of colleges surveyed (55.5%) indicated that they had at least one course which was entirely web-based (Green, 2000). However, despite the attention focused on "on-line learning" and virtual universities, the reality is quite different. At most universities, distance learning activities remain a small proportion of their enrolments, and technology supplements, not supplants classroom learning (Green, 2000). In addition, most universities report that the majority of their students own computers (Green, 2000) and a number of universities have established mandatory requirements for students to lease laptop computers (Burg and Thomas, 1998). Though the increased use of ICT by universities is evident, it is not clear that universities have made the same "paradigm shift" that industry has gone to move from cost-based to value-based for their ICT investments. 4. THE DEBATE ON LEARNING TECHNOLOGY: A REVIEW OF THE LITERATURE While expenditures have risen considerably, debates about the adoption and utility of new technologies often degenerate into parallel narratives by those who are enthusiastic about the technology and those who critique it. On one hand, there are those who propose "A Learning Revolution" (Oblinger and Rush, 1997), which will render universities obsolete (Twigg and Miloff, 1998) as technology offers a higher quality of learning more efficiently. On the other hand, there are those who suggest that the costs of new technology are enormous and that "Digital Diploma Mills" threaten the very foundations of education (Noble, 1998). These debates remain unresolved in part because competing understandings of new technologies and underlying assumptions are rarely articulated. Moreover, while those who question the value of technology are often labeled "Luddites", part of the irrational "resistance to technology", there is seldom a corresponding critique applied to technology "enthusiasts". Even a cursory review of the literature on learning technology reveals that most of the emphasis is on the benefits of technology, while costs are seldom considered. In addition, even though they appear infrequently, critical views are often marginalized, overtly or subtly. For example, Educom published David Noble's critical "Digital Diploma Mills" with not one but three critiques (Noble, 1998; Shneiderman and Herman, 1998; Argre, 1998; Denning, 1998). The advocates of technology in education have maintained that "Higher education is becoming part of a 'knowledge and learning industry' in which competition forces every institution to rethink its products and markets". Some believe that half of all education beyond high school will soon be on-line (Finkelstein et al., 2000:7). Distance education is defined as "a killer application" offering universities a competitive advantage (Fornaciari, 1999). Resistance, characterized as the voice of "Neo-Luddites", is futile in the face of this "Learning Revolution" (Oblinger and Rush, 1997). Some challenge learning technology on political grounds, warning that "digital diploma mills" may destroy the foundations of education by promoting an uncritical or sub-critical "corporate agenda" (Noble, 1998). However, others critique the empirical basis of these claims suggesting, that benefits are overstated (Feenberg, 1999). Apart from the "paucity of empirical evidence that interactive learning technologies are any more effective than other instructional approaches", there are questions about the quality of much of the research, in part because it often confounds media with methods (Reeves, 1999). In 1997, after examining research on technology in schools, the US President's Committee of Advisors on Science and Technology (PCAST, 1997) severely criticized the existing body of ICT research and called on the government to undertake research in order "to ensure both the efficacy and cost-effectiveness of technology use within our nation's schools" (cited in Reeves, 2000:4). Similarly, a review of empirical research on technology in US universities maintains that most of the studies are not well designed. They do not control for extraneous variables, they do not use randomly selected subjects, the instruments focus on satisfaction rather than achievement of learning outcomes, and the studies do not control for the feelings and attitudes of the subjects (Phipps and Merisotis, 1999). Other studies suggest that students "select the distance education delivery method because of convenience, not quality, since distance education was found to be the least effective and least satisfying method of delivering for the students studied" (Ponzurick, Logar and France, 2000:180). Finally, others have cautioned against buying into the "myths of information technology" insisting that technology will not become cheaper nor will it prove to be a cash cow. Overall, it would seem that the results achieved by using ICT in education should be understood to be affected by a wide range of variables including the type of technology, the type of course, the type of learners, the instructional design, support services, etc. (Bates, 2000). The most consistent finding is that "the instructional methods students experience and the tasks they perform matter most in learning" (Reeves, 1999). Technology may be a useful tool, but is no panacea. 5. CITATION ANALYSIS AND RESEARCH Standardized searches of the ABI database, using the guided search format within ABI/Inform Global reveal interesting patterns in the broad discussions of technology generally and technology in education during the period 1986-2000 (see Appendix 1). While discussions of "instructional technology" are relatively consistent during the period, there is a dramatic increase in citations of the "information highway" in the 1993-1998 period, which peak in 1996 then decline. Discussions of "virtually learning" and "the virtual university" explode towards the end of the period, but show signs of tapering off. References to "online learning" have just taken off and although "distance learning" is not a new concept, interest in it has grown substantially. Other work examining the discursive practices related to learning technology has noted that benefits tend to be emphasized and costs ignored. The discourse, in general, tends to reinforce the positive aspects of the technology explicitly, through its argumentation and use of "experts", and implicitly, through its use of such metaphors and associative language (Cukier and Bauer, 2001). It is also worth noting throughout the period the references to "revolutions" and "paradigm shifts" in learning increase dramatically. In general, the patterns in the refereed literature are similar although they account for fewer citations. For example, the search terms "online" and "learning" produced 143 hits in 1999; 273 in 2000 and 304 in 2001. Of these 26, 49, and 63 were refereed publications, respectively. A standardized search using the terms "on-line" or "distance" and "learning" or "education" and "evaluation" or "assessment" or "impact" or "costs" on the Proquest database (all major fields) for refereed journals during the period 1999 - 2001 produced a total of 65 citations. Of these, 50 were from 2001, 18 from 2000 and 15 from 1999. Inspection of the articles revealed some duplication but more significantly, only 16 actually contained empirical evaluations of on-line learning projects. A total of 14 articles were not relevant at all. Another 35 were general articles, descriptions of planned projects, or discussions of on-line learning making generalized claims but containing no new empirical data. Many simply claimed advantages based on other studies or reports. Given the obvious limits of the available refereed literature, an additional 25 articles and reports assessing projects were also collected. In total, 41 publications were examined covering projects from more than 30 institutions (see Table 1). Table 1: Institutions American Assembly of Collegiate Schools of Business (AACSB) R University of Central England (Birmingham) R Centers for Disease Control R Southwest Missouri State University (SMSU) Central Missouri State University State University of New York (SUNY) R Drexel University State's World Campus Hofstra University R Texas A & M University Iowa State University(ISU) University of Bradford R Moorhead State University R University of British Columbia New Jersey Institute of Technology University of Illinois Northern Virginia Community College (NVCC) University of North Carolina (Pembroke R and Charlotte R) Ontario Institute for Studies in Education, University of Toronto (OISE/UT) University of Hawaii Open University R University of Iowa Pace University University of Maryland Queen's University, Belfast University of North London Rensselarr Polytechnic Institute University of Wisconsin R (2) RMIT University, Australia R Virginia Polytechnic Institute and State University Rochester Institute of Technology NA - R (3) R = Refereed These articles were categorized according to the forms of evaluation used and the findings reported. Among these we observed several patterns. * most of the "evaluations" assess student satisfaction, but surprisingly few evaluate other aspects of online learning * many (50%) of the studies use this as the only research method * fewer than one third attempt to assess learning outcomes or impacts of the technology on student performance * few examine student enrolment or retention data * only 8 of the articles actually evaluate costs and of these only 1 is peer reviewed. Table 2: Forms of Evaluation Refereed All Method # % # % Student survey 12 71 15 60 Faculty survey 3 18 5 20 Interviews 2 8 Data Analysis: Student participation/ enrolment/retention data 1 6 5 20 Student Performance/ Achievement 5 29 9 36 Economic Cost/Benefit 1 6 7 28 Transaction/Message Analysis 3 18 2 8 Case Control (traditional vs. TEL) 3 18 3 12 Participant Observation 1 6 Experimental Design 1 6 Other 1 4 n=17 n=25 (n=40 but some used more than one method) Generally, the findings focus on student satisfaction although some of the studies also report on performance, costs and benefits (some, however, have limited data on which to base their claims). In general, student satisfaction is more positive than not. Evidence regarding student performance is mixed. Many of the studies report that online students are comparable to traditional students in terms of standardized performance but some found that this varied with the caliber of the students and the types of activities. Many of the studies' conclusions focused on instructional design and implementation issues. Table 3: Findings Reported Refereed All Findings # % # % Student satisfaction 12 75 12 48 Faculty satisfaction 3 12 Teaching/Learning Process 5 31 5 20 Technology 1 6 1 4 Student Performance 6 38 6 24 Costs/benefits 1 6 11 44 Student Participation 1 4 Technology issues 1 4 Other implementation issues 1 6 Retention 1 6 2 8 Institutional impacts 1 4 Total n=16 n=25 (n=40 note some reported more than one result) Certainly this study is based on a sample of articles from a single database supplemented with selected other reports and, given its limits, must be modest in its claims. The principal conclusion of this study is not that learning technology is good or bad but that given the relatively large volume of publishing on learning technology in the general and academic press, the limited number of systematic evaluations of impacts (beyond student satisfaction) is surprising. Moreover, it would seem that many of the articles making claims about the technology are not based on empirical assessments. 6. FUELING THE "TECHNOLOGY PUSH" In spite of the limited empirical evidence regarding the impact of learning technology and even less information about the costs and benefits, governments have embraced it enthusiastically. Often this is in the context of popularization of the Internet and the growth in information technology markets. The marked increase in articles referring to the information highway in 1994 seems to coincide with Vice-President Al Gore's pronouncements. Government played a critical role in the institutionalization of the concept. The "information highway" was constructed at the nexus of technological and institutional interests (King et al., 1994:162). In Canada, the federal government most recently created the Advisory Council on Online Learning, chaired by David Johnson. Its final report noted "Online learning can contribute to the quality, accessibility, mobility or portability, and relevance or responsiveness, of post-secondary education" (Advisory Committee for Online Learning, 2001: xi). The starting point for the committee was the assumption that on-line learning is beneficial. Indeed, in the research reports commissioned by the group, we are told, "Whether or not Universities and Colleges should involve themselves in online or collaborative learning, and/or take a 'market driven' approach to education, is a policy issue that is beyond the scope of this project" (Keenan, 2000:2). Similarly, the report on Financial Strategies and Resources to Support Online Learning (Bates, 2000:2) does not question the value of the investment, but merely the advantages and disadvantages of using various sources of funding. It is interesting to look in a preliminary way at the report issued by the committee. The results are hardly surprising. The value of on-line learning is assumed because "We are in an information society". "Online learning [is] the way for lifelong learning and access" and "will enhance traditional education". Moreover, if we do not engage in this opportunity, the need will be met by foreigners and we will lose our cultural sovereignty. The report of the committee reinforces the arguments and language discussed above. Although the committee was supposed to assess the pros and cons of online learning, there is little discussion of them, as these questions are deferred until there is appropriate funding for more research. Rather, the report focuses on the potential and on removing the barriers without stepping back to question whether or not the undertaking will actually prove worthwhile. The report insists more research is required but it never questions its underlying premise, that online learning is essential to Canada's economic development. Moreover, the results of the "research" seem to be forgone conclusions. The report also acknowledges that on-line learning is expensive. But at no point does the committee question whether or not the investment is worth it in light of the uncertain benefits or the opportunity costs. The only question asked is how it will be funded. Similarly, there is no discussion of alternative means for achieving these ends, or acknowledgement that a high speed broadband network infrastructure is critical. Not only is the language familiar, but also the arguments seem recursive; we need online learning so we can lead in online learning. The report insists that we cannot wait, we must reengineer universities and the learning enterprise or face disaster: Some colleges and universities might disappear: Some might actually acquire other institutions devouring their competition in "hostile takeovers". In other words, it is conceivable that some Canadian institutions might disappear or be expatriated or seriously weakened. (The Advisory Committee for Online Learning, 2001:37). Why? Because there is a lot of money at stake: This two trillion dollar global education and training industry is going through radical changes. Mega trends such as demographics, the Internet, globalization, branding, consolidation, and outsourcing all play major roles in this transformation (The Advisory Committee for Online Learning, 2001:32). In searching for the roots of the relentless enthusiasm for online learning it is worth examining some of the interests involved. Vendors are well represented on the committee, which includes AT&T, IBM, Bell Canada Enterprises and Lucent, along with post-secondary administrators (13) including four with IBM laptop programs. While not to dismiss the committee's findings, it is important to consider the extent to which their interests were served by an enthusiastic endorsement of online learning. Certainly learning technology is a major thrust of the marketing efforts for vendors in the ICT sector. Laptop Universities such as Acadia Advantage are featured prominently on IBM's website: Partnered with private industry, the three-stage Acadia Advantage program will make Acadia the first electronic campus in Canada. Offices, libraries, classrooms, laboratories, the student's center and all the residence rooms will be equipped with the latest fiber optic, data video and voice transmission systems. By the year 2000, IBM ThinkPads will be a standard part of every student's admission... reinforcing Acadia's position as a Canadian leader in teaching innovation (IBM, 2000). But even IBM-sponsored research is inconclusive (NODE, 2000). For example, a report commissioned from the Network Ontario Distance Education (NODE) states that NODE undertook extensive research into the state of mobile computing in North American Colleges and Universities. "Institutions are observing positive changes in their campuses and confirming their commitment to the vision of ubiquitous computing" (NODE, 2000:i). However, the same study provided no empirical evidence of benefits, and acknowledges: At worst, there is no change. At best, there may be a slight improvement. ...But with so many variables involved, there is little hard evidence to show that having any time any place access to technology improves learning (NODE, 2000: 4). 7. DISCUSSION AND CONCLUSIONS This paper has not been an effort to dismiss the value of technology in education; indeed, the authors are the architects of a major laptop initiative. However, it does offer evidence that suggests that many of the "taken for granted" assumptions about technology should be challenged and that, in particular, a hard look should be taken at claimed benefits and costs. Organizational theory has shown a strong tendency towards "isomorphism" or conformity in institutions, and this paper has suggested some of the ways in which the dominant discourse on learning technology and its benefits has been shaped. If we consider the relative power of the various actors engaged in this discourse and what is at stake, particularly the role of the suppliers of technology in this US$3 trillion market, it is not surprising that the assessments of the potential of learning technology are not particularly "objective" or "balanced" (DiMaggio in Alvarez, 1996: 94). Public relations, advertising, indirect funding and other forms of influence in the production and reproduction of discourse warrant further exploration. Certainly, it is fair to assume that not all stakeholders are equal in their efforts to ensure their message is heard and this, in part, may account for the consistency and strength of the dominant discourse on learning technology. While at one level it seems absurdly obvious, this paper suggests there is almost no consideration given to the costs of technology and often the benefits are assumed. What are the benefits? What evidence is there to support the claims? What do we really know and what do we not know? What are the costs? What other impacts might the technology have? What do the acronyms really mean? Are there taken for granted assumptions that should be questioned? Who should be involved? What perspectives should be considered? We would suggest that faculty in ICT business faculties have a double responsibility: to use "best practices" to plan for and implement new learning technologies. Given the level of investment by students and by the university, costs and benefits need to be determined and more specific attention paid to understanding how the technology can be used effectively, determining the impact on pedagogy and on the student experience. This study suggests that further work should include: * More critical analysis of the claims and the assumptions that frame our research and teaching of technology. * Better determination of the advantages, disadvantages and real costs involved in ICT-enabled education (technology-enabled education). * A more specific focus on the learning outcomes expected from such interventions. * A richer discussion of how faculty and students can learn how to use these tools effectively. * The challenges of establishing ways in which all stakeholders can have a more effective input to the discourse. In addition, this study demonstrates how cross-disciplinary research approaches and techniques can provide deeper and richer insights in areas which are often dominated by more restricted perspectives and encourages ICT-focused researchers to consider how to expand their perspectives APPENDIX 1: Citation Analysis (ABI Inform, all fields) 8. BIBLIOGRAPHY The Advisory Committee for Online Learning (February, 2001), "The e-learning e-volution in Colleges and Universities". Ottawa: Industry Canada. Argre, Phil (1998), "Meet me at the Crux". Educom Review, 33 (3): 28-30. Baily, Martin Neil and Robert J. Gordon (1988), "Economic Activity". The Productivity Slowdown, Measurement Issues and the Explosion of Computer Power. In William C. Brainard and George L. Perry Brookings (eds.), Paper on Economic Activity, (2): 347-431. Barua, Anitesh, Charles Kriebel and Tidas Mukhaopadhyay (1995), "Information Technologies and Business Value: An Analytic and Empirical Investigation". ISR, 6 (1): 3-23. Bates, Tony (2000), Financial Strategies and Resources to Support Distance Learning. Report to Advisory Committee for Online Learning. Ottawa: Industry Canada. Boettcher, Judith (2000), "How much does it cost to put a course online? It all depends". In Finklestein, Martin J.,Carol Frances, Frank I. Jewett and Bernhard W. Scholz (eds.), "Dollars, Distance and Online Education: The New Economics of College Teaching and Learning". The Oryx Press: 172-198. Brynjolfsson, E. (1993), "The Productivity Paradox of Information Technology". Communications of the ACM, 36 (12): 20-26. Brynjolfsson, E. and Lorin M. Hitt (September, 1998), Beyond Computation: Information Technology, Organizational Transformation and Business Performance, Working Paper, http://ccs.mit.edu/erik/. Burg, Jennifer J. and Stan J. Thomas (1998), "Computers Across Campus". Communications of the ACM, 41 (1): 22-25. Cukier, Wendy and Robert Bauer (2001), "Communications Distortions in Learning Technology Discourse". EGOS, Barcelona. Council of Financial Officers of Universities of Ontario (CFOUO) (1998), Financial Report of Ontario Universities 1996/97. Ottawa: Industry Canada. Daniels, John S. (1996), "Mega Universities and Knowledge Media: Technology Strategies for Higher Education". London: Korgan Page Ltd. Denning, Peter J. (1998), "Skewer the Stereotype". Educom Review, 33 (3): 30-34. Dewan, Sanjeev and Kenneth L. Kraemer (1998), "International Dimensions of the Productivity Paradox". Communications of the ACM, 41 (8): 59-61. Diewert, W. Erwin and Kevin J. Fox (1999), "Can measurement error explain the productivity paradox?" Canadian Journal of Economics, 32 (2): 251-274. DiMaggio, Paul and Walter Powell (1984), "The iron cage revisited: institutional isomorphism and collective rationality in organizational fields". American Sociological Review, 48: 147-60. Finklestein, Martin J., Carol Frances, Frank I. Jewett and Bernhard W. Scholz (2000), Dollars, Distance and Online Education: The New Economics of College Teaching and Learning. American Council on Education and the Oryx Press: The Oryx Press. Fornaciari, J. (1999), "Distance Education as Strategy: How can your school compete". Journal of Management Education, 23 (6): 703-718. Fournier, Luc and Kim MacKinnon (1994), Educational Opportunities on Canada's Information Highway: The Use and Deployment of Communication and Information Technologies in Education. Industry Canada Report, Ottawa: Industry Canada. Frances C., R. Pomerantz and J. Caplan (1999), "Planning for Information Technology: What You Thought You Knew Could Lead You Astray". Changes, 31 (4): 25-33. Goodridge, Elisabeth (2001), "Hot e-learning sector may be cooling". Information Week, 846 July 16: 36. Greco, JoAnn (1999), "Going the Distance for MBA Candidates". The Journal of Business Strategy, 20 (3): 30-34. Green, Kenneth (2000), The 2000 Survey of Information Technology in US Higher Education. Encino: Campus Computing Survey. Henderson, J.C. and H. Venkatraman (1999), "Strategic alignment: Leveraging information technology for transforming organizations". IBM Systems Journal, 473. IBM (2000), "IBM partners with Acadia University". In www.pc.ibm/US/thinkpad/. Irani, Zahir and [??] Love (2000), "The Propagation of Technology Management Taxonomies for Evaluation Investments in Information Systems". Journal of Management Information Systems, 17 (3): 161-178. Johansen, Robert (1992), Collaborative Learning Report. Menlo Park: Institute for the Future. Keenan, Thomas P. (July 2000), "Removing Barriers to Online Education". Report to Advisory Committee for Online Learning. Ottawa: Industry Canada. King, J. L., V. Gurbaxani, K. L. McFarlan, K. S. Raman and C. S. Yap (1994), "Institutional Factors in Information Technology Innovation". Information Systems Research, 5 (2): 139-199. Landry, Stephan (2000), "Costs of Ubiquitous Computing: A Case Study of Seton Hall". In Finklestein, Martin J., Carol Frances, Frank I. Jewett and Bernhard W. Scholz (eds.), "Dollars, Distance and Online Education: The New Economics of College Teaching and Learning". The Oryx Press: Oryx Press. Lewington, Jennifer (1998), "Trapped on the upgrade mill". The Globe and Mail, June 22: C1. Lohr, Steve (1999), "Computer Age Gains Respect of Economists". The New York Times: A1. Martin, Wainwright E. DeHayes, Walter Daniel, Jeffrey A. Hoffer and William C. Perkins (1994), Managing Information Technology: What Managers Need to Know. Upper Saddle River: Prentice Hall. Noble, David (1998), "Digital Diploma Mills: the Automation of Higher Education". Educom Review, 49 (9): 48-53. Node Learning Technologies Network (2000), "The Mobile Classroom Matures: Learning Technologies Report". Node Learning Technologies Network Report, (www.theNode.org). Oblinger, D. G. and S. G. Rush (eds.) (1997), The learning revolution: The challenge of information technology in the academy. Bolton, MA: Anker Publishing Company, Inc. President's Committee of Advisors on Science and Technology (PCAST) (1997), Report to the President on the Use of Technology to Strengthen K-12 Education in the United States. Washington, D.C.: PCAST. Phipps, Ronald and Jamie Merisotis (1999), What's the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education. Washington: The Institute for Higher Education Policy Report. Ponzurick, Thomas G., Karen France and Cyril Logar (2000), "Delivering Graduate Marketing Education: An Analysis of Face to Face versus Distance Education". Journal of Marketing Education, 22: 3 180-187. Reeves, Thomas C. (2000), Enhancing the Worth of Instructional Technology Research through "Design Experiments" and Other Development Research Strategies. Presented at Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. Reeves, Thomas (1998), The Impact of Media and Technology in Schools, A Research Report, Bertlesmann Foundation http://www.athensacademy.org/instruct/media_tec. Shneiderman, Ben and Herman Richard (1998), "Technology in Education: The Fight for the Future". Educom Review, 33 (3): 22-43. Twigg, Carole and Michael Miloff (1998), "The global learning infrastructure: The future of higher education in Tapscott. In Don Alex Lowey and David Ticoll (eds.), Blueprint for the Digital Economy. New York: McGraw-Hill: 179-201. Weill, P. and M. Broadbent (1998), "Leveraging the New Infrastructure: Ho Market Leaders Capitalize on Information Technology". Boston: Harvard Business School. 1 wcukier@ryerson.ca 8