Incorporating Non-scholarly Literature with Academic Literature: A starting point for teaching research methods to Masters candidates in Information Systems Linda Jo Calloway1 School of Computer Science and Information Systems, Pace University 1 Pace Plaza, New York NY 10038 USA Abstract Information Technology practitioners and researchers cope constantly with the problems of evaluating new and emerging phenomena. The worldwide web technologies are a prevalent example of these phenomena. These technologies and business trends are discussed, described and advertised in newspapers, web-releases and trade press as well as academic journals and conference publications, consulting reports and government reports. Information from some of these media often falls short of the rigorous screening processes that define academic research. Nevertheless, the information in these sources may be the only information available on emerging technologies to the practitioners. These sources provide the information used by practitioners in the field to make decisions about technology. The people making decisions about these emerging phenomena desperately need valid ways to assess these phenomena. The Research Seminar course we require addresses this need. This brief paper discusses this course, and the methods used to explore emerging phenomena. It is not a research paper. Rather, it describes the course and some of the philosophies used to design the course. A summary of research topics explored during the last few semesters is catalogued. Keywords: Teaching research methods, Information Systems curriculum, qualitative research, and emerging technologies 1. INTRODUCTION The majority of candidates for "Masters of Information Science" degrees at our Urban University are working primarily in the Information Systems field. They are faced with the issues of how to make informed decisions about information systems phenomena. We need our graduates to have exposure to and experience with research methods aimed at discovering knowledge in Information Systems. We were faced with the question of how to establish curricula. Our mandate for the course is to engage the practitioner students in research projects of relevance to them, while providing a survey of research methods with exemplars from the IS scholarly literature. The deliverables include a research paper that develop research questions based on literature reviews, a methodology and study design, and a pilot project using the methodology the paper develops. The objectives include educating candidates so that they can apply these research methods in their work and personal endeavors. To meet these challenges, we developed a research methodology for creating research proposals and pilot studies that uncovers thematic information about emerging phenomena by integrating non-academic sources and scholarly research. The method for integrating various streams of literature into workable, researchable proposals is based on sturdy conceptual foundations. The methods for discovering valid maps of phenomena in an orderly fashion come from interpretive research and grounded theory (e.g., Klein and Myers, 1999; Glaser, 1992; 1967; Spradley, 1979, 1980). The pedagogical basis for the method comes from education research (see Joyce and Weil, 1986; Vialle, et al. 1997). This paper first discusses the state of teaching research methods in Masters programs in Information Systems, both in Arts and Science and Business curricula. Next, selected literature on teaching research methods and using grounded theory is discussed. There follows a brief outline of the course and of the methodology for discovering appropriate topics. The American Heritage College Dictionary 1997 edition defines lay of the land as "The nature, arrangement, or disposition of something", which captures the nature of the methodology. The methodology integrates scholarly research with popular and trade articles to form an emerging theory of the subject. We call this process finding themes from popular and trade press and integrating them with academic literature "discovering the Lay of the Land". Then, the paper explores the relevance of the method in terms of the topics selected over the past five semesters. The paper concludes with a short discussion. 2. THE CURRENT STATE OF TEACHING RESEARCH A search of the ISWorld Net page on Graduate IS programs in the United States (Galletta, ed., 2000) listed 120 programs. Very few generalized research methods courses are offered at the Masters level (about four). Several programs listed independent research projects and thesis projects that undoubtedly cover research methods. It appeared that none of the MBA or MS programs in Information Systems listed had a required course in Information Systems Research Methods. The world of Information Systems research uses a multiplicity of research methods, The case was made at our large Urban University School of Computer Science and Information Systems that individual research projects alone do not provide sufficient coverage of the breadth of IS methodologies, and a formal, required course was needed. Among the first objectives for teaching research methods to Information Systems professionals is to develop practitioners who are able to reflect on their endeavors and to change and evaluate their work. These activities are especially critical in Information Systems practitioners because the change in landscape of tools, tool usage, management, predictions and change itself is accelerating. As an ideal, we would like to create reflective practitioners (after Schon, 1987). Reflective practitioners take on the role of reflecting on and inquiring into the practice of their work. Reflective practitioners are empowered to understand and improve their own practice. Other disciplines require courses in research methods that offer conceptual, empirical and subjective methodologies. For example, Mass Communication practitioners are taught research methods in order to appropriately report and interpret statistical information such as political polls. Journalists must also be competent consumers of research using conceptual and subjective methodologies as well as empirical methods in order to summarize research papers for consumption by non-expert audiences. As another example, the Psychology profession offers required research methods courses at both undergraduate and graduate levels. The National Council of Schools of Professional Psychology acknowledges the validity of multiple ways of knowing, including objective knowledge derived from empirical investigation and subjective knowledge based on personal experience. These courses include methods for interpreting human behavior such as hermeneutics that focuses on discovering meanings inherent in human activity (McConnell & Lewis, 1998). These methods are all appropriate for investigating IS. 3. THE RESEARCH SEMINAR The seminar we teach is similar to many that require a substantial research deliverable at the end of the term, and ours grew out of the Independent Research Project course that preceded it. The focus is on exploring and discovering a research topic, searching the literature to develop specific research questions, designing an empirical study including results that might be anticipated from the literature, doing a pilot study with a convenience sample of appropriate subjects and discussing the results of the pilot. The course is accompanied by a text on research methods. Few individuals without experience using research methods seek out scholarly literature to find either research topics or information on emerging phenomena. It seems the usual learning motivation rests in the perceived practical. We elected, therefore, to devise a method for exploration that included literature aimed at practitioners. One motivation for exploring the literature in the world of the practitioner is based on the idea of inquiry training. Using inquiry training methods for selecting and investigating research topics is based on fundamental concepts of early education (Suchman (1962), in Joyce and Weil, 1986). Inquiry training trains students to engage in causal reasoning. The process leads students to collect and verify information, develop concepts, and build and test hypotheses. The purpose is to move students from facts to theories by presenting them with a puzzling event to raise their curiosity. By generating and testing various hypotheses about the facts or variables, and isolating and testing these variables singly, the student generates theory. This method of inquiry is very like Grounded Theory generation (Glaser, 1992; 1967). The main difference between inquiry training and grounded theory generation is that in inquiry training the teacher knows the method and how the facts and variables fit together. Grounded theory generation seeks to use organized, systematic sampling of the available data from some phenomena and then to use specific methods for creating a topology of categories that represent the data. These categories are a theoretical way of understanding the underlying phenomena, because they are based on, or grounded in, the data. By inference, the category sets represents the organization of similar sets of data from similar situations (see Glaser, 1992; 1967). In grounded theories, the data represent the independent variable, while the context of the data represents the dependent variable. The object is that by looking at the data one can make inferences about the context. If the categorized data is similar, the inference is that the contexts of the data are also similar. Once a preliminary categorization is developed, ethnographic methods are used to further define the "theory". These methods are used to discover the relationships among categories and to clarify and refine categories (Spradley, 1979), or to discover what we are calling the Lay of the Land 4. USING THE LAY OF THE LAND METHODOLOGY We begin the course by requiring students to select a research topic (subject to change or modification). The topic must meet two criteria: First, it must be especially interesting to the individual and second, it must relate to Information Systems (we use IS to include Information Technology (IT), organizational issues, and human concerns). On the first day of class, students perform an exercise in categorizing things. We might ask each student to categorize all things in a pocketbook or bookbag, for example. The actual method of categorization is very straightforward, and is based on ethnographic domain analysis methods of Spradley (1979, 1980). First, the person sorts everything into piles, putting like things in like piles. Most people get between 3 and 7 piles. If the person misses these limits, they must add things or force stuff into different or more piles one way or another. Second, the person is to take one pile and separates it into 3 to 5 smaller piles. Third, the person must take the large piles 3 at a time, and decide which two of the three are more alike and which one is different. Then the sub-piles within the large piles are taken three at a time and the same procedure is done. For each set of three piles, the person must describe or write down why the two piles are similar and why the third is different. During this stage, the piles might merge or divide or otherwise change. Fourth, the individual must describe the contents of each pile and sub-pile in a few words, using an example from the pile. Fifth, the person must name each pile including the sub-piles. The intent is that the names of the piles (and sub-piles), the descriptions of the piles, and the interrelationships among piles is the basis for a grounded theory of the phenomena being categorized. This exercise provides a basic introduction to grounded theory. More detailed readings and assignments are given later in the course. From the first day the student searches any literature anywhere, with the objective of getting a broad picture of their subject. In week two, the assignment is to summarize and categorize information on an assigned subject from the trade press, and the next week the student must summarize two short research studies in IS according to a specific annotation template. For many students, this is the first clear idea of the differences in focus and value of the two streams of information, scholarly and non-scholarly. During these weeks the students continue to read, developing their topic concepts, research purposes and broad questions of interest. Concomitantly, the course includes assignments from a standard research text such as Punch, Creswell or Nachamais. In the fourth week the student must have annotated summaries of at least three research articles and three non-research articles related to the questions they are investigating. Consulting reports are allowed under the rubric of crossover publications. The fifth week work includes designing possible specific qualitative and quantitative research questions in their subject, and then designing a possible qualitative and quantitative methodology to address the questions. The student is discovering the Lay of the Land. Among the early results of the Lay of the Land approach is to discover what themes run through the popular and trade analyses on the subject in question. The themes are grounded in the popular and trade press, and are the bases for the emerging theory. These themes guide the search in the scholarly literature for similar patterns of phenomena, abstractions of the thematic patterns, and historical analogs. For example, privacy issues on the Internet are echoed in numerous IS research studies on privacy. Biometrics technology commercial adoption reaches into the privacy issues perceived by customers as well literature on technology adoption. Satisfaction issues in shopping experiences online reflect other satisfaction issues. The short history of on-line financial management is easily framed in Porter's industry forces models, and e-retail delivery questions look to research on strategy and virtual organization literature. Growth predictions of Internet phenomena include variables found in other telecommunications adoption studies. These scholarly research strands provide the anchors for current investigations. 5. THE SCOPE OF THE RESEARCH STUDIES First this section points out some of the characteristics of the studies, to explore the subject scope. Next, the studies are categorized and described in a table. There are 61 papers used in this compilation of research subjects explored during five semesters of the course, including this semester. These semesters are (1) Spring 1998, (2) Fall 1998, (3) Spring 1999, (4) Fall 1999 and (5) Spring 2000. * EDI on the internet came and went in spring 98. * Privacy interests and the internet consistently occupy prominent places in all semesters. * Telecommuting's positive and negative consequences on individuals surfaced in Fall 1998, flourished in Spring 1999 and drifted off by Fall 1999. * Banking and brokerage activities online first appeared in Fall 1999 with a study on the influence of online banking on customer relationship management and a study relating inferior good economic theories to attitudes about online economic analyses. The studies continue in Spring 1999, with quantitative studies of online IPO economics, the relationship of total cash available to stocks traded and profits in day trading, and assessing Established Brokerage house entry into on-line trading using models from Broadcast Television's entry into cable. The studies continued in spring 2000. * Database Management Systems, Data Warehousing and knowledge management systems studies appeared in Spring 1999 and continue. * The Music industry and the internet have made a strong debut in Spring 2000, with four studies out of 17 students. Other subjects that appear regularly are: * E-commerce, which continues to be a hot topic for research studies in all five semesters, with studies relating to privacy, channel management, marketing, user satisfaction, organization structure, and studies of the effects of e-commerce on specific industries such as travel, restaurants, retail and education. * Education delivery on the internet is the subject of two studies--delivering English as a Second Language to Latino adults and secondary music education. Corporate training also occurs in three of the five semesters. Other education studies include an experiment on the effects of pretest on learning at a database conference, comparing subsequent usage of advanced spreadsheet macros with training on premises and off premises, and a correlation study of webboard use and course grades in computer literacy classes. * Studies on attitudes towards hardware sprinkle through the five semesters, beginning with an investigation of the acceptance of biometrics technology in commercial applications, followed by attitude vs. technology acceptance in internet connectivity media and palm pilots. * Other studies include a Fall 1998 study using an organizational modeling approach to understand why some large organizations view the internet paradigm as hype while others view it as reality, and a quantitative study of the effects of thin client computing on personnel costs related to enduser computing. The 61 research studies are grouped in the following table according to subject Category # Subjects Internet privacy 8 Privacy vs. tailoring, kid sites, user awareness, user attitudes Electronic businesses 14 Music, loyalty programs, EDI, Banking, Brokerage, startups, spinoffs, retail, marketing Data and knowledge 5 Knowledge management, data warehousing, new database management, merging knowledge management systems Training and education 9 Corporate training, MBA programs, distance learning, course augmentation, virtual classrooms, music, ESL Hardware attitudes 5 Biometrics, voice response, smart cards, internet access, personal digital assistants Telecommuting 5 Productivity, stress, virtual office Software and design 4 UML, linyx, open source development, capability maturity model Macro Assessments 5 Hospital emergency, IS in small businesses, Global internet availability, responding to new paradigms Costing 4 Total cost of ownership, thin client support, outsourcing Management tools 2 Collaborative technologies, improving project management 6. DISCUSSION The research studies concentrate on subjects relevant to practitioners today. As might be expected, the choice of subject reflects issues prominent in the trade press and the popular press. These investigations and research designs are grounded in today's business literature. Using categorization and grounded theory to uncover thematic information and then to anchor studies in the relevant scholarly literature is designed to use reflective inquiry to discover the Lay of the Land and to help mold reflective practitioners. The subjects are relevant to the interests of the practitioners and students, since the selection method enforces the criteria. The students find the course rewarding, and a number of papers have been accepted for publication. The next step for us is to develop workable longitudinal evaluation methods. 7. REFERENCES Argyris, C. & Conant, J. B., 1984, "The Executive Mind and Double-Loop Learning", Reprinted in Beyond Productivity: Information Systems Development for Organizational Effectiveness. Th. M. A. Bemelmans (ed.) Elsevier Science Publishers B. V. Benbasat, Robert & Zmud, W., March 1999, "Empirical research in information systems: The practice of relevance". MIS Quarterly, Minneapolis. Calloway, L. J., 2000, Assessing research studies resulting from using the Lay of the Land methodology (in progress). Denham, Bryan, winter 1997, "Teaching research methods to undergraduates", Journalism & Mass Communication Educator, Columbia. Klein, H. K. & Myers, M. D., March 1999, "A set of principles for conducting and evaluating interpretive field studies in information systems", MIS Quarterly. Minneapolis. Glaser, B.G., 1992, Basics of grounded theory analysis, Mill Valley, CA: Sociology Press. Glaser B. G., & Strauss, A. L., 1967, The Discovery of Grounded Theory: Strategies for Qualitative Research, New York: Aldine Publishing Company. Joyce, B. & Weil, M., 1986, Models of Teaching, McConnell, S. C. & K. L. Lewis, 1998, "Practical Hermeneutic Tacking: Research Methods and Self-Development", presented at the Annual Convention of the American Psychological Association, August 14-18, San Francisco. Schon, D. A., 1987, Educating the Reflective Practitioner, Jossey-Bass Publications, San Francisco. Spradley, James P., 1979, The Ethnographic Interview. Out of print. Spradley, James P., 1980, Participant Observation Out of print. Vialle, W., N. Hall & T. Booth, July 1997, "Teaching research and inquiry in undergraduate teacher-education programmes", Asia - Pacific Journal of Teacher Education, Abingdon, third edition. Prentice-Hall, Inc., Englewood Cliffs. 1 Lcalloway@pace.edu