Preface to IS Research (Advice to the Novice) Ronald I. Frank, DPS rfrank@pace.edu Information Systems Department, Pace University Pleasantville, NY 10570, USA Abstract This is not a manual on how to do Information Systems research. There are many good books on the subject. Rather, it is a set of meta-comments about research in the Information Systems field. This includes writing doctoral dissertations, AND conducting corporate Research and Development (R & D). There is a common misunderstanding that research is only of interest to doctoral students or academic researchers. Research is an attitude and a skill set applied to problems we don’t yet know how to solve. I discuss why we do research, where research ideas come from, what the differences are between research and merely big projects, and how we do IS research. I present some famous examples of research which are NOT from IS but which provide general cultural background. I also provide some examples of recent IS research. This article itself falls into the category called cultural literacy. Finally, I provide some personal comments based on guiding doctoral IS students at Pace, having done Ad-tech in IBM Scientific Centers, having done research for IBM Research, and having managed research for IBM Research. In the beginning of IS, research was conducted to show customers how computers can solve their problems. Not much has changed in the ensuing 50 years but now we can look back at how some problems were solved (and how to improve on the solutions). The main difference between dissertation research and industrial research is the source of the problem. In industry, we tend to work on what promises to return value fastest (“most bang for the buck”). Keywords: Problems found in doing research, research characteristics, research process, researcher’s psychological problems. 1. PREFACE: WHAT IS A PREFACE? In a book, the preface is a set of comments about the book itself and the writing of the book but not about the book’s subject. The Introduction is where the book’s topic begins. So, a Preface is a set of meta-comments. This paper is made up of comments about the research process and the researcher. They are aimed at providing a common cultural background to the research process to better understand why we do it, what it is, what kinds there are, and what to expect if you do it. There are books on how to do research, (Leedy and Omrod, 2005) and there are also articles on teaching research (Shaffer, 2006). 2. WHY DO WE DO RESEARCH? Research is the way society learns and progresses. It brings in new knowledge for all to share. History (experience) can be viewed as unguided experimental research. Today, with so much of research being secret using government funding, or proprietary using corporate funding, we could lose sight of this societal reason. We might think that research is merely problem solving on a grand scale. We also have to make distinctions between solving specific product problems, and solving more general problems. We want to: * Discover new knowledge * Develop new products (innovate) * Solve meaningful problems * Find new relationships * Answer important questions * Fulfill a vision * Find useful results (improve products) It is important to realize that research is the only reliable way to solve NEW problems or discover NEW knowledge. We can’t rely on serendipity although the prepared mind can have an “aha!” moment (Hadamard, 1954). 3. THE KINDS OF RESEARCH The primary research dichotomy is Basic (or Fundamental) vs. Applied. Basic usually means we are looking for time invariant truths. Applied means we are looking for something explicitly useful – which is usually something more related to technology. In both types, the researcher attempts to either theorize or experiment. Theoretical means we try to form laws, usually couched in mathematics. Experimental means we do something active, even if it is only in our heads (thought experiments). Einstein’s relativity experiments were all in his head (called “Gedanken” Experiments). Experimental research is sometimes called Laboratory research. Basic and applied research are not exclusive opposites, as some may think. They are related and interdependent. Basic knowledge has a history of being found useful, and applied knowledge often uncovers further problems that lead to basic knowledge. Neither are theory and experiment exclusive opposites. Theory leads to experiments for verification, which sometimes uncover anomalies that guide further theorizing. See the Scientific Method diagram in the appendix. A subset of applied and experimental research is the shorter-term (less than five year) product oriented development we see a lot of in IS. This is often called Advanced Technology research, or Ad-tech. A smaller part of that is early product development. In terms of funding, Figure 1. is not to scale. More money goes into early product development (including weapons) than the other parts. This is sometimes phrased “small r (research) and large D Development)”. It is hard to draw a clear line between Ad-tech and Engineering – the application of well-known principles and facts. I think of engineering as having little or no doubt about feasibility where as in Ad-tech the motivation is often to prove feasibility for later engineering. The product development part of Figure 1 refers to the early stage of product development, which is followed by later engineering development. 4. EXAMPLES FOR IS Figure 1. Boxs I. and II. Theoretical and basic research includes much of CS research. An experimental basic research example is quantum encryption. Figure 1. Boxs I, III, and IV. An application of the research in quantum encryption is the Magiq Encryption product, (Magiqtech-1, 2006) which moved from I to II to III to IV to Ad-tech to product in about twenty years (1984-2004). Magiqtech is now partnering with Verizon in an Ad-tech feasibility study for a product service (Magiqtech-2, 2006). The Basic/Applied dichotomy is sometimes termed Theoretical vs. Applied, forgetting about basic experimental work. Some folks distinguish applied research from Advanced Technology (Ad-Tech), which is described as being more product oriented and having a more limited time horizon. In the past, basic research (theoretical or experimental) had a time horizon of decades – or we didn’t care if it did not have clearly foreseeable uses. Ad-tech had a very limited time horizon related to specific product life cycles – around five years. However, today, especially in software but also in hardware, product life cycles are much shorter – more like two to three years. An interesting example was the Ad-tech effort to develop atom bomb proof networking (ARPA NET) which had a long time horizon of indefinite years but was not considered basic research at the time. It was not viewed as research but as an engineering problem. There were lab experiments, performance data analysis, some very complicated mathematical models (simulations), and all the other trappings of basic research. There was a massive amount of invention – new knowledge. Was this actually basic research? I think so. I consider development of new architectures, especially in software, to be an applied theoretical activity (box III). The actual feasibility work on the new architecture is applied experimental work (box IV). 5. WHAT RESEARCH IS AND IS NOT What Research IS * NEW: Create a NEW contribution. * VALIDITY: Prove it is correct. * RELIABILITY: It is a robust result. * VERIFIABLE: It is repeatable. What Research IS NOT * Assertion w/o reference. * Presentation of the known. * Argument by authority. * Opinion (even by an “expert”). * Fact without reference. * Reference without vetting. * Literature search. * Library “research”. Characteristics of Research vs. Characteristics of Projects: * For doctoral research: intensely personal activity vs. teamwork. For large research there are teams. * Discovery of new knowledge vs. use of known knowledge. * Societal benefits vs. benefits gained for a sponsor (but government research is for society). * Publication of discovery vs. publication not necessary or not allowed. * Creative in nature vs. derivative in nature. * Don’t know if it can be done vs. know it can be done (Probability >.5). * Public activity and results vs. proprietary activity and results. 6. WHERE DO RESEARCH IDEAS COME FROM? SOME EXAMPLES: [Notice it in Nature] (de Broglie’s Paragraph) In 1923, 18 years after Einstein wrote the two equations in Figure 2. below, de Broglie put them together and predicted the wave nature of matter. The equations say that matter m has a wave frequency ?, i.e., an associated wave nature. Louis de Broglie noticed that if they were equated, there was a possibility in nature that it made sense. He wrote a short paper, the first paragraph of which contained the idea for which he received the Nobel Prize for electron optics in 1929. Figure 2. What he did: * He added some analysis of the particle’s wave frequency value. * He analyzed a light particle (slightly incorrectly) using basic relativity. * He analyzed an electron in a circular orbit and showed the same results as had Bohr and Sommerfeld (the Old Quantum Theory). (AIP, 2006). * He said he had derived the optics of particles elsewhere – which he did. * He did all of this in four pages with the basic idea in the first paragraph. * He was prepared in Quantum Mechanics and Relativity. * He was free to be “stupid – clever”. * It took Einstein’s intervention to get de Broglie his degree because his faculty of “Great Men” thought it was nonsense. * It took some years until experiment proved him right (electron optics). * His article (1923); Schrödinger’s Wave Equation (1925); his Nobel Prize (1929). (Box I in Figure 1.) [Notice it in Formalism] (Ron Frank’s Array Decomposition) The count of cells in an N dimensional cube array of side length n is nN. This form has a binomial expansion which can be interpreted as a decomposition in terms of arrays of length (n-1) and all dimensions <=N. Figure 3. Applying this recursively, we get a canonical decomposition of any cube in terms of null arrays (length 0). This is a generating function for all sub arrays. This is generalized to any non-equilateral array of any dimension. It can then be generalized to arrays that are not discrete, then to arrays with complex dimension, then to arrays with infinite dimension (countable or not). This decomposition has been used to compute all sub arrays of a given array. This has application to machine architecture (hyper-cubes) and to data mining (OLAP “Cubes”). (Boxes I and III in Figure 1.) [Notice it in Human Factors] (Malcolm Cohen’s Dying Navy A4 Pilots) (Private Communication by an applied Aviation Psychologist). (Cohen, M. M. et al 1973). Problem * Crusader A-7s were Navy aircraft launched by catapults from carriers. * Some A7s crashed (randomly) for no apparent cause after being catapulted off the carrier. “Similar problems were encountered with the A-4 Skyhawk”. * “They were ‘flown’ into the water, with wings level and nose down, about one minute or less after the catapult launching.” * “The planes always appeared to be under pilot control, and the pilots never declared an emergency, indicating that they had no indication that their airplane was not climbing as expected.” * “The accidents only occurred on dark moonless nights, when there were probably no external visual cues to let the pilots know that they were flying into the water.” * “The A-7 airplanes did not have after-burners, and the angle of attack was critical to avoid stalling. Immediately after the planes were launched, they started to climb out normally, but before reaching 1000 feet, they were observed to slowly fly towards the water. Probably, (and as demonstrated in simulation studies) pilots felt that their climb angle was "excessive", and to avoid stalling, they gradually lowered the nose of the aircraft to the point that it was in a shallow dive (although they still believed it to be climbing).” “Data” * An “Artificial Horizon” gauge problem had been fixed, therefore the problem was not the gauge. Survey Data * Artificial Horizon gauge not trusted. Figure 4. Actual situation. Malcolm Cohen’s – Theory 1) The Crusader A-7s had noted reliability. 2) Pilots can’t see outside and don’t trust gauge even though it is correct. 3) Some perception must cause “stick forward” into water. It has to be a positional perception miscue. Figure 5. Perceived situation – M. Cohen. “Data” Dr. Cohen’s theoretical model of a pilot’s psychological misperception was verified by experimenting with pilots in an A-7 cockpit mockup on a human centrifuge which could simulate the gravity + catapult vector sum as in Figures 4 and 5. Malcolm Cohen’s – Solution: * Train pilots to understand incorrect perception. * Train pilots to trust artificial horizon gauge. Result: No more crashes due to this cause (Box III in Figure 1 with verification in box IV). [Notice it in Current New Technology] (Web Services and/or XML Languages) * XML based. * Standards based. * “Applications” require defining general languages for new uses (language architecture == vocabulary.) * Requires understanding of an application domain. Often used for application integration. * Limited window of opportunity for dissertation level work: there was a two to three year window for innovative work. See section 7 below, Problem Characteristics 2). Example: Product Costing Framework; Best price subassemblies (Konstaninou, 2004) * WS / XML based – Industry integration. * Standards based – some not yet in place. * Needs many specific languages – some exist * Small restaurants as a real example * Dozens of DFDs and Use Case Diagrams. * Very intricate NEW/USEFUL architecture (Box III in Figure 1.) * Literature search was global: the UN was a standards player. Example: SARBOX Compliance Architecture [In process] Enterprise Accounting / Auditing Integration (Burns, 2004). Its characteristics are: * WS / XML based – Enterprise integration. * Standards based plus it needed one new one. * Needs one or two specific languages. * Accounting Standards as a real driver. * ~4/Ea. DFDs and Use Case Diagrams. * Straight forward architecture. * Literature search was US-based only. * Virtual close is a side effect (Box III in Figure 1.) 7. DEFINING A RESEARCH PROBLEM “Advanced Technology” often describes problem solving where we can see the problem and we know that the solution probably exists but we don’t know how to solve it. Ad-tech involves using known methods in new and clever ways. Weaker than Ad-tech but possibly as large a job, is what is called a “project”. In a project there is no real new problem to solve, only a job to make or do something. It might take great ingenuity and effort, but it adds nothing to the store of general knowledge. We know we can do the project, although it might take a lot of clever work. In research, we often have to first find the exact problem, state it, and then try to solve it. The aim is to find new knowledge. Usefulness is a sometimes a business criterion, but usefulness can be interpreted in ways that don’t always involve profit. Defining a problem involves a paradox: we have to narrow our focus to a specific problem (don’t boil the ocean) vs. we want to produce a general result, one that has applicability to more than just the problem we choose to solve. Some Problem Characteristics to Consider * Totally new area (this is difficult) vs. extensions of known work (easier). EX: Array Structure vs. Web Services for B2B. * Early explorations of a new area – “picking the low lying fruit”. EX: The early applications of linear programming. vs. Late into an area – it is now harder to find a meaningful problem. EX: Web Services applications. * Popular (everyone is working in the area) vs. “A Waste of Time” EX: Web Services and XML vs. Einstein’s attempts at grand unification (too risky for the “young”. Electro-Weak unification was done by people aged much less than 50). Some Opinions: Ron Frank (about Dissertations) It is best to find your own topic, one that YOU are interested in. Look for needs to be met. Think like a science fiction author - imagine. Look for “really stupid” systems that “anyone could design better”. General new architectural solutions are fair game. You can get a topic from your advisor (as a last resort). If you are in an institution where you MUST take a topic from your advisor, understand that you are possibly being exploited to do your advisor’s work and that you are not learning to be a self-sufficient innovative researcher. Have a Vision: * It does not yet exist (show this). * It can exist (argue this). * It should exist (argue this). * Make it exist (do this). * Feasibility is often enough. You don’t wait to “get an idea” and then start writing. You start writing and then get ideas! Lipman Bers’ quote: (Bers, 1967) (About Math Research) “There is an infinite number of true theorems. Work only on important ones.” (Work on what appears to be “useful” elsewhere). Eugene Wigner’s quote: (Wigner, 1960) (About Math in Physics) “The unreasonable effectiveness of mathematics.” [Any mathematics is probably eventually useful.] Aristotle (Aristotle) Beauty’s Razor: Let beauty lead you. “The mathematical sciences particularly exhibit order, symmetry, and limitation; and these are the greatest forms of the beautiful.” This can be inferred from the writings of many philosophers, including Einstein too. 8. SOME GENERAL COMMENTS Work on what you know and like or will have fun learning, if you have a choice. For example my recent array work: * I worked in APL development on and off for over 8 years. * I worked on the array problem on and off for 10 other years on my own time. * I generalized cubes to N-D Bricks. * I found a master equation in null arrays (by recursion). * I developed an explicit algorithm for all sub arrays’ cells. * I developed code in APL, C++, and Java for experimental work. * I found potential applications in Hyper cube machine architectures and data mining OLAP multi-dimensional data arrays. A good topic should be rich in further questions. EX: In the array structure problem: * What if N (dimension) is negative or complex or continuous? * What if n (length) is negative or complex or continuous? * The master equation allows subtraction of arrays – what does this mean? (it is array annihilation!) * Generalization to ragged arrays? * What about arrays of arrays (it yields set theory? (More, 1979). * What about n-continuous arrays and quantum computing? (The master equation does hold!) Work estimation is a problem (notice the cause of estimation errors) 1) Defining the problem is 50% of the work but more than ½ the emotional pain. 2) Finding the solution is 50% of the work but more than ½ the fun. 3) Writing it up and doing production is 50% of the work but most of the real pain. ;) Plan for doing new reading and learning (equivalent to nine graduate credits in one year for a dissertation). Literature search is NOT Research! Research is CREATIVE! Have fun or you may not finish. Prepare for failures and false starts along the way. For example: * Cape Canaveral, became Cape Kennedy, is now back to Cape Canaveral again. * It was called “Cape Carnival” Because of all of the early spectacular rocket failures. * Learn to go down in flames with grace and style. Remember the Phoenix. “The road to success is paved with the bricks of failure” (RIF). Your real problem is to find how to create added value by solving a problem. Being able to state a topic or an idea does not mean that you know what you are talking about - or better – that you are talking about what you know. Einstein: About Research “ If we knew what it was we are doing, it would not be called research.” (Einstein , 2002). Doing research is itself a project; apply your project management skills. This should be a SMALL but ongoing effort! 1) Lay out a Gantt chart (back of the envelope). 2) Do a work breakdown structure. 3) Constantly monitor actual against baseline. 4) Do an earned value analysis (value added vs. time spent). 5) Don’t estimate your time to completion based on your writing schedule. Research time has to be factored in. Writing about your own contribution is not a matter of just describing an idea. It is a matter of proving it or showing how to do it. Ideas are ~~ $0.10 / dozen. Usefully articulated and proven ideas are not common. Implemented ideas are rare. Accepted new ideas are extremely rare. It is up to YOU to schedule. It is up to YOU to control the process: i.e., initiate meetings, set goals, and set criteria. “Drs.” don’t try to fake it. They boldly take large slips in the schedule. (Brooks, 1995 - pg. 24.) It is up to YOU to find help if needed. It is up to YOU to distinguish hand-waving bull from insightful analysis. If you have spent a lifetime being sloppy, your advisors can’t cure you. “Repent oh you sinners!” Warning About “DATA” * “DATA” often means SURVEY DATA. * A few research topics use surveys. * Most research topics do not. * “DATA” might mean relevant examples. * “DATA” might mean similar systems. * “DATA” might mean previous work. * “DATA” might mean environmental variables. Notice the similarity of the research iteration below to iterative software development: Start outer loop: 1.Define the Question or Problem. 2.Define Over-All Goal. •Subdivide to N parts: a. Posit solution part i. b. Find solution part i. •Repeat for all i in N. 3.Evaluate Total Solution vs. Goal. •Might find new problem facets. •If Satisfied – Submit Results. 4.Else: Go to 1 using new insights. Notice the similarity to The Scientific Method: (see the state diagram in the appendix, section 16). 9. PROBLEMS FOR NEW RESEARCHERS * No previous experience doing research. Research is a skill. Take a course in it. Do research. * Growing up (the world does not owe you a topic). EX: Getting an idea - start out working in someone’s lab. * Having a passion for the topic EX: Goddard and Rockets in 1926. Yes - modern rocketry was invented here – not in Germany. * Narrowing the focus in the beginning. * Finding the most general problem solved by the solution at the end. * Writing something every day (even a list of what’s not known). * Keeping a notebook. Writing in general. See section 10 below. * Doing something on your own, (not just what your advisor tells you to do). * Taking command of the topic. YOU are the world’s greatest expert in the topic. * Library “research” is NOT research. It is the Literature Search. * Research requires independent creative thought and often experimentation: not just getting an idea and writing, writing, writing, writing, writing, and writing. * Beware ANXIETY BLOCK when you are getting started. Just write something to start and “Plan to Throw One Away” (Brooks, 1995 - Chapter 11). Just do it. * Do something, anything – it doesn’t matter what. Don’t think; do. Once started, correct your mistakes. * Once Started: Beware AVOIDANCE BEHAVIOR. Procrastination is well known. But Substitution is more insidious: There are always more important or more pressing activities to substitute for your research work, thus enabling avoidance. * Avoid the content free highly referenced discussion. This is developing a discussion by just catenating a set of references and providing the reader with no direct information in your own words. This is wrong and not research! * You are probably ”ilgraphic”. Most of us can’t read or write diagrams effectively (graphic literacy). I don’t mean pictorial data analysis à la Tufte (Tufte, 2006) nor do I mean Spatial Literacy (Newcombe, 2006). I mean the use of concept diagrams such as DFDs, ERDs, Flowcharts (Hoffer et al, 2004), and the UML (Fowler 2003). Learn this skill. * Coping unemotionally with interpersonal clashes. The research tutorial lead-up to the dissertation and the defense of the thesis is inherently an adversarial process. * Learning to “read between the lines”. Research papers are written in an era by personalities who have made other contributions. Understanding a paper often requires the reader to have a cultural background similar to the author’s. * Finishing. I. Bernard Cohen: About a dissertation: [This also applies to corporate research] (He had many dissertation students in the history of science and mathematics. He was the first American doctorate in the history of science). “Don’t get it perfect; get it done.” (Gabiner, 2004). 10. MAINTAIN A NOTEBOOK!!! Buy a bound notebook, usually with sewn in signatures. Metal Spirals are not acceptable. There are Computation Notebooks (AMPAD, 2006) [~$13.00] or Lab Notebooks (SNCO, 2006) [~ $23.00-$42.00] that are good. They have a left margin marked, quadrille paper, preprinted pagination, and maybe a signature area. I recommend the # 2001 (SNCO, 2006). This is The Only Place On Earth That You Write, Diagram, Or Doodle About Your Research. Most corporate research groups require that a series of notebooks be kept. They are by definition the property of the company. The VP of Dow Chemical Research once told me that 80% of their net revenue at that time was directly traceable to two research notebooks. This is not a class notebook. It is not a personal diary. It is not used for ANYTHING else. It is not your research report or your dissertation. It is only a place to put ideas, thoughts, and data. Remember, paper is cheap. Every page is sequentially numbered. Each new topic or coding project starts with a line across the whole page and a date in the left margin with a short descriptive name in the margin or at top. Never remove pages!!! You can make margin notes to keep track of the structure of the notes. For example, use circles and dates for TO-DO items. Cross them out when they are completed. If you need to, you can generate an index of topics and dates on the last few pages as you go along. This helps if you are jumping topics a lot. Some notebooks (SNCO, 2006) come with a separate section for a table of contents. Advisor and other meeting notes go in here, as do phone notes and numbers and email-IDs of colleagues. No page-width line is put in until the start of the next topic. Interpolated sections from other topics are marked by a vertical mark in the margin, a blank area, and the name of the topic. Paper is cheap. You can tape in small listings, or other documents from outside sources or physical bugs (Hopper’s moth, 1945). Don’t use staples - they tear paper. There should be no other paper you use to write on - ever. Only IMPORTANT email (decision agreements) get pasted in. Write under your paste-ins what they are, just in case they fall out. Put your name, advisor’s name, start date, and leave a space for end date on the front outside. You may generate more than one notebook. Put your address, phone number and email ID on the inside front cover so that when you lose it, the finder can contact you for pickup. Near your info, put a polite request to the finder to return your notebook - it can help. Label the book “Personal and Confidential” just to emphasize how you view it. Offer a small reward for its return. If you have a patentable idea, document it in great detail with tutorial comments and immediately get (or leave space for quickly getting) two colleagues to read and UNDERSTAND the idea. Then have them write, sign, and date a statement right there that they have read the idea and have understood it. That is why some lab notebooks come with a signature area on the bottom of every page. This is also why the book must be bound, every page numbered, and every item dated - so that there can be no suspicion that the idea pages or statements were inserted at a later date. You too can win a patent lawsuit for billions of dollars (the Laser) based solely on this kind of documentation of a good idea. If you pocket-record (dictate) daily notes, they get written in THAT NIGHT if not immediately. Notice that Grace Murray Hopper’s original notes (including the original computer bug scotch taped into it) were written into just such a notebook – paginated, bound, quadrille paper with defined wide margins. (Hopper, 1945). 11. THE REALITY OF THE SUBCONSCIOUS Many great researchers have testified to the reality of the subconscious (the unconscious) and the virtue of using it (Hadamard, 1954). The idea is that if you work on a specific problem and prime your subconscious with facts and background material, you can then consciously tell yourself to work on the solution while you do other things. The subconscious then reliably comes back later with a contribution to the solution. One version of this is the admonition to “sleep on it” when you can’t make progress. Unfortunately, the testimony tells us that, in some cases, this subconscious process can take twenty years or more for really hard problems. This use of the subconscious is itself a skill that has to be developed by repetition. In the beginning, it requires faith until you experience its reality. 12. RECOGNIZE THE STATE OF FLOW It can happen, if you are lucky, that your subconscious and your conscious minds start producing results in what seems a continuous flow. It is important to recognize this, if and when it happens, and devote all your energies to doing your research. This creative state is hard to reach and goes away quickly – so take full advantage of it when it hits. 13. THERE IS NO RESEARCH IN THE IS CURRICULUM Some schools, such as Pace, include an introduction to research in the IS curriculum, at both the BS/BA an MS/MA level. However the Proposed MSIS Curriculum 2006 (Gorgone, et al, 2005) does not. It should. 14. CONCLUSIONS Research is a skill, so do it to learn it. It is based on writing, so be prepared to write. It is the way we as a society learn, innovate, and progress, so get involved. 15. REFERENCES AIP. The Bohr Sommerfeld or Old Quantum Mechanics. AIP last accessed 10/15/2006. http://www.aip.org/history/heisenberg/p05c.htm AMPAD, Computation Notebook. Quadrille, and paginated, but not preprinted for book serialization and no signature area. http://www.officemax.com/ search AVE43648. Last accessed 6/30/2006. http://www.ampad.com/ashop/default.asp Search 22-156. Last accessed 6/30/2006. Aristotle. Metaphysica, 3-1078b. Bers, Lipman. (1967) Personal communication. Brooks, Fredrick P. Jr.. (1995). The Mythical Man Month. , Addison Wesley. ISBN 0-201-83595-9 Burns, Kevin. (2004). Personal Communication. Cohen, M. M., Crosbie, R. J. and Blackburn, L. H. (1973) “Disorienting Effects of Aircraft Catapult Launchings.” Aerospace Medicine, 1973, 44:37-39, and personal communications over the years. Einstein, Albert. (2002) Quoted in Scientific American September 2002 Vol. 287 No. 3 (a special issue on Time) in the Antigravity column by Steve Mirsky: “Einstein’s Hot Time” pg. 152.] Fowler, Martin. (2003). UML Distilled: A Brief Guide to the Standard Object Modeling Language. Addison-Wesley ISBN: 0321193687. Gabiner, Judith V. (2003) “I. Bernard Cohen (1914-2003)” A memorial article by one of his students in the MAA Focus Vol. 23 # 6, August/September 2003 pg. 24. Gorgone, John T., Gray, Paul, Stohr, Edward A., Valacich, Joseph S., Wigand, Rolf T. 2006, MSIS 2006 Model Curriculum and Guidelines for Graduate Degree Programd in Information Systems (Draft 10-20-2005) ACM and AIS. Hadamard, Jaques. (1954). The Psychology of Invention in the Mathematical Field. Dover Publications. ISBN: 0486201074 Hoffer, Jeffrey A., George, Joey F., Valacich Joseph S. (2004). Modern Systems Analysis and Design (4th Ed). Prentice Hall. ISBN: 0131454617 Hopper, Grace Murray. Notes on Panel F, Relay #70 of the Mark II Aiken Relay Calculator at Harvard. (1945) Last accessed 7/7/06. .http://www.history.navy.mil/photos/pers-us/uspers-h/g-hoppr.htm Konstantinou, Stella. (2004), A Web Services Framework for Automating Constituent Product Costing Operations of Businesses Producing and Selling Their Own Products. Pace University Dissertation. Leedy, P. D. and Omrod, J. E. Practical Research 8th Ed. (2005) by Merrill Prentice Hall, ISBN-0-13-110895-6. Magiqtech-1. http://www.magiqtech.com/ Last accessed 7/7/2006. Magiqtech-2. http://www.magiqtech.com/press/MagiQ_VZ.pdf Last accessed 7/7/2006. More, Trenchard. (1979) “The Nested Rectangular Array as a Model of Data”. International Conference on APL. Proceedings of the international conference on APL: part 1 New York, New York, United States Pages: 55 – 73 1979 ISSN:0163-6006 Newcombe, Nora S. 2006. “A Plea for Spatial Literacy”. The Chronicle of Higher Education Pg. Sec. B20, March 3, 2006. Shaffer, C. A., (2006) “Experiences Teaching a Graduate Research Methods Course”. PP. 97-101 in ACM SIGCSE Bulletin Vol. 38 No. 2 June 2006. SNCO, Scientific Notebook Company. Laboratory Style: Quadrille, paginated, preprinted for book serialization [~$23 w/S&H] http://www.snco.com/ (#2001) Last accessed 6/30/2006. Tufte, Edward. http://www.tufte.com/ Last accessed 7/12/06. Wigner, Eugene. (1960) "The Unreasonable Effectiveness of Mathematics in the Natural Sciences," in Communications in Pure and Applied Mathematics, vol. 13, No. I (February 1960). New York: John Wiley & Sons, Inc. 16. APPENDIX: THE SCIENTIFIC METHOD STATE DIAGRAM (Notice the iteration within an iteration. Also notice how this is very like agile code development and system development.)