EMBA Program Assessment: 
Benchmarking for Success

Owen P. Hall, Jr., P.E., Ph.D.
ohall@pepperdine.edu

Thomas J. Dudley, D.B.A.
tdudley@pepperdine.edu
Pepperdine University
Graziadio School of Business & Management
6100 Center Drive
West Los Angeles, CA 90245

Keywords: EMBA, program design, alumni survey, benchmarking

ABSTRACT

Schools of business are under increasing pressure to enact significant reforms within the traditional educational process. These developments call for new and innovative curriculums for providing cost-effective management education. Executive management education in particular is undergoing a transition to a more experiential learning environment.   The purpose of this paper is to present the results of a satisfaction survey of alumni from an executive MBA program (EMBA) and to outline how learning systems technology can be used to improve program performance. The data show that program quality and perceived student value are strongly linked.  The insights gained from this study suggest that the increased use of technology in EMBA type programs can assist in improving quality and will require a coordinated effort among faculty, suppliers and administrative staff.


1. INTRODUCTION
The number of working managers returning to the classroom is growing (Edgington, 2004). The student/manager is interested in a practical curriculum that features convenience and focuses on results. To meet these basic requirements, the general curriculum direction in management education is moving increasingly towards a learning-centric perspective (Driver, 2002). Learning support technology (LST) is a key ingredient in this new delivery stratagem. Many student/managers who have been exposed to LST tend to favor this approach over the traditional classroom-centric model (Lundgren, 2003).  These systems are receiving increased attention because they offer students engaged in management education both a flexible and an integrated learning experience (Kathawala, 2002).

Executive MBA (EMBA) programs are usually conducted in a style and format different from standard MBA programs.   Some specific characteristics unique to most EMBA programs include the following:
* Reduces emphasis on traditional lecture format.
* Uses lock-step cohort student groups. 
* Focuses on collaboration and hands-on exercises.
* Takes into account student work demands and travel schedules.
* Permits students to use actual work projects in courses.
* Features more learning from other students (andragogical).

The characteristics of an EMBA learning paradigm call for both an integrated as well as a results oriented style which in many ways mimics modern business practice (Schelfhaudt, 2005). This approach is particularly appropriate for working managers since many already possess a rich work related experiential base that can contribute to the collective learning environment (Monks, 2001).  In an integrated management education pedagogy the focus is on understanding how basic management functions such as operations, finance and marketing are linked. Furthermore, graduate students tend to participate more in learning systems that are content rich and that feature extensive variety (Neo, 2004). These perspectives provide the impetus for the use of learning system technology to support program and curriculum goals (Jorgensen, 2002). 

Ongoing assessment represents another essential element in the curriculum design process. Figure 1 illustrates a curriculum assessment process based on benchmarking.   Benchmarking has seen growing usage throughout schools of business (Amin, 2003). An essential ingredient in the benchmarking process in education, as illustrated in Figure 1, is student feedback (Drexler, 2000). Internet based surveys provide one means of obtaining student feedback on curriculum design, delivery and related program dimensions such as student support services. Figure 1 also highlights the large number of stakeholders that need to be considered in the curriculum assessment and design process (administration, faculty, students, business, accrediting body).
As one component of this assessment paradigm a survey of EMBA graduates was conducted on a variety of program issues. The results of the survey are presented in the following section.

Figure 1 – Assessment Process

2. ALUMNI SURVEY
A 30-question instrument was developed to assess the alumni’s overall satisfaction with a 20 month executive MBA (EMBA) program. After a pre-test, the survey was distributed to approximately 500 alumni that had graduated between 1999 and 2005. The survey was conducted over the Internet.  Some specific alumni demographics of the surveyed cohort group are reported in Table 1 along with a comparison with the top 25 EMBA worldwide programs as ranked by Businessweek (2005). These data shows a higher percentage of females in the EMBA program under study compared with the average for the top 25 ranked programs.

Table 1 – Comparison of Selected Descriptive Statistics

Factor
Cohort Group
TOP 25 EMBA
(average)
Age (yrs)
36
36.1
Gender(% female)
37
18.8
Experience (yrs)
12
12.7
Graduate Deg. (%)
9
26
The number of initial respondents to the 500 surveys distributed was 63. An Excel database was developed from the questionnaire responses. The demographic factors (age, gender, income, organization type) were characterized using dummy variables due to the limited size of the preliminary response rate. The alumni preferences were measured on a standard 1-to-5 Likert scale. Table 2 provides a sampling of selected survey preference questions along with the associated metric and ranking (5=strongly agree). 

Table 2 – Selected Alumni Satisfaction Statistics (5=strongly agree)

Question
Mean
Rank 
Glad enrolled 
4.35
1
Learning groups important
4.35
1
Capstone project important
4.21
3
Recommend
4.03
6
More technology
3.87
10
Program Value
3.73
13
Program Quality
3.13
19
These data suggests a wide variance of student perspectives (4.35 to 3.13). “Glad I enrolled in the program” is ranked first while “Consistent Program Quality” is ranked last at 19th.  Clearly, improving quality consistency throughout the program should be a top priority. One approach for addressing this challenge is through the use of learning technologies.

3. RESULTS ANALYSIS
The database was further explored using correlation analysis and neural nets. The only two statistically significant correlations (Pearson) at the 0.05 level were:
* Gender and Age (r=-.32, p=.02)
* Quality and Value (r=.55, p=.00)
As a result, quality was added to the predictor variable set as part of the neural net analysis. Neural networks (NN) have seen increased use in educational applications since the underlying relationships between variables are somewhat ill defined as in the case of student learning (Gonzalez, 2002). Table 4 shows the relative importance of the predictor variables where perceived value is the target variable (Huang, 2002). 

Table 4 – Model Comparisons for Perceived Value (Target)

Variable
NN (weights)
Quality
0.81
Age
0.12
Income
0.06
Gender
0.01
Org. Type
0.00


COD
0.36
     
These results simply underscore the importance of quality as related to perceived program value. 

4. TECHNOLOGY INITITIVE
The survey data indicated that program quality was ranked last and the subsequent analysis revealed that quality is directly related to perceived value.  Learning systems technologies (LST) represent one approach for improving quality consistency in an EMBA type program (Nambisan, 2004). This observation is particularly insightful due to the fact that technology was used rather sparingly in the surveyed EMBA program. Some specific LST applications that appear particularly attractive for EMBA programs in general and the surveyed program in particular include the following (Creaser, 2002):
* Blogging (Complements “checking in” process).
* Chatrooms (Linear and threaded case analysis).
* Simulation (Strategic and discipline specific).
* Virtual experiences (Industrial tours and seminars).
* AI based support (Search engines). 
Within this context there are a number of program structural issues that need attention. These include system implementation and operation. Some specific administrative challenges in implementing learning system technology are highlighted in the following:
* Training faculty for successful system deployment and usage.
* Providing the highest quality standards.
* Setting specific performance goals and metrics for measuring student performance and expectations.
* Maintaining format consistency throughout the program
* Preparing students for entry and ongoing use of learning nets.
* Establishing the overall school culture that fosters technological innovation.
Successful system implementation requires a coordinated effort among program faculty, technologists and administrative staff. One implementation strategy is to deploy a prototype program using a specific executive MBA section. Typically, EMBA class sizes tend to be relatively small. This condition tends to help ameliorate the standard problems associated with implementing significant changes in management education programs with a large student body.  Student acceptance is arguably the most significant factor in deploying a new program (Martins, 2003). Developing the internal capability to institute an increase in management technology can be complex and expensive. Furthermore, an internalized approach may not take advantage of the ongoing developments in delivery technology, e.g., search engines. One emerging implementation strategy that is designed to help overcome the aforementioned issues consists of developing institutional partners with both content and application service capabilities (Sorel, 2001). This approach focuses specifically on the basic ideas behind supply chain management and is consistent with the increased use of suppliers in continuous operations such as those found in most business programs. 

5. CONCLUSIONS
The call for the increase use of learning technology throughout EMBA programs is on the rise. The purpose of this paper is twofold: 1) to present the results of a survey of EMBA alumni and 2) to outline how learning systems technology (LST) can enhance EMBA delivery and quality. LST provides a vehicle for moving from a teaching-centric towards a learning-centric educational paradigm, which is particularly attractive for executive graduate management education. The survey data show that assessed quality and perceived program value are strongly linked. A neural net analysis of the database further underscores the link between program quality and perceived value. One strategy for implementing learning systems technology is through institutional partners that possess both content and application service capabilities

6. REFERENCES
Amin, M.; Amin, N. 2003. Benchmarking learning outcomes of undergraduate business education. Benchmarking: An International Journal, 10 (6), 538. 
 Creaser, K. 2002.  Online education tools. Australian CPA, 72 (10), 49.
Driver, M. 2002. Investigating the benefits of web-centric instruction for study learning: An exploratory study of an MBA course. Journal of Education for Business, 77(4), 238.
Drexler, J. 2000. Using total quality process and learning outcome assessments to develop management curricula. Journal of Management Education, 24 (2), 167.
Edgington, R. 2004. Application Trends, Survey Executive Report 2004, Graduate Management Admission Council.
Gonzalez, J. 2002. Artificial neural networks: A new approach to predicting application behavior. Research in Higher Education, 43 (2), 235.
Hitz S.; Turnoff, M. 2005. Education goes digital; the evolution of online learning and the revolution in higher education. Association for Computing Machinery, Communication of the ACM. 48 (10), 59.
Huang, H. 2002. Student perceptions in an online mediated environment. International Journal of Instructional Media, 29 (4), 408.
Jorgensen, D. 2002. The challenges and benefits of asynchronous learning networks. Reference Librarian, 77, 3. 
Kathawala, Y.; Abdou, K.; Elmuti, D. 2002. The global MBA: a comparative assessment for its future. Journal of European Industrial Training, 26 (1), 14.
Lundgren, T.; Nantz, K. 2003. Student attitudes towards Internet courses: A longitudinal study. Journal of Computer Information Systems, 43 (3), 61.
Martins, L.; Kellermann, F. 2003. A model of business school students' acceptance of a web-based course management system. Academy of Management Learning & Education, 3 (1), 7.
Monks, K.; Walsh, J. 2001. The role of postgraduate education in management development. Journal of European Industrial Training, 23 (2), 148.
Nambisan, S.;Wilemon, D. 2004. Industry should help redefine the agenda for technology management education. Research Technology Management, 47(6), 9.
Neo, T., Neo, M. 2004.  Classroom innovation: engaging students in interactive multimedia learning.  Campus – Wide Information Systems, 21 (3), 118.
Schelfhaudt, K.; Crittenden, V. 2005. Specialist or generalist: Views from academia and industry. Journal of Business Research, 58 (7), 946.
Sorel, R.; Dear, R.; Edge, D. 2001.  Evolution of web-based distance learning strategies.  International Journal of Educational Management, 15 (4), 245. 










Owen P. Hall, Jr. received his Ph.D. from the University of Southern California and undertook post-doctoral studies at the Center for Futures Research. Dr. Hall is a Professor of Decision & Information Systems at the George L. Graziadio School of Business and Management, Pepperdine University. He is the recipient of a Charles Luckman teaching fellowship. He is a registered professional engineer, State of California. Dr. Hall is the author of four textbooks on computer based decision support systems. He has also authored over 50 technical papers on the subjects of decision support systems, forecasting software design, distance learning systems and artificial intelligence. Dr. Hall’s current area of research includes the application of artificial intelligent agents for search engine technology and the development of Internet based hybrid learning nets (HLN). 

Dr. Dudley received his DBA from the University of Southern California. In a career spanning some 40 years, the last 29 at Pepperdine University, he has consulted in varied industries as well as with nonprofit organizations and start-up companies. He sits on the boards of directors of SpaceLabs Medical, Inc., listed on the NASDAQ, and two nonprofit organizations--the Los Angeles branch of Recording for the Blind and Dyslexic and the America-China Association for Science and Technology Exchange. Dr. Dudley has written extensively and delivered numerous seminars to management groups and to the State Statistical Bureau of the People's Republic of China. He is a 1994 recipient of the Harriet and Charles Luckman Distinguished Teaching Fellows Award.