Performance Evaluation - Ingrid J. Guerra-Lopez - ebook

Performance Evaluation ebook

Ingrid J. Guerra-Lopez

0,0
239,99 zł

Opis

Performance Evaluation is a hands-on text for practitioners,researchers, educators, and students in how to usescientifically-based evaluations that are both rigorous andflexible. Author Ingrid Guerra-López, an internationally-knownevaluation expert, introduces the foundations of evaluation andpresents the most applicable models for the performance improvementfield. Her book offers a wide variety of tools and techniques thathave proven successful and is organized to illustrate evaluation inthe context of continual performance improvement.

Ebooka przeczytasz w aplikacjach Legimi na:

Androidzie
iOS
czytnikach certyfikowanych
przez Legimi
Windows
10
Windows
Phone

Liczba stron: 463




Table of Contents

Cover

Title

Copyright

PREFACE

ACKNOWLEDGEMENTS

THE AUTHOR

PART 1: INTRODUCTION TO EVALUATION

CHAPTER 1: FOUNDATIONS OF EVALUATION

A BRIEF OVERVIEW OF EVALUATION HISTORY

EVALUATION: PURPOSE AND DEFINITION

PERFORMANCE IMPROVEMENT: A CONCEPTUAL FRAMEWORK

MAKING EVALUATION HAPPEN: ENSURING STAKEHOLDERS’ BUY-IN

THE EVALUATOR: A JOB OR A ROLE?

THE RELATIONSHIP TO OTHER INVESTIGATIVE PROCESSES

WHEN DOES EVALUATION OCCUR?

GENERAL EVALUATION ORIENTATIONS

CHALLENGES THAT EVALUATORS FACE

ENSURING COMMITMENT

BENEFITS OF EVALUATION

BASIC DEFINITIONS

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 2: PRINCIPLES OF PERFORMANCE-BASED EVALUATION

PRINCIPLE 1: EVALUATION IS BASED ON ASKING THE RIGHT QUESTIONS

PRINCIPLE 2: EVALUATION OF PROCESS IS A FUNCTION OF OBTAINED RESULTS

PRINCIPLE 3: GOALS AND OBJECTIVES OF ORGANIZATIONS SHOULD BE BASED ON VALID NEEDS

PRINCIPLE 4: DERIVE VALID NEEDS USING A TOP-DOWN APPROACH

PRINCIPLE 5: EVERY ORGANIZATION SHOULD AIM FOR THE BEST THAT SOCIETY CAN ATTAIN

PRINCIPLE 6: THE SET OF EVALUATION QUESTIONS DRIVES THE EVALUATION STUDY

KEY POINTS

REFLECTION QUESTIONS

PART 2: MODELS OF EVALUATION

CHAPTER 3: OVERVIEW OF EXISTING EVALUATION MODELS

OVERVIEW OF CLASSIC EVALUATION MODELS

SELECTED EVALUATION MODELS

SELECTING A MODEL

CONCEPTUALIZING A USEFUL EVALUATION THAT FITS THE SITUATION

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 4: KIRKPATRICK’S FOUR LEVELS OF EVALUATION

KIRKPATRICK’S LEVELS

COMMENTS ON THE MODEL

STRENGTHS AND LIMITATIONS

APPLICATION EXAMPLE: WAGNER (1995)

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 5: PHILLIPS’S RETURN-ON-INVESTMENT METHODOLOGY

PHILLIPS’S ROI PROCESS

COMMENTS ON THE MODEL

STRENGTHS AND LIMITATIONS

APPLICATION EXAMPLE: BLAKE (1999)

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 6: BRINKERHOFF’S SUCCESS CASE METHOD

THE SCM PROCESS

STRENGTHS AND WEAKNESSES

APPLICATION EXAMPLE: BRINKERHOFF (2005)

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 7: THE IMPACT EVALUATION PROCESS

THE ELEMENTS OF THE PROCESS

COMMENTS ON THE MODEL

STRENGTHS AND LIMITATIONS

APPLICATION EXAMPLE

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 8: THE CIPP MODEL

STUFFLEBEAM’S FOUR TYPES OF EVALUATION

ARTICULATING CORE VALUES OF PROGRAMS AND SOLUTIONS

METHODS USED IN CIPP EVALUATIONS

STRENGTHS AND LIMITATIONS

APPLICATION EXAMPLE: FILELLA-GUIU AND BLANCH-PANA (2002)

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 9: EVALUATING EVALUATIONS

EVALUATION STANDARDS

THE AMERICAN EVALUATION ASSOCIATION PRINCIPLES FOR EVALUATORS

APPLICATION EXAMPLE: LYNCH ET AL. (2003)

KEY POINTS

REFLECTION QUESTIONS

PART 3: TOOLS AND TECHNIQUES OF EVALUATION

CHAPTER 10: DATA

CHARACTERISTICS OF DATA

SCALES OF MEASUREMENT

DEFINING REQUIRED DATA FROM PERFORMANCE OBJECTIVES

DERIVING MEASURABLE INDICATORS

FINDING DATA SOURCES

FOLLOW-UP QUESTIONS AND DATA

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 11: DATA COLLECTION

OBSERVATION METHODOLOGY AND THE PURPOSE OF MEASUREMENT

DESIGNING THE EXPERIMENT

PROBLEMS WITH CLASSIC EXPERIMENTAL STUDIES IN APPLIED SETTINGS

TIME-SERIES STUDIES

SIMULATIONS AND GAMES

DOCUMENT-CENTERED METHODS

CONCLUSION

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 12: ANALYSIS OF EVALUATION DATA: TOOLS AND TECHNIQUES

ANALYSIS OF MODELS AND PATTERNS

ANALYSIS USING STRUCTURED DISCUSSION

METHODS OF QUANTITATIVE ANALYSIS

STATISTICS

GRAPHICAL REPRESENTATIONS OF DATA

MEASURES OF RELATIONSHIP

INFERENTIAL STATISTICS: PARAMETRIC AND NONPARAMETRIC

INTERPRETATION

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 13: COMMUNICATING THE FINDINGS

RECOMMENDATIONS

CONSIDERATIONS FOR IMPLEMENTING RECOMMENDATIONS

DEVELOPING THE REPORT

THE EVALUATOR’S ROLE AFTER THE REPORT

KEY POINTS

REFLECTION QUESTIONS

PART 4: CONTINUAL IMPROVEMENT

CHAPTER 14: COMMON ERRORS IN EVALUATION

ERRORS OF SYSTEM MAPPING

ERRORS OF LOGIC

ERRORS OF PROCEDURE

CONCLUSION

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 15: CONTINUAL IMPROVEMENT

WHAT IS CONTINUAL IMPROVEMENT?

MONITORING PERFORMANCE

ADJUSTING PERFORMANCE

THE ROLE OF LEADERSHIP

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 16: CONTRACTING FOR EVALUATION SERVICES

THE CONTRACT

CONTRACTING CONTROLS

ETHICS AND PROFESSIONALISM

SAMPLE STATEMENT OF WORK

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 17: INTELLIGENCE GATHERING FOR DECISION MAKING

PERFORMANCE MEASUREMENT SYSTEMS

ISSUES IN PERFORMANCE MEASUREMENT SYSTEMS

CONCLUSION

KEY POINTS

REFLECTION QUESTIONS

CHAPTER 18: THE FUTURE OF EVALUATION IN PERFORMANCE IMPROVEMENT

EVALUATION AND MEASUREMENT IN PERFORMANCE IMPROVEMENT TODAY

WHAT DOES THE FUTURE HOLD?

CONCLUSION

KEY POINTS

REFLECTION QUESTIONS

REFERENCES AND RELATED READINGS

INDEX

End User License Agreement

List of Tables

CHAPTER 1: FOUNDATIONS OF EVALUATION

TABLE 1.1. Unique Perspectives of Needs Assessment and Evaluation

TABLE 1.2. Dimensions of Investigative Processes

CHAPTER 5: PHILLIPS’S RETURN-ON-INVESTMENT METHODOLOGY

TABLE 5.1. Overview of Findings

CHAPTER 7: THE IMPACT EVALUATION PROCESS

TABLE 7.1. List of Relevant Performance Indicators

CHAPTER 8: THE CIPP MODEL

TABLE 8.1. CIPP Applicability in Formative and Summative Evaluations

TABLE 8.2. Measured Indicators

CHAPTER 9: EVALUATING EVALUATIONS

TABLE 9.1. Program Evaluation Standards, 1994

CHAPTER 10: DATA

TABLE 10.1. Scales of Measurement

TABLE 10.2. Data Collection Plan Flow

TABLE 10.3. Commonly Used Indicators Grouped by General Categories

TABLE 10.4. Example of Linkages Between Results and Required Data

TABLE 10.5. Data Sources

CHAPTER 11: DATA COLLECTION

TABLE 11.1. Example of Ranking and Record Keeping in Using a Nominal Group Technique

TABLE 11.2. Typical Experimental Design for a Single Treatment

CHAPTER 12: ANALYSIS OF EVALUATION DATA: TOOLS AND TECHNIQUES

TABLE 12.1. Comparisons and Ways to Assess the Results

CHAPTER 13: COMMUNICATING THE FINDINGS

TABLE 13.1. Framework for Identification of Solution Recommendations

List of Illustrations

CHAPTER 4: KIRKPATRICK’S FOUR LEVELS OF EVALUATION

FIGURE 4.1. Kirkpatrick’s Four-Level Model of Evaluation for Learning

FIGURE 4.2. Comparison of Test Performance

FIGURE 4.3. Combined Totals of Quotes Sent Out by Sales

CHAPTER 7: THE IMPACT EVALUATION PROCESS

FIGURE 7.1. Impact Evaluation Process

CHAPTER 12: ANALYSIS OF EVALUATION DATA: TOOLS AND TECHNIQUES

FIGURE 12.1. Bell Curve

FIGURE 12.2. Positively and Negatively Skewed Distribution

FIGURE 12.3. Bar Graph

FIGURE 12.4. Line Chart

FIGURE 12.5. Pie Chart

FIGURE 12.6. Progression from Analyzed Data to Recommendations for Action

Guide

Cover

Table of Contents

Begin Reading

Pages

C1

iii

iv

v

xi

xiii

xiv

xv

1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

39

40

41

42

43

44

45

47

48

49

50

51

52

53

54

55

56

57

58

59

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

239

240

241

242

243

244

245

246

247

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

295

296

297

298

299

300

301

302

303

304

e1

PERFORMANCE EVALUATION

Proven Approaches for Improving Program and Organizational Performance

INGRID GUERRA-LÓPEZ

Copyright © 2008 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-BassA Wiley Imprint989 Market Street, San Francisco, CA 94103-1741—www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at www.wiley.com/go/permissions.

Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002.

Jossey-Bass also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data

Guerra-López, Ingrid.Performance evaluation : proven approaches for improving program andorganizational performance / Ingrid Guerra-López. — 1st ed.p. cm.Includes bibliographical references and index.ISBN 978-0-7879-8883-8 (pbk.)1. Employees—Rating of. 2. Performance standards.3. Organizational effectiveness. I. Title.HF5549.5.R3G84 2008658.3'125—dc22

2007050973

ACKNOWLEDGMENTS

Recognition for any achievement is usually owed to a group of people rather than any one individual alone. First, I thank Jossey-Bass and, in particular, Andrew Pasternack for his interest in my potential contributions, which was the major catalyst for this work. I also thank Seth Schwartz for his graciousness and flexibility as I encountered a glitch or two along the way. Much gratitude is owed to Beverly Miller and Barbara Armentrout for their outstanding editing and proofing work.

I thank my mentor, Roger Kaufman, who continues to push me to think critically and support my ideas with evidence. A special recognition is owed to Hillary Andrei for her invaluable support in various aspects of my evaluation work. My gratitude is also owed to Rahat Sharma for her formatting support. I am grateful as well to my students, colleagues, and clients who are constant sources of ideas and reflections. The work in this book is the product of such ideas and reflections.

Finally, I am thankful to my husband, Jorge, for being once again excited, and patient, as I embarked on yet another project.

PREFACE

There are excellent books on evaluation already in print. Some focus on the history and theory of evaluation, some provide a comprehensive analysis of evaluation models and approaches, and some primarily offer insights into methodology. Although this book addresses these concepts, it is not intended to replicate or refute any other work. Rather, it is organized in such a way as to illustrate evaluation in the context of performance improvement. This work is directed at the following audiences:

Performance improvement practitioners who seek to do evaluation well, wish to become sufficiently versed in evaluation so as to work well with evaluators, or want to integrate an evaluative perspective in all phases of performance improvement, from needs assessment to implementation, to evaluation itself

Evaluators who seek to do systemic, performance-focused evaluations with the ultimate end of improving not only programs or solutions but also the organizations and the clients, including our shared society, whose needs they are meant to meet

Students who want a solid conceptual grounding in evaluation and a guide for applying such concepts in courses and their own work and who will generate the future evaluation and performance improvement models, in part based on concepts presented here

Instructors who are looking for a text that addresses important foundations of evaluation and presents the models and approaches in the context of the performance improvement field

Clients, including those external to the organization stakeholders, who wish to become better-informed consumers of and partners in evaluation efforts

If you want to more clearly understand the impact this book has had on your thinking about evaluation, then I suggest that before reading it, you respond to the questions listed below. Once you have completed reading the book, I recommend answering each of the questions again, paying close attention to how your views have changed, if at all, as a result of reading this book. This exercise is meant to support deeper reflection and insight about fundamental assumptions associated with evaluation and implications for practice.

What is your current model or approach to evaluation? Use these questions to guide your response (if you don’t have a current model or approach, think about what your basic assumptions about evaluation are):

What aspects of evaluation are addressed by your theory?

What are the main purposes, processes, or mechanisms associated with your model?

Does your model target a specific setting? If so, what is it?

With what other models (if you know of any) is your model most compatible? Explain how they are compatible.

With what other models is your theory least incompatible? Explain what the incompatibilities are.

In what ways has your model of evaluation been applied? Was it successful? What, if anything, would you have done differently? What were the biggest lessons you learned? Provide specific situations and examples.

The book is divided into four parts, beginning with an introduction to the foundations of evaluation, then proceeding to a collection of models chosen specifically for the reputation and applicability in the performance improvement field. Part Three looks at the tools and techniques that are common in various evaluation perspectives, and Part Four concludes with a look at continual improvement and the future of evaluation in performance improvement.

THE AUTHOR

Ingrid Guerra-López is an associate professor at Wayne State University in Detroit, associate research professor at the Sonora Institute of Technology in Mexico, and principal of Intelligence Gathering Systems. She publishes, teaches, consults, and conducts research in the areas of performance measurement and evaluation, assessment and analysis, and strategic alignment. She has published four other books, including Evaluating Impact: Evaluation and Continual Improvement for Performance Improvement Practitioners, and is coauthor of Practical Evaluation for Educators: Finding What Works and What Doesn’t. She has published numerous chapters in books in the performance improvement field, including the 2006 Handbook for Human Performance Technology. Her articles have appeared as well in Performance Improvement, Performance Improvement Quarterly, Human Resource Development Quarterly, Educational Technology, Quarterly Review of Distance Education, International Public Management Review, and the International Journal for Educational Reform, among others.

PART 1INTRODUCTION TO EVALUATION

CHAPTER 1FOUNDATIONS OF EVALUATION

This chapter defines and describes evaluation and sets the frame for this book within the principles of performance improvement. Various kinds of evaluation, as well as some closely related processes, are differentiated from each other. The basic challenges that evaluators face are laid out, and the reason that stakeholder commitment is so important is examined. The benefits of evaluation to an organization are listed. Finally, definitions are provided for some key terms used throughout the book and in the evaluation field.

In our daily lives, we encounter decision points on an almost continuous basis: Should I do this, or should I do that? Should I go right or left? Should I take the highway or the back streets? Should I buy now or later? Should I take my umbrella today or not? Life in an organizational setting is no different: We face decisions about which programs to sustain, which to change, and which to abandon, to name but a few organizational dilemmas. How do members of an organization go about making sound decisions? With the use of relevant, reliable, and valid data, gathered through a sound evaluation process aligned with desired long-term outcomes.

Unfortunately, these data are not always available, and if they are, many decision makers do not know they exist, or do not have access to them, or do not know how to interpret and use them to make sound decisions that lead to improved program and organizational performance. In fact, Lee Cronbach (1980) and others have argued that decisions often emerge rather than being logically and methodically made.

Effective leaders are capable of making sound decisions based on sound data, and evaluators can do much to influence the leadership decision-making process. Evaluation can provide a systematic framework that aligns stakeholders, evaluation purposes, desired results and consequences, and all evaluation activities, so that the evaluation product is a responsive and clear recipe for improving performance. This in essence allows the decision-making process to become clearer and more straightforward. Evaluation is the mechanism that provides decision makers with feedback, whether through interim reports and meetings or a final report and debriefing.

A BRIEF OVERVIEW OF EVALUATION HISTORY

Michael Scriven (1991) describes evaluation as a practice that dates back to samurai sword evaluation. Another type of evaluation was in evidence as early as 2000 B.C.: Chinese officials held civil service examinations to measure the ability of individuals applying for government positions. And Socrates included verbal evaluations as part of his instructional approach (Fitzpatrick, Sanders, & Worthen, 2004).

In response to dissatisfaction with educational and social programs, a more formal educational evaluation can be traced back to Great Britain during the 1800s, when royal commissions were sent by the government to hear testimony from the various institutions. In the 1930s, Ralph Tyler issued a call to measure goal attainment with standardized criteria (Fitzpatrick et al., 2004). And during the 1960s, Scriven and Cronbach introduced formative (used to guide developmental activities) and summative (used to determine the overall value of a program or solution) evaluation, and Stufflebeam stressed outcomes (program results) over process (program activities and resources) (Liston, 1999).

In 1963, Cronbach published an important work, “Course Improvement Through Evaluation,” challenging educators to measure real learning rather than the passive mastery of facts. Moreover, he proposed the use of qualitative instruments, such as interviews and observations, to study outcomes. In the latter part of the 1960s, well-known evaluation figures such as Edward Suchman, Michael Scriven, Carol Weiss, Blaine Worthen, and James Sanders wrote the earliest texts on program evaluation.

In 1971, Daniel Stufflebeam proposed the CIPP model of evaluation, which he said would be more responsive to the needs of decision makers than earlier approaches to evaluation were. In that same year, Malcolm Provus proposed the discrepancy model of evaluation. In 1972, Scriven proposed goal-free evaluation in an effort to encourage evaluators to find unintended consequences. In 1975, Robert Stake provided responsive evaluation. In 1981, Egon Guba and Yvonna Lincoln proposed naturalistic evaluation on the basis of Stake’s work, feeding the debate between qualitative and quantitative methods (Fitzpatrick et al., 2004).

All of this was occurring in the context of a movement to account for the billions of dollars the U.S. government was spending on social, health, and educational programs (Fitzpatrick et al., 2004; Patton, 1997). In order to address a demand for accountability, those responsible for programs soon began to ask evaluators for advice on program improvement. Thus, the initial purpose of program evaluation was to judge the worthiness of programs for continued funding.

When Sputnik became the catalyst for improving the U.S. position in education, which was lagging compared to other countries, educational entities in particular began to commission evaluations, partly in order to document their achievements. The need for evaluators soon grew, and government responded by funding university programs in educational research and evaluation. In the 1970s and 1980s, evaluation grew as a field, with its applications expanding beyond government and educational settings to management and other areas. Evaluations are now conducted in many different settings using a variety of perspectives and methods.

EVALUATION: PURPOSE AND DEFINITION

While some rightly say that the fundamental purpose of evaluation is the determination of the worth or merit of a program or solution (Scriven, 1967), the ultimate purpose, and value, of determining this worth is in providing the information for making data-driven decisions that lead to improved performance of programs and organizations (Guerra-López, 2007a). The notion that evaluation’s most important purpose is not to prove but to improve was originally put forward by Egon Guba when he served on the Phi Delta Kappa National Study Committee on Evaluation around 1971 (Stufflebeam, 2003). This should be the foundation for all evaluation efforts, now and in the future. Every component of an evaluation must be aligned with the organization’s objectives and expectations and the decisions that will have to be made as a result of the evaluation findings. These decisions are essentially concerned with how to improve performance at all levels of the organization: internal deliverables, organizational gains, and public impact. At its core, evaluation is a simple concept:

It compares results with expectations.

It finds drivers and barriers to expected performance.

It produces action plans for improving the programs and solutions being evaluated so that expected performance is achieved or maintained and organizational objectives and contributions can be realized (Guerra-López, 2007a).

Some approaches to evaluation do not focus on predetermined results or objectives, but the approach taken in this book is based on the premise of performance improvement. The underlying assumption is that organizations, whether they fully articulate this or not, expect specific results and contributions from programs and other solutions. As discussed in later chapters, this does not prevent the evaluator or performance improvement professional from employing means to help identify unanticipated results and consequences. The worth or merit of programs and solutions is then determined by whether they delivered the desired results, whether these results are worth having in the first place, and whether the benefits of these results outweigh their costs and unintended consequences.

An evaluation that asks and answers the right questions can be used not only to determine results but also to understand those results and to modify the evaluation so that it can better meet the intended objectives within the required criteria. This is useful not only to identify what went wrong or what could be better but also to identify what should be maintained. Through appreciative inquiry (Cooperrider & Srivastva, 1987), evaluation can help organizations identify what is going right. Appreciative inquiry is a process that searches for the best in organizations in order to find opportunities for performance improvement. Here too the efforts are but a means to the end of improving performance. Although the intentions of most evaluators are just that, the language and approach used are charged with assumptions that things are going wrong. For instance, the term problem solving implies from the start that something is wrong. Even if this assumption is not explicit in the general evaluation questions, it makes its way into data collection efforts. Naturally the parameters of what is asked will shape the information evaluators get back and, in turn, their findings and conclusions. If we ask what is wrong, the respondents will tell us. If we ask what went right, again they will tell us. The key point is that evaluation should be as unbiased as possible. Evaluators should ask and answer the right questions, so that the data they get are indeed representative of reality.

In specific terms, before evaluators start to plan, and certainly before they collect data, they must determine why they are conducting an evaluation. Is this their initiative, or were they directed to do this work? What is the motivation for the study? What are they looking to accomplish and contribute as a result of this evaluation? Here are some general reasons for conducting an evaluation:

To see if a solution to a problem is working, that is, delivering valued ends

To provide feedback as part of a continual monitoring, revision, and improvement process

To provide feedback for future funding of initiatives

To confirm compliance with a mandate

To satisfy legal requirements

To determine if value was added for all stakeholders

To hold power over resources

To justify decisions that have already been made

Although the last two in this list are particularly driven by political agendas, in reality most reasons can be politicized; thus, it takes an insightful evaluator to recognize the feasibility of conducting an honest evaluation. An experienced evaluator will recognize, most of the time, whether evaluation stakeholders are truly interested in using evaluation findings to improve performance or are more concerned with advancing their political interests. With careful attention to detailed planning, either goal can be made to fit a data-driven and results-oriented action approach to evaluation. But if taken too narrowly—in isolation and without proper context—each has its own narrow set of problems, blind spots, and special data generation and collection issues. Perception of the purpose of the evaluation can shape and limit the data that are observed (or not observed), collected (or not collected), and interpreted (or ignored). Thus, evaluators and stakeholders must begin the planning process with a clear articulation of what decisions must be made with the results of their findings, decisions that are linked to the overall purpose for conducting the evaluation.

PERFORMANCE IMPROVEMENT: A CONCEPTUAL FRAMEWORK

The field of performance improvement is one of continuous transition and development. It has evolved through the experience, reflection, and conceptualization of professional practitioners seeking to improve human performance in the workplace. Its immediate roots stem from instructional design and programmed instruction. Most fundamentally, it stems from B. F. Skinner and his colleagues, whose work centered on the behavior of individuals and their environment (Pershing, 2006).

The outgrowth of performance improvement (also called human performance technology) from programmed instruction and instructional systems design was illustrated in part by Thomas Gilbert’s behavioral engineering model, which presented various categories of factors that bear on human performance: clear performance expectations, feedback, incentives, instruments, knowledge, capabilities, and internal motives, for example. This landmark model was published in Gilbert’s 1978 book, Human Competence: Engineering Worthy Performance, and was based in large part on the work Gilbert conducted with Geary Rummler and Dale Brethower at the time. Pershing (2006) declares that Joe Harless’s 1970 book, An Ounce of Analysis Is Worth a Pound of Objectives, also had a significant impact on the field and was well complemented by Gilbert’s work. Together these works served as the basis for many researchers who have contributed to and continue to help develop the performance improvement field.

Currently the International Society for Performance Improvement, the leading professional association in the field, defines performance improvement as a systematic approach to improving productivity and competence, using a set of methods and procedures—and a strategy for solving problems—for realizing opportunities related to the performance of people. More specifically, it is a process of selection, analysis, design, development, implementation, and evaluation of programs to most cost-effectively influence human behavior and accomplishment. This series of steps, commonly known as the ADDIE model, is the basic model from which many proposed performance improvement evaluation models stem. Pershing (2006) summarized performance improvement as a systematic combination of three fundamental processes: performance analysis (or needs assessment), cause analysis (the process that identifies the root causes of gaps in performance), and intervention selection (selecting appropriate solutions based on the root causes of the performance gaps). These three processes can be applied to individuals, small groups, and large organizations. The proposition that evaluation of such interventions should also be at the core of these fundamental processes is presented in the final chapter of this book.

This is the context in which evaluation is seen and described in this book—not as an isolated process but rather as one of a series of processes and procedures that, when well aligned, can ensure that programs and organizations efficiently and effectively deliver valuable results.

MAKING EVALUATION HAPPEN: ENSURING STAKEHOLDERS’ BUY-IN

One of the most important elements of any evaluation is its stakeholders. Before we define the stakeholders, it is worthwhile to define the term stake. A stake is essentially a claim, an interest, or a share in some endeavor and how that claim or interest might be affected by anything that is used, done, produced, or delivered. The traditional view of a stake used to be limited to the financial realm (for example, stockholders), but in fact a claim or interest can be financial, legal, or moral (Carroll, 2000). Thus, a stakeholder is any individual or group with a stake in an endeavor and can either affect or be affected by the decisions and actions of the organization.

Stakeholders can be broadly categorized as internal (owners, employees, and management) and external (customers, customers’ customers, the community, suppliers, competitors, the government, and the media, to name a few), and both categories can then be subdivided into various groups.

Not every individual within each stakeholder group has to participate directly in an evaluation; what is important is that those who participate are seen as representative by their group members. The greater the sense of stakeholder involvement and influence there is, the less likely it is that the evaluator will encounter resistance to the evaluation process and the findings.

While ideally evaluators will select stakeholders who will help define useful evaluation expectations, questions, and criteria, in fact, they realistically will be faced with stakeholders who have their own special interests or represent a powerful lobby. Although it is not particularly unusual for human beings to have their own special interests, evaluators should neutralize as much as possible the risk that the evaluation will become a manipulation tool for the special interests of one—or some—at the expense of others.

A vital challenge in working with stakeholders to help all be successful is to keep them focused on results and consequences rather than on politics of means. Single-issue politics from both within and outside organizations have a tremendous impact on defining objectives and selecting means. It is essential that evaluators learn enough about the specific political climate of a given evaluation to understand how it will affect the evaluation and the implementation of its recommendations. If evaluation recommendations are not implemented or are implemented improperly, performance probably will not improve, and the evaluation may have been conducted in vain.

THE EVALUATOR: A JOB OR A ROLE?

The term evaluator describes not only one profession or occupation, but also a given role at a particular time. Individuals conducting evaluation often wear many hats. They may be internal employees, members of the management team, faculty members, or consultants who have acquired interest and expertise in measurement and evaluation through education, training, or experience. In some cases, individuals arrive at this point by default and face an unexpected request to conduct an evaluation. They could be trainers who are charged with demonstrating the value of their training programs and departments. They may even be individuals who because of their status as a subject matter expert in some solution or program are also faced with demonstrating the value of their efforts.

Their common function is, or should be, an interim goal to document the results and impact achieved by a given solution: a program, a project, a tool, or the use of a resource. The final goal should be to use this information to make sound decisions and help the organization take appropriate action to improve performance at all levels.

Evaluators should be competent in some basic areas. Sanders (1979) proposed that at a minimum, evaluators should be able to

Accurately describe the object (the evaluand) and context of that which is being evaluated

Conceptualize the purpose and framework of the evaluation

Derive useful evaluation questions, data requirements, and appropriate data sources

Select the means for collecting and analyzing data

Determine the value of the evaluand

Effectively communicate results and recommendations to the audience

Manage the evaluation project

Maintain ethical standards

Adjust to external factors influencing the evaluation

Evaluate the evaluation

THE RELATIONSHIP TO OTHER INVESTIGATIVE PROCESSES

The results and consequences we want to accomplish are the primary drivers for deriving the useful questions of an organizational study. Another driver is the types of decisions that have to be made; in large part, they will determine what data have to be gathered and for what purpose. For instance, if decisions have to be made about what programs, interventions, and solutions should be continued, revised, or discontinued, then the data collection approach may take an evaluative perspective. That is, the data collected will be used to compare predetermined objectives with what was actually achieved. If the need is to make decisions about what results the organization should be targeting and, in turn, what types of programs, interventions, and solutions will help it get there, the data collection approach will take on a needs assessment perspective. Notice that in both cases, results—and gaps in results—are the primary drivers.

Table 1.1 illustrates some sample questions from both perspectives that could apply to any organization in any sector. Both approaches to data collection should be systematic and designed to answer specific questions that can be used to improve performance.

Assessors and evaluators may share data collection techniques, but the types of questions they seek to answer differ. In this sense, the roles of assessor and evaluator differ in purpose or function rather than in importance and methods.

Needs assessors help create the future by providing hard and soft data for identification of performance-based, vision-aligned missions and building-block objectives, as well as the gaps between current and desired results. In addition, they help identify the best solutions for closing these gaps and thereby ultimately reaching the organizational vision. It should be noted that asking people what they need is not a needs assessment; this simply creates a “wants list” or “wish list” without rigorous applicability (Kaufman, 2000). Evaluators help to determine whether they are heading toward reaching the future they set out to create during the needs assessment process. One of the primary ways they do this is by determining the effectiveness and efficiency of the implemented programs and solutions, as well as the causal factors associated with any gaps between expected and accomplished results. Measurably improving organizational and individual performance depends heavily on these two roles and processes.

TABLE 1.1. Unique Perspectives of Needs Assessment and Evaluation

Source: Guerra (2003b).

Needs Assessment Questions

Evaluation Questions

What value-added results should we be targeting?

How much closer did we get to reaching our vision and mission?

What value-added results are we now getting?

Did we add to or subtract value from our external clients and our shared society?

Who or what is the primary client of the results and their consequences?

Which objectives in our mission did we achieve?

How do we get from current results and consequences to desired ones?

How are we doing in comparison to last quarter? Last year?

What interim results must be accomplished and when?

Which internal results targets were reached? Not reached?

What are our options?

Which implemented programs, projects, or solutions were effective?

What are the most effective and efficient ways for reaching our desired or required results?

How efficient are these implemented programs, projects, or solutions?

What will it cost us to reach those results?

In which of these should we continue to invest?

What will it cost us to ignore those results?

What results do we have to justify our continued programs?

How far do we have to go to reach those results?

What should we discontinue?

Which results take priority over others?

Which projects, programs, or solutions could be successful with some modifications? Is it worth it?

Where do we have the most—and least—leverage?

Did we add or subtract value from our internal clients and employees?

Although both assessors and evaluators collect data with regard to the results of a process or activity, evaluators collect data to determine whether results match the results expected from solutions that have already been implemented—for example, new programs, new technologies, new processes, or any other means selected to help achieve objectives. Assessors, in contrast, seek to anticipate the expected return on investment of potential interventions before they are implemented by collecting data about both current results (what is) and potential results (what should be). With these data in hand, decision makers are able to choose among competing alternatives.

So how does scientific research come into the picture? Before answering this question, let us first explore the meaning of science. Science is based on a series of assumptions about the world—assumptions that can be true today but false tomorrow. Scientists are always testing these assumptions, ready to change them when the findings support such a change. To this end, scientists collect data about reality and consult with other sources to ensure the reliability of the data. Results are considered basic data, later subject to repeatable observations in order to confirm findings and scientific reports. Thus, we want to make decisions and take action based on what is currently known through scientific inquiry.

Research is essentially another systematic process of inquiry, with the purpose of finding, interpreting, and updating facts, events, behavior, and theories. In this sense, research skills are a basic requirement in today’s world and can be applied in just about any context, whether needs assessment, evaluation, or scientific inquiry. In fact, the heart of the data collection plan is very much the same for all of these. Following are the common elements among these three inquiry processes. These are stated generically but can be made specific to investigative contexts.

Important decisions that must be taken by stakeholders are identified. They lead to element 2:

Guiding questions, purposes, or hypotheses that the inquiry process must answer or test, which are related to element 3:

Key variables or results that are the central focus of the questions or hypotheses.

When results are not directly observable, measurable and observable indicators must be identified.

These indicators become the data to collect and inform the data source used.

The types of data sought inform the data collection tools appropriate for use.

The types of data sought determine the types of data analysis tools qualified to summarize and test these data.

The process concludes with findings, interpretations, and reporting that are supported by the data collected.

The key in the methodology is the alignment of all the elements: from adding value to all internal and external stakeholders to linking with resources and methods to deliver worthy results.

Certainly in much basic research, generalizability of findings is critical, and thus there is a strong push for controlled environments and the isolation of effects. However, the complexity of real-world evaluation does not easily lend itself to the control of variables. Perhaps evaluation overlaps more closely with applied research, where the goal of the study is the solution of real organizational problems rather than the advancement of the theoretical body of knowledge. However, both evaluation and applied research benefit from the knowledge obtained through basic research. Table 1.2 provides a side-by-side comparison of basic research, applied research, and evaluation. Although the dimensions of each process are described generally, modification of any of them is possible, thereby blurring these distinctions. For example, an evaluation report could be used as part of the literature review of a basic research study, thereby influencing what research questions are studied and how.

WHEN DOES EVALUATION OCCUR?

Having measurable performance objectives in the correct format does not guarantee that the objectives address the right things. Decades ago, people realized that focusing only on objectives could cause an evaluator to miss important data on process and environment. In the 1960s, the realization that evaluation could play a role in the development of educational programs to adjust content and process along the way to the final results gave rise to a famous distinction when Scriven (1967) introduced the terms formative and summative as well as goal-free evaluation. Since then, evaluators have had a term for the type of evaluation activity used to guide developmental activity in programs (formative) and another term for evaluation that is used to comment on overall final value (summative).

TABLE 1.2. Dimensions of Investigative Processes

Dimensions

Basic Research

Applied Research

Evaluation

Goal

Advancement of knowledge and the theoretical understanding of relevant variables

Application of scientific knowledge to the solution of a specific, defined problem

Identification of relevant information to improve specific objects and organizations

Approach

Exploratory and often driven by the researcher’s curiosity and interests

Generally descriptive rather than exploratory and conducted by educational or other institutions

Generally guided by the need to make important organizational decisions

Use

Conducted without a currently practical end in mind

Done to solve specific, practical questions

Done to solve specific, practical questions

Basis

As its name suggests, it could provide the basis for further, often more applied research

Often done on the basis of basic research or on previous valid research findings

Should be done on the basis for a needs assessment, while also considering past basic and applied research findings

While determining the overall value added—or potentially subtracted—by programs and organizations should be one of its key functions, formative evaluation is also quite important to the overall contributions of programs and other solutions. Moreover, formative evaluation can be designed in such a way that it continuously monitors the alignment of a program with its subsystems and suprasystems to facilitate the achievement of its ultimate value.

Formative evaluation should start at the same time as the identification, design, development, and implementation of the program or solution of interest. Some general questions include the following ones:

Are we targeting the right objectives?

Are they based on assessed needs (gaps in results)?

Are the criteria measurable and soundly based?

Are we using the right criteria to judge the effectiveness and efficiency of our solution?

Did we identify the appropriate program or solution?

Did we base our selection on an analysis of alternatives?

Did we weigh the pros and cons?

Did we weigh the costs and consequences?

Is our design useful and relevant?

Is it aligned with the front-end analysis findings (causes for gaps in results)?

Is it appropriate for the ends we want to reach?

Is the development of the program or solution aligned with its intended design?

Is our pilot designed to capture the right data required for improvements?

Does our prototype meet the requirements of our users?

Is the program or solution being implemented appropriately?

Incidentally, implementation questions may also be appropriate during summative evaluation approach, where we not only look at the results and consequences but also at the factors that may have led to those results and consequences. Obviously, if the intent is to ensure the effectiveness of the solution, we want to know if we are implementing it effectively before and during implementation, not just after the fact:

Were those affected by the program or solution included in the problem identification, solution selection, and every other stage?

Were fears and unfounded ideas about the implications of the program or solution confronted, clarified, or disproved, as appropriate?

Is the program or solution being implemented according to initial plans?

Is the implementation of the program or solution flexible and responsive to the current situation (for example, challenges not previously foreseen)?

Evaluating each stage and using evaluation data to improve it will allow evaluators and stakeholders to stay on track in order to reach the short- and long-term objectives of the program or solution.

GENERAL EVALUATION ORIENTATIONS

Two common distinctions in evaluation are formative and summative. Formative evaluation typically occurs during the developmental stage of a program and can be used to improve the program before it is formally launched. The formative approach can also be used to improve all stages of performance improvement, from assessment to implementation, and the evaluation itself.

Summative evaluation occurs after the implementation of a program or solution and usually requires some appropriate amount of time to have transpired so that the object of evaluation has the opportunity to have the full impact required on performance at various levels of the organization. It is worth noting that summative evaluation can also be used to improve programs and solutions. Stufflebeam and Webster (1980) hold that an objectives-based view of program evaluation is the most common type of evaluation. Once the results that have been accomplished have been determined, the evaluator is well advised to identify causal factors contributing to those results. These data should provide insights as to what the drivers and barriers to the success of the program are, thereby providing the basis for recommendations for improving performance.

Another distinction often made among evaluation orientations is that of process evaluation versus results evaluation. These terms are used to describe the same processes that formative and summative approaches, respectively, take. Depending on how these are interpreted and implemented, they can also differ somewhat from their counterparts described above. For instance, the Canadian Evaluation Society uses the term process evaluation (also referred to as efficiency evaluation) to describe the monitoring of the implementation of programs. Obviously, there should be a well-planned logic model with specified results and processes, but modifications are made if a discrepancy between the program design and the actual implementation is found. For example, one might want to determine if the program is being delivered as intended, if it is being delivered to the targeted clients or participants, or if it is being delivered with the intended effort or in the intended quantity.

Process evaluation is critical in helping evaluators address the variations in program delivery. The greater the variation in program delivery, the greater the requirement is for useful data gathered through a process evaluation approach. For instance, there may be differences in staff, clients, environments, or time, to name a few variables.

Stufflebeam and Webster (1980) have argued that objectives-based program evaluation is the most prevalent type used in the name of educational evaluation. Scriven (1972) proposed goal-free evaluation to urge evaluators to also examine the process and context of the program in order to find unintended outcomes.

Results evaluation, also referred to as effectiveness evaluation, is used to determine whether the immediate outcomes of a program meet predetermined objectives specified by program planners; impact evaluation tends to refer to an evaluation that looks at not only immediate outcomes but also the long-term outcomes of a program and their interdependency. A results evaluation approach is important because it allows us to ensure and document that we are on track by gathering data that show quality accomplishments. It also helps us stay accountable and our programs to stay cost-effective by making program benefits and costs tangible.

Other evaluation approaches are associated with effectiveness evaluation. Cost-benefit evaluation is the translation of costs and benefits into monetary terms, which is used to compare the relative net benefits of doing one thing versus another. However, monetary terms are not always applicable, and they are seldom sufficient to appreciate costs and benefits. Cost-effectiveness evaluation considers alternative forms of program delivery according to both their costs and their effects with regard to producing some result or set of results. Of course, a stable measure of result should be defined. The least costly program is not necessarily the best one. And in the context of technology solutions, an additional orientation to evaluation is usability testing, which focuses on whether people are using the product and how well they are using it to meet required objectives.

CHALLENGES THAT EVALUATORS FACE

A common excuse for avoiding evaluation is insufficient resources to conduct one. In fact, it can often take more resources to maintain programs blindly and indefinitely than it does to conduct a rigorous and focused evaluation. The decision about whether to conduct an evaluation in the first place requires thinking about not only its cost but also the benefits it can render. Both cost and benefit categories contain monetary and nonmonetary items, and they should be honestly and carefully considered before making decisions about conducting or not conducting an evaluation.

One of the most serious challenges faced by evaluators—and probably researchers in general—is getting people to use the findings and recommendations. One study (Henderson, Davies, & Willis, 2006) cited lack of key stakeholder and consumer involvement as a factor that reflects the adoption of evidence for changes in practice. Lack of leadership support was also identified as a factor. When these two factors are combined, there is no support at all for creating and promoting change. The default stance is maintaining the status quo, even if “changing” is the logical proposition.

Limited expertise can also become a barrier. When no one, or few people, in the organization understands the benefits of evaluation or the process itself, finding a champion is difficult. Even if evaluation efforts are undertaken, these are frequently undermined by poor evaluation direction, design, findings, and recommendations. The consequences of conducting a poor evaluation can be worse than not doing one at all. Once the expectation for improvements has been created, failure to see such improvements can severely harm the morale and trust of organizational members.

Fear and cynicism are supported not only by poor evaluations but also by past efforts to use evaluation as a means of control and intimidation. Findings—or even the mere “threat” of evaluation—have been used to point the finger at the inadequacies of programs, organizations, and human competence. In fact, even when evaluation has provided useful information for improving programs, it is not uncommon for people to disbelieve the evidence.

Another challenge is the low awareness of the utility and benefits of evaluation. People are not often short of ideas about what to do; the challenge begins with helping them articulate how they will know whether the things they have done or implemented have delivered valuable results. Our culture is one of action, so there is often a false sense of accomplishment in just doing something. Verification and documentation of desired results are often neglected and not viewed as an integral part of what we do unless there is a funding source or a mandate to do so.

Perhaps the biggest challenge—and the most important one—is helping those around us understand that every organization, program, department, function, employee, and resource must be ultimately aligned with positive results and consequences for society (Kaufman, 1992, 2000, 2006a). If what is being used, done, produced, and delivered is not adding benefit to society, it is probably doing quite the contrary. Evaluation and needs assessment are uniquely positioned tools for helping stakeholders make sound decisions about what direction to set, how best to get there, how close they have come to getting there, and what improvements must take place in order to ensure the attainment of organizational and societal ends. In fact, even the business community is embracing this reality through movements like corporate social responsibility. Milton Friedman’s old paradigm about the “business of business is business” is being disputed even by the heads of top management consulting firms like McKinsey, who argues that “social issues are not so much tangential to the business of business as fundamental to it” (Davis, 2005, p. 1).

There are enormous societal needs to fill in all areas: education, physical and mental health, economic development, crime, and discrimination, to name a few. Kaufman has set out his ideal vision of “the world we want to create for tomorrow’s child,” which identifies the basic indicators of societal ends, and thus of needs, and has been used as the basis for strategic planning, needs assessment, and evaluation. Kaufman (2000) defines needs as gaps between what should be accomplished and what is currently accomplished:

There will be no losses of life nor elimination or reduction of levels of survival, self-sufficiency, or quality of life from any source including (but not limited to) the following:

War, riot, terrorism, or unlawful civil unrest

Unintended human-caused changes to the environment including permanent destruction of the environment and/or rendering it non-renewable

Murder, rape, or crimes of violence, robbery, or destruction to property

Substance abuse

Permanent or continuing disabilities

Disease

Starvation and/or malnutrition

Destructive behavior (including child, partner, spouse, self, elder, others)

Accidents, including transportation, home, and business/workplace

Discrimination based on irrelevant variables including color, race, age, creed, gender, religion, wealth, national origin, or location

Consequences: Poverty will not exist, and every woman and man will earn at least as much as it costs them to live unless they are progressing toward being self-sufficient and self-reliant. No adult will be under the care, custody or control of another person, agency, or substance: all adult citizens will be self-sufficient and self-reliant as minimally indicated by their consumption being equal to or less than their production (p. 95).

Societal ends are not defined by a single organization, and it is not expected that any one organization will accomplish them on its own. These strategic-level objectives represent the shared ambitions of the organizations, individuals, and other partners that stand for our shared communities and society.

ENSURING COMMITMENT

It is vital to get the ownership of evaluation and performance by those who define, deliver, and receive organizational objectives, products, and services. Evaluators and stakeholders must define the required contributions each will make so that they can create a solid partnership for success. Trust, understanding, and agreement on a common destination—results to be achieved—are all key to a successful enterprise. Without the commitment and participation of all of the stakeholders, the success of the evaluation will be less than it could be.