Alternatives to Kirkpatrick: Kaufman’s 5 Levels of Evaluation

Meaningful instructional design requires meaningful evaluation. However, evaluation, like organizational development itself, requires buy-in at many levels. This buy-in’s necessary…

  •  To identify meaningful metrics
  •  To collect data
  • To react to data, making appropriate  improvements
  • To undertake change management necessary for these improvements

If you’ve been in this field for any length of time, you’ve probably come across Kirkpatrick’s four levels of evaluation (reaction, learning, performance, & results).  Kirkpatrick’s approach has come under fire for a number of reasons (e.g., emphasis on training events, implied linearity and causality, and more) and many articles have provided thoughtful critiques.

However, in this series of posts, I haven’t come to bury Kirkpatrick’s approach or to praise it. Instead, I’m going to discuss some alternatives.

One alternative: Kirkpatrick Plus

Articulated by Kaufman, Keller, and Watkins (“Kaufman”) (1995), this evaluation framework connects performance to expectations. Kaufman proposes 5 levels of evaluation

Level 1: Resources and processes
Level 1 is actually divided into two levels, 1a and 1b.

  • Level 1a  focuses the evaluation lens on inputs, e.g., such as the availability and quality of materials needed to support a learning effort.
  • Level 1b considers processes. What’s their quality? Are they efficient? Are learners satisfied with them?

Compared to Kirkpatrick’s Level 1 (Reaction), Kaufman’s Level 1 focuses not only on learner satisfaction, but on the organizational factors that can impact learner satisfaction.

Level 2: Acquisition
This level is focused on individual and small group payoffs—what Kaufman calls  “micro” benefits.  Are the objectives or desired outcomes of the learning intervention met? It’s pretty analogous to Kirkpatrick’s Level 2 evaluation (Learning), but Kaufman notes that the learning intervention may not necessarily be training.

Level 3: Application
This is still a micro analysis, examining individual and small group impacts. The relevant inquiry here is whether newly acquired knowledge and skills are being applied on the job. Level 3 also is quite similar to Kirkpatrick’s Level 3 (Behavior/Performance).

Level 4: Organizational payoffs
Here, the analysis examines macro benefits. What are the benefits from an organizational standpoint? Level 4 is analogous to Kirkpatrick’s Level 4 (Results).

Level 5: Societal contributions
Kaufman considers this a mega analysis. How is the organization contributing to its clients and society?  Is it responsive to client/societal needs?

Issues of health, continued profits, pollution, safety, and well-being are central [in this level]. The basis for mega-level concerns is an ideal vision, which is a measurable statement of the kind of world required for the health, safety, and well being of tomorrow’s children.

Level 5 has no analog in Kirkpatrick’s Evaluation Model.

A better model?

The “Kirkpatrick Plus” framework doesn’t stray that far from Kirkpatrick’s Evaluation Model and so can be subject to many of the same criticisms.  Notably,  while measuring organizational payoff’s  an important part of a meaningful evaluation, teasing apart the effects of a learning intervention from all the other variables that impact ROI is notoriously difficult. And if you think measuring organizational payoff is challenging, imagine how hard it is to measure societal impact. (This isn’t to say this evaluation aspiration isn’t a worthy one.)

I do think making the organization’s efforts part of the evaluation process (as in Kaufman’s Level 1) is an important step in the right direction. The organization’s commitment to success (e.g., by providing necessary resources, processes, and other supports) should be subject to as much scrutiny as the learner’s performance.

Still shopping for a better model? Stay tuned for the next post.

Reference

Kaufman, R., Keller, J., & Watkins, R. (1995). What works and what doesn’t: Evaluation beyond Kirkpatrick. Performance and Instruction, 35(2): 8-12. Retrieved from http://home.gwu.edu/~rwatkins/articles/whatwork.PDF

6 responses to “Alternatives to Kirkpatrick: Kaufman’s 5 Levels of Evaluation

  1. Pingback: Alternative’s to Kirkpatrick: Brinkerhoff’s Success Case Method | Instructional Design Fusions

  2. Nice post. I’m not sure if I understand correctly the difference in the first level. Does this model considers also processes and resources that were available to learner before and during the training (or only during)?

  3. Hi Bartlomiej,

    Kirkpatrick’s level 1 is only learner-focused (gauging learner satisfaction) and doesn’t consider the responsibility of an organization to support learning. You bring up a good point though about considering the timing of organizational inputs. Kaufman’s model does consider processes and resources available to learners before and during learning not just during training. I think that’s a good aspect of the model–asking what context the learning is situated in.

  4. Pingback: What’s New In Enterprise Learning? « « DediCommDediComm

  5. I don’t know if organizations will be willing to dump the traditional Kirkpatrick model so this alternative might be an easier way to slowly incorporate new ideas in the area of evaluation. Thanks for the ideas you have shared.

  6. Pingback: What's New In Enterprise Learning? | Social Learning Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s