Alternative’s to Kirkpatrick: Brinkerhoff’s Success Case Method

As learning experience designers (aka instructional designers aka performance improvement consultants), we constantly try to improve upon our own processes. Are we meeting the needs of our learners? Are we creating learning opportunities that impact organizational performance?

When it comes to learning solutions leveraged in organizations, Kirkpatrick’s evaluation model is often looked to as the assessment method, despite its focus on training events rather than on learning processes.  In my previous post, I  described an alternative approach to Kirkpatrick’s model—Kaufman’s 5 levels of evaluation.  In this post, I’ll take a look at a more flexible model, Brinkerhoff’s Success Case Method.

Brinkerhoff: A focus on systems

Brinkerhoff (2005) squarely addresses one of the critiques of Kirkpatrick’s model: “Performance results can’t be achieved by training alone; therefore training should not be the object of evaluation” (p. 87). According to Brinkerhoff, this is like saying that the success of a marriage depends on the quality of the wedding ceremony. Multiple variables contribute to the impact of a learning opportunity (be it training, performance support, or another solution) and multiple stakeholders own these results—from senior executives to managers to HR professionals/L&D team members to  employees.

Given that the factors that influence performance operate at the level of systems, Brinkerhoff urges that the proper focus of evaluation is also at the system level. Evaluation should address the following questions:

• How well is an organization using learning to improve performance?
• What organizational processes/resources are in place to support performance improvement? What needs to be improved?
• What organizational barriers stand in the way of performance improvement?

(Brinkerhoff, p. 88)

The Success Case Method: Evidence- and narrative-based

The Success Case Method (SCM) “combines the ancient craft of storytelling with more current evaluation approaches of naturalistic inquiry and case study” ( p. 91). Essentially, when you apply SCM, you ask:

• What groups/individuals have been successful in applying a learning opportunity to achieve a business result? Why have they been successful?
• What groups have been unsuccessful? Why have they been unsuccessful?

There are a number of steps involved in SCM.

1. Develop an impact model: Identify the goals of the learning opportunity and determine how these goals are connected to business needs. The impact model defines what success should look like.
2. Survey participants to identify best cases and worst cases. (For example, a survey question might ask: How have you applied what you learned to achieve a business result?)
3. Obtain corroborating evidence that would “stand up in court” (p. 91) (e.g., using interviews, document reviews or other methods).
4. Analyze the data.
5. Communicate findings: Share what successes have occurred and what organizational resources have supported these successes. As important, share examples of non-successes. What barriers kept people  from applying what they learned?

A better model?

There are some appealing aspects to Brinkerhoff’s SCM.

• It’s do-able: You do need to collect data but you can do this in a cost-effective manner.
• Because SCM utilizes purposeful, not random sampling, it collects information that would be missed if you only looked at averages or central tendencies.
• SCM allows for the discovery of emergent success factors: Although you do create an impact model in advance, your surveys and interviews may uncover additional, unexpected  business results.
• The focus is on systems and leveraging learning resources into continuously improved performance  (p. 89).
• Outputs include relatable stories you can share.

Although there are the typical caveats associated with surveys (potential biases and halo effects), these can be mitigated through the step of collecting corroborating evidence. I think that the focus on successes and case stories  is extremely attractive because it provides a way to involve stakeholders in a meaningful dialog about continued process improvement.

Reference

Brinkerhoff, R. O. (2005).  The Success Case Method: A strategic evaluation approach to increasing the value and effect of training. Advances in Developing Human Resources, 7(1),  86-101. Retrieved from http://aetcnec.ucsf.edu/evaluation/Brinkerhoff.impactassess1.pdf