Measuring Training Effectiveness - based on Kirkpatrick's four level evaluation model
Return on Investment is the most talked about aspect while planning a learning intervention. Most learning interventions are assumed not effective because their effectiveness cannot be measured or quantified. This in turn is because either there is not sufficient pre-work done or because there is no clarity on the objectives of the intervention.
Kirkpatrick is a well known name in the world of training and development, especially popular for his contributions towards evaluating training programs. He first introduced the Four Level evaluation model in 1959, which has since been updated a number of times, the last being in 1994. This model is simple to implement and helps measure the effectiveness of a learning intervention on various levels. However, the implementation of this model requires maturity and commitment on part of the Organisation as well as the Learning and Development function. This can be time-consuming and requires involvement from various stakeholders, the subject matter experts and the Trainer. Implementing this model also means, the entire learning process will have to be structured, organised and planned much in advance so that the intent of implementing this model is achieved.
Four Level Evaluation Model -
This model has been designed to help evaluate the effectiveness of any learning intervention on four levels based on its -
- Reaction
- Learning
- Behaviour
- Results
According to this, the participants' Reaction to the learning intervention is as important as the actual Learning, followed by the change in the Behaviour of the participants and the Resultsachieved thereby.
In order to implement this model successfully the pre-work with respect to and Learning Intervention is absolutely vital. Here's the various stages that need to be focused upon -
- Learning Objectives
Why is this learning intervention required and what are we trying to achieve by way of it ? Are we trying to solve a problem, learn a new skill that will be used in an upcoming project, or are we expecting the participants to take up additional responsibility, or do we want the participants who are currently at Level 2 of a certain competency to reach Level 3 ? Answering these and similar questions is essential. Sometimes, we assume that a particular learning intervention will solve a particular problem for us, but is this the most suitable solution to the problem at hand is a must ask question. If you know what you want to achieve, you will know when it is achieved too, hence there comes the measuring mechanism.
- Mechanism to gauge participants response
Once you know what you are set out to achieve, the next step is to zero down on certain indicators to let you know that you are half way there. A Participants Feedback form to be filled after the learning intervention has been successfully implemented and attended, is one such indicator. An ideal form should cover all aspects of the Learning intervention like the Trainers ability and knowledge, the quality of the training delivered, whether the participants thought it was useful and whether their expectations were met, the training venue, training material etc. Based on the ratings received, it is clear whether things have gone right or not.
- Mechanism to assess participants existing knowledge or level of expertise on the subject before they attend the training
This is a vital step in the process of evaluating training effectiveness. this is also the most neglected aspect due to various reasons, time constraint, no commitment or no interest. Measuring learning post training is a very common step and most trainers and facilitators put in enough efforts to ensure this is done, so as to prove their credibility. However, this is only half effective unless the competency or skill levels of the participants prior to attending the learning intervention are known. Without this being gauged, it is almost not possible to state with confidence if the training has been of real use to them or whether where they stand currently is a direct result of attending the training or something else like self-learning. It also helps, if this assessment mechanism is same as the post training evaluation mechanism.
- Mechanism to gauge Learning or Knowledge Transfer
Based on the kind of learning intervention and the objectives of the same, an appropriate mechanism to evaluate the participants on their learning should be designed. For an out and out technical training, a pen-paper test may serve the purpose. For more softer interventions or Leadership interventions, a facilitated role play or a situation response test designed to gauge responses may be more appropriate. Based on how the participants fare in the test, in itself is a testimony to the skills and knowledge transferred. A comparison of the same with their assessments prior to the training, help see a clear spiked learning curve. Also, an evaluation mechanism similar to or same as the pre-training assessment mechanism will ensure a comparable result, leading to clear inferences.
- Expected Behaviour changes and ways to record the same
Most learning interventions aim at bringing about a behavioural change in a participant, even if the learning intervention is a technical training. By way of attending this training, a participant may be expected to display more confidence in dealing with customers, write better code and save time, negotiate better business deals, be more assertive, manage the team well etc. It is important to articulate the kind of behaviour changes expected out of the participants after they undergo a particular training. Once this is clear, the next thing to do would be to decide on a mechanism to observe and record the changed behaviours. In most cases, the participants' Managers and/or peers play an important role here in observing the behavioural changes, by way of recording behaviour events or conducting Behavioural Event Interviews. This however needs to be done over a period of time and not immediately after the participants have attended the training. Ideally, one month after the training should be reasonable time to begin. Before the recording begins, the Managers need to answer a simple question about the participants, "Has the participant had a change to use the skill or display the behaviour". for Eg, until someone has had a chance to negotiate a business deal after he or she has attended the training, it is almost worthless to begin the process of recording behavioural changes.
*** For both the pre training assessments and the post training evaluation, the trainers role is of utmost importance. In almost all situations, the trainer will be able to design both the mechanisms.
- Measure Results
Coming back to the ROI, there can be an enormous amount of skills learned and an excellent behaviour displayed, but you know your investment has paid off only when these are translated into quantifiable results. This is the most tricky and complicated aspect of the measurement. Most organisations struggle with the last level of Kirk Patrick's evaluation model. However, in most cases it should be possible to implement this level also, with some efforts in the pre-work area. Now is the time to ask the question WHY ? Why are we trying to achieve the objectives that we detailed out ? Why is it so important for us ? There lies your answer. If a particular employee, say John, attended the Negotiation Skills workshop, he will become a better negotiator and be able to negotiate business deals independently. This means, his Manager Alfred, will not need to be a part of all of John's business dealings. So this in turn means, if John and Alfred together spend 8 hours on preparing for a negotiation and them another 5 actually negotiating the deal, which has been costing the Organisation x+y dollars, will now be reduced to x dollars, since John is independent. Also, this means, while John is working on one, Alfred can now work on another deal simultaneously, leading to optimum utilization of resources in terms of time and efforts and increasing the probability of getting more business.
Now that we are well equipped with the entire training effectiveness measurement plan, implementation of the Kirkpatrick's evaluation model cannot be any smoother. Here's the way ahead
- Reaction
Run the participants feedback form for everyone who attended the learning and analyse the responses. Decide on what is the benchmark and compare
- Learning
Get the participants to appear for the pre training assessments and later the post training evaluations. The results and a comparison between both will help identify the learning. Again, decide on what the benchmark is.
- Behaviour
Run the behavioural event interviews after a certain time gap, with the help of the Managers or the participants' peers record findings.
- Results
If the first three levels are a success, in most probability, this level will be a success too. The pre-work helps put in the numbers in place, so that it can safely be inferred that the cost saved, efficiency increased or business increased is an outcome of the learning intervention. An analysis of the investments versus the returns, is the ROI achieved. Also this obviously means the learning intervention has been effective.
Happy Training !
