There are many resources written on Kirk Patrick’s four level evaluation model and Jack Phillips’ five level ROI model. Each model is a subject of a book and it is not possible to explain both models in a single article. In my post I will try to point out essential points in the training effectiveness measurement process and will share some hints with you.

During measurement of a training program’s return on investment, what you need most is the support of the related business unit because through the process you will need data, reports and comments of the business unit. Especially when you decide to measure ROI of a training program for the first time, select the program of a business unit that will support you most. In addition to this before beginning the measurement process, define all your targets you aim to achieve with this training program and write them down.

In the first step of the measurement; participants’ reaction to the program is evaluated. When you decide to apply Phillips’ five level measurement model, you need to prepare an evaluation sheet especially for your training program. In the form ask the participants the following questions:

• How satisfied are you with the program?
• Did the program cause you to gain new skills and information?
• Is the knowledge and skill set you gained is applicable on the job?
• How much will the training program improve you performance on the job?

By asking these questions, you may have detailed information regarding the participants’ views and can compare these views with the results you obtain at other levels of measurement.

How much learning occurred due to the training program is evaluated in the second step. In this step generally used evaluation tools are assessments, tests and simulations. In order to obtain reliable results, make sure that the questions you prepare for these tools are in line with the training program and they are covered by the content of the program.

In the next phase; on the job application of the training program is assessed. The competencies that are aimed to be developed with the program are evaluated with the methods such as survey, interview, focus groups and observation. If you plan to use a survey or conduct interviews, the questions you prepare should directly indicate behaviors and be concrete. Additionally, it is better to examine a single topic at each question. There is also another crucial point regarding the survey method: You should prefer to send and collect the survey methods via an anonymous platform. In this way, participants will not have to share their identities and answer your questions objectively.

In the fourth phase, program’s impact on the business results is evaluated. The measures used in this phase should be determined according to the program targets. The targets may be grouped under two main headings: financial benefits and intangible benefits. Financial benefits are the business results whose monetary value can be calculated. Intangible benefits are the results whose monetary value is very difficult or impossible to calculate. Customer satisfaction level, leadership development are some examples of intangible benefits.
The business results you have in this point may not only be linked to the training program. In order to sustain reliability of your measurement, you need to isolate the effects of the training from other factors. In his model, Jack Phillips offered some strategies to isolate the effects of the program. Among these strategies most commonly used ones are use of control group and estimation method. In the use of control group method, an experimental group that attended the training and a control group that did not are compared. In this method the essential point is that the two groups you compare should be identical. In the estimation method, participants’, their managers’ and/or their colleagues’estimates on the impact of the training are taken into consideration.

In the last and the fifth step training program’s Return on Investment (ROI) is measured. In order to be able to calculate ROI in this step, you need to target at least one financial benefit in the fourth step. ROI is generally expressed as a percent when the fractional values are multiplied by 100. In formula form ROI becomes;

ROI (%) = ((Program’s Financial Benefit – Program Costs) / Program Costs)*100

While doing this calculation, keep in mind that there are also intangible benefits. A training program’s ROI may not be too high, it may even be negative but the intangible benefits of the program may have important value for the company.

Resources
http://www.roiinstitute.net/tools/. (n.d.). Retrieved 08 12, 2013, from www.roiinstitute.net
Phillips, J. J. (1997). Return On Investment in training and performance improvement programs. In J. J. Phillips, Return On Investment in training and performance improvement programs. Gulf Publishing Company.