How can managers know that classroom training has "paid off," that training objectives have been achieved? Are managers satisfied with what training has accomplished?
These questions have dogged trainers, performance improvement professionals, and managers since the earliest days of industrial adult education - that is to say, from about the turn of this century. Kirkpatrick ( 1976, 1983), among others (Bramley, 1990; Brinkerhoff and Dressler, 1990; Clegg, 1987) have made Herculean efforts to provide tools to help answer these questions, and the reader will certainly want to consult these evaluation gurus for detailed help on the variety of means available to evaluate training effectiveness.
Among the most useful of these tools are the four evaluation levels proposed by Kirkpatrick (4, 5) who suggests that training can be evaluated at four different levels:
Level 4: Impacts of training on business performance.
Level 3: Application of classroom learning to on-the-job situations.
Level 2: Learning acquired by participants through training.
Level 1: Participant reactions to training, and the training environment.
In this model, each successive level provides more credible data that permit an objective evaluation of training impacts. Also, as one climbs the evaluation "ladder," the evaluation effort becomes more complex and more time-consuming. Levels 2 and 3 are the backdrops for the following discussion. This article contends that the Evaluation Meeting can be an effective tool for management to assess the extent to which new skills were acquired, and if these skills are being used in work situations. A Level IV evaluation remains to be carried out.
This article discusses the “Evaluation Meeting" as a way that line managers can be directly involved in evaluating training effectiveness and impacts. This is our focus because the author had the opportunity of developing and presenting thirty-two hours of leadership skills training for three key employees of a small manufacturing facility: a production manager, supervisor, and lead person. The General Manager of this facility decided to follow-up on the classroom training by conducting a series of these meetings, who thought this format to be an effective means to determine if training was successful. The author embraced this idea because of its immediately apparent advantages, namely:
(1) such a format clearly demonstrates top management interest in determining training success and impacts, and did not have to be convinced of the need to support training efforts, as is sometimes the case with line management;
(2) this manifestation of interest was clearly communicated to training participants, who were plainly alert to management's interest in their performance during - and following - classroom training.
The Evaluation Meeting is fairly straightforward in both design and execution. Its purpose is to help management – in this case, the General Manager – understand what and how much was learned by participants. Thus, the Meeting is most clearly seen as a means to gather data that assess learning acquired through training: that is, Level II evaluation data.
However, this Meeting is also an opportunity to evaluate impacts at Level III – that is, on-the-job application. This will be seen from the discussion below.
Background to the Project
Ceram-Tek manufactures ceramic parts and accessories, and is located in Corona, California. A subsidiary of Homexx International, Inc., it is the major supplier to Homexx of such ceramic products as bath and kitchen fixtures. Homexx, in turn, sells to its customers who are principally in the kitchen and bath industry.
Ceram-Tek numbers about twenty full- and part-time employees, working in two shifts, and is led by the General Manager. Reporting to him is a Production Manager, Supervisor and Lead person. None of the three trainees had participated in any prior leadership skills training. Training was conducted in Spanish because it was the first language of all three participants. Training materials were also in Spanish. The average educational level of the participants was estimated to be < 6th Grade, in Mexico, which means that students are functionally literate and are able to read and write, but somewhat below their grade level. With the exception of one participant, none had participated in any formal skills training.
The Training Design: Classroom plus Coaching
The goals of this training were to equip participants to identify and resolve interpersonal and technical problems, manage conflicts and disagreements more effectively, and to communicate more clearly with subordinates, with each other, and with management. Of the thirty-two hours of this program, twenty of these were classroom training. This was complemented with twelve hours of one-on-one coaching of trainees. The purpose of this coaching feature was to facilitate transition of skills acquired in the classroom, to shop floor application. Coaching was structured so that each trainee met with the instructor for one hour, once per week, for a total of twelve coaching hours. Coaching sessions were structured around the training objectives for that particular week, e.g., how to apply the four basic leadership styles that were being taught, how to use active listening skills, etc. and personalized to the particular learning needs of each participant. The author also submitted recommendations on how to reinforce training with participants, in their daily work.
Gauging Training Results: The Evaluation Meeting
Towards the end of training, the General Manager told the participants that he wanted to meet with them after training to determine how much they had learned as a result of training. He asked the three trainees to review their training and to be prepared to present individual thirty-minute summaries of key ideas or skills acquired during training, in a meeting to be held two weeks hence. He asked participants to use that time to discuss and decide among themselves which topic and skills areas each person would present, and offered to provide them the means to prepare for their presentations, e.g., help in preparing an outline for their presentation, providing them with appropriate supplies and administrative support.
About a week later, in informal discussions between the General Manager and training participants, the General Manager concluded that the three employees had not agreed about how to make their presentations. To provide direction and structure, he asked each person to refer to their training manuals, and divide them among themselves by subject matter so that each participant presented approximately one-third of the training content. Participants agreed to do this. In the interim, the General Manager continued to make himself available to the three employees, in the event they needed help in preparing their presentations. The Meeting was held two weeks following training, and lasted about ninety minutes.
There were three presentations during this Evaluation Meeting. The first trainee presentation covered Modules 1-4 of an eight-module training program, namely: "Interpersonal Communication," "Leadership," and "Conflict-Resolution," and "Delegation." The second presentation accounted for Modules 5-8 of training: "Motivating and Disciplining," "Problem-Solving," Train-the-Trainer Skills," and "Legal Issues." The third and last presentation dealt exclusively with the how to use the Ishikawa Diagram to solve work problems.
During and following each presentation, and to test learning, the General Manager asked questions; e.g., " What was the most important skill you learned?" "What are the most common conflict-resolution styles that you see us using here at Ceram-Tek?" "How do you use the four basic communication styles (taught in training) with the people you supervise?"
Overall results were mixed, but mostly positive. According to the General Manager, the employees reported that the most useful skills acquired were interpersonal ones. For instance, training reportedly enhanced participant's ability to communicate and listen. One Level 3 evidence of improved communication at work was less time spent by the Manager and the three employees in locating materials, messages, and equipment. Another evidence is that some workflow improvements have resulted from improved communication in message flow and reduced lost time. For instance, subsequent to training, the General Manager instituted thirty-minute overlap meetings between Day and Night shift Supervisors. Previously, shift overlap lasted no more than five minutes, which resulted in unclear, confusing, or missed communication opportunities. With increased overlap, Day and Night shift problems related to scheduling, problems with machine operation, and personnel issues were minimized, and resulted in shortened lines of communication and a smoother "handover" of work shifts.
A third evidence to support the conclusion that some classroom learning was being applied to workplace situations was the institution of Monday morning meetings between the General Manager, the Production Manager, the Supervisor and the Lead, to review production and other issues for the upcoming week, and to anticipate possible problems. Prior to training, these meetings were not held, or -when held - did not always include all four key personnel. Meeting results were recorded and used to improve productivity, workflow, communication, etc.?
Concurrent with classroom training and coaching, was training the Supervisor and Lead on how to use a new e-mail system. All three personnel have adopted use of this system, which - according to the General Manager - has further contributed to clearer and more opportune communication among them. Clearer communication, in this case, was measured by a reduction in the number of machine, personnel, and labor scheduling problems that was the norm before training and which resulted from the sporadic and informal communication between the Supervisor and Leads. Improved communication between these employees and the General Manager has resulted in a "just-in-time" process for scheduling of temporary labor, and more timely scheduling for machine maintenance and tooling.
The Evaluation Meeting demonstrated that the trainees had gained some measure of skill in using the Ishikawa Diagram to solve problems. The Meeting satisfactorily demonstrated that all three had acquired the basic tools associated with such other training Modules as "Delegation," "Conflict-Resolution," "Motivating and Disciplining," and "Legal Issues."
The principal author submitted a report at the conclusion of training, with recommendations for management action. Two of recommendations were: (1) collaborative development of job descriptions that define main duties, tasks and responsibilities of the Production Manager, the Supervisor and Lead; (2) developing performance and career development goals for the three trainees. Both of these recommendations have been implemented: job descriptions have been developed and, while the General Manager has initiated career-planning discussions with his three key personnel, more work remains to be done in this area. Additional recommendations to: (3) formally train these personnel in basic Quality tools, (4) systematically plan toward development of work teams, and to (5) formally train temporary employees and other shop floor personnel, are either under consideration by management, are in the process of being implemented, or as yet have not generated any Level 2 or Level 3 evidences.
How the Process Can be Improved
Both authors agree that the Evaluation Meeting was a useful and reliable way to gauge and assess the overall "value" of training to the Ceram-Tek organization; that is, the Meeting enabled management to reasonably conclude that training was worth the investment. At the same time, both agree that any future Evaluation Meetings should reflect the following improvements:
1. Participants should have the option of making at least part of their presentations in Spanish, provided they have adequate time to prepare and that they can choose which portions to present, and in which language.
2. Participants need to be more formally prepared to make their presentations. This preparation should include coaching on how to prepare a presentation, selecting appropriate presentation media, e.g., overhead? flipchart? etc., how to prepare a written presentation outline, and a presentation rehearsal. This preparation requires an investment of time if the Meeting is to yield desired results.
3. There was little prior discussion among all parties about what each expected from the Meeting, and no written agenda. Trainee participants and management should discuss and agree on what each expects from such a Meeting. A Meeting agenda should reflect these expectations, and be shared beforehand among all attendees.
4. Document results and conclusions from the Meeting. This documentation can then be used to plan further Meetings, and as templates for discussion between management and key personnel about career development needs, including possible future skills training for trainees.
5. One-on-one coaching between the Instructor and individual participants was a feature of this training intervention. However, the contribution of coaching to the overall effectiveness of this training effort is unclear. This is partly because the Evaluation Meeting did not directly incorporate learnings from the coaching sessions; consequently, there was no format to evaluate the extent to which coaching did or did not supplement the classroom activities. Future coaching efforts could be improved by more formally and explicitly linking coaching to classroom work, and by using the Evaluation Meeting to assess coaching's relationship to classroom accomplishments.
Conclusions
Management concluded that the Evaluation Meeting is a cost-effective way to assess training impacts at three of Kirkpatrick's evaluation levels. It permits management to draw conclusions about the value of training to the organization, and is also a clear demonstration of management interest in training. It requires a formal agenda and investment of organization time in preparing trainees to present key learnings. Its potential pitfalls, as discussed above, can be anticipated and avoided, with proper use of meeting planning skills.
The Meeting clearly communicates management interest and involvement to training participants. In the above case, participants plainly saw the need to apply classroom skills training to work on the shop floor. This was largely driven by management requiring them to review training material, and make formal presentations to management, at the conclusion of training. The Meeting, therefore, is a means to draw some conclusions about (1) the skills participants acquired, through training, and; (2) the degree to which participants are using these acquired skills in their work. Management reached these conclusions by relying on "evidences;" for example, improvements in workflow, shortened communication lines, and improved delegation practices.
Author Biographies
Anthony Griffin has delivered human resource development and performance improvement services through his company, Teamworks, since 1994. He specializes in training and consulting interventions for employers with large numbers of Latino employees. He can be reached at: training@teamworks1.com. Phone: (951) 784-9330. Fax: (951) 784-5003. His website is: http://www.teamworks1.com.
Adam Lang has served as General Manager of Ceram-Tek since 1995. Prior to coming to
Ceram-Tek, he worked as a ceramic engineer, and in marketing, for Sylvania, He has a B.S. in Ceramic Engineering from Alfred University, and an M.B.A. from Pepperdine University. He serves on the Executive Committee of the Association of American Ceramic Component Manufacturers, and is a member of the American Ceramic Society. He can be reached via e-mail at: adam.lang@ceramtek.com or at www.ceramtek.com.
References
(1) Bramley, Peter (1990). Evaluating Training Effectiveness. Berkshire: McGraw-Hill.
(2) Brinkerhoff, R.O. and Dressler, D.E. (1990). Productivity Measurement: A Guide for Managers and Evaluators. Thousand Oaks: Sage.
(3) Clegg, William (1987, February). "Management Training Evaluation - an Update." Training and Development Journal, pp. 65-71.
(4) Kirkpatrick, Donald (1983, November) "Four Steps to Measuring Training Effectiveness." Personnel Administrator, pp. 19-25.
(5) Kirkpatrick, Donald L. (1976) "Evaluation in Training." In Robert L. Craig (Ed.), Training and Development Handbook, 2nd ed. (Chap. 18), New York, NY, McGraw-Hill.
