Best Practices for
the Development, Delivery, and Evaluation
of Susan Harwood Training Grants

OSHA 3686-09 2010

Contents

• Introduction
• Organization
• Background: Purpose of Susan Harwood Training Grants
• The Value of Safety and Health Training
• Characteristics of Sound Training Programs
• Overview of Best Practices for Susan Harwood Training Grants
• Principles of Adult Education
• The Principles of Adult Education: A Checklist
• Program Design, Delivery and Evaluation Elements
• Element One. Staffing
  • Element Two. Training Facilities and the Learning Environment
  • Element Three. Training Course Materials and Content
  • Element Four. Training and Overall Program Evaluation
  • Element Five. Training Program Quality Control
  • Element Six. Specific Populations to Consider
• Appendix A. Multilingual Resources
• Appendix B. Program Evaluation and Quality Control Resources
• Appendix C. References

Introduction

The Best Practices for Development, Delivery, and Evaluation of Susan Harwood Training Grants document was created in order to assist Susan Harwood grantees in developing, delivering, and evaluating training for workers and employers. The training you are providing can help protect workers' health and lives. Being a good training provider takes preparation, skill, and flexibility.

Unless specifically noted, the information in this document is not a Susan Harwood Training Grant requirement; however, the information has been shown to yield results and it is suggested that you incorporate as much of it as possible into your training program.

Organization

The document is organized in a manner that suggests best practice elements to help grantees to better develop, deliver and evaluate training to workers and employers.

A brief background on the purpose of the Susan Harwood Training Grants provides a short description of the objectives of the Susan Harwood grants. The Value of Safety and Health Training section describes the importance of quality and effective safety and health training.

In the Program Design, Delivery and Evaluation Elements section, six main program elements are identified that training organizations should consider as they develop and assess their programs. These elements are: Staffing; Training Facilities and the Learning Environment; Training Course Materials and Content; Training and Overall Program Evaluation, which also includes information on documentation and recordkeeping; Training Program Quality Control; and Specific Populations to Consider. This section provides best practice guidelines and techniques to help organizations organize and plan for a high quality and effective training program. Also included throughout are examples to use as references. At the end of each Element section, a "Checkpoint" section provides a brief summary of the important points for that topic.

Finally, the appendices of the document provide grantees with additional resource materials and information for the development of their program.

Background: Purpose of Susan Harwood Training Grants

The Susan Harwood Training Grant program provides funds for developing training materials and providing training to workers and/or employers to recognize, avoid, abate, and prevent safety and health hazards in their workplaces. The program emphasizes:

1. Educating workers on their rights and educating employers on their responsibilities under the Occupational Safety and Health Act.
2. Educating workers and employers in small businesses. (For the purposes of this grant program, a small business is one with 250 or fewer employees.)
3. Training workers and employers about new OSHA standards.
4. Training at-risk worker populations.
5. Training workers and employers about high-risk activities or hazards identified by OSHA through the Department of Labor's Strategic Plan, as part of an OSHA special emphasis program or other OSHA priorities.
6. Providing technical assistance to employers and workers.

The Value of Safety and Health Training

Quality safety and health training provides the tools to protect workers' health and lives, and to prevent work-related injury or illness. Effective training develops workers who are educated and empowered to improve the working conditions in their places of employment.

Several factors contribute to successful training. One of the most important is ensuring that the training facilitator exhibits safety and health expertise, sound instructional skills and flexibility. The Best Practices for the Susan Harwood Training Grant program provides a framework for organizations to ensure quality training is developed and delivered. In effective training, participants should learn:

• How to identify the safety and health problems at their workplace;
• How to analyze the causes of these safety and health problems;
• How to bring about safer, healthier workplaces; and
• How to involve their co-workers in accomplishing all of the above.

Other voluntary standards and minimum criteria guidance on developing and delivering safety and health training exist and OSHA encourages you to review these documents as well:

• ANSI/ASSE Criteria for Accepted Practices in Safety, Health, and Environmental Training, ANSI/ASSE Z490.1-2009
  • ANSI: American National Standards Institute
  • ASSE: American Society of Safety Engineers
• NIEHS WETP Minimum Health and Safety Training Criteria Guidance for Hazardous Waste Operations and Emergency Response (HAZWOPER); HAZWOPER Supporting and All Hazards Prevention, Preparedness and Response.
  • NIEHS: National Institute of Environmental Health Sciences
  • WETP: Worker Education and Training Program

Characteristics of Sound Training Programs

A general review of training "best practices" reveals four characteristics that sound training programs have in common. The best training programs are accurate, credible, clear and practical.

Accurate. Training materials should be prepared by qualified individuals, updated as needed, and facilitated by appropriately qualified and experienced individuals employing appropriate training techniques and methods.

To ensure continued technical accuracy, Susan Harwood Training Grant program providers are required to have OSHA review new or revised training materials for technical accuracy.

Credible. Training facilitators should have a general safety and health background or be a subject matter expert in a health or safety-related field. They should also have experience training adults or experience working with the target population. Practical experience in the field of safety and health as well as experience in training facilitation contribute to a higher degree of facilitator credibility.

Clear. Training programs must not only be accurate and believable, but they must also be clear and understandable to the participant. If the material is only understandable to someone with a college education or someone who understands the jargon, then the program falls short of meeting workers' needs.

Training materials should be written in the language and grammar of the everyday speech of the participants. Training developers should ensure that readability and language choices match the intended audience.

If an employee does not speak or comprehend English, instruction must be provided in a language the employee can understand. Similarly, if the employee's vocabulary is limited or there is evidence of low literacy among participants, the training must account for that limitation. Remember that workers may be fluent in a language other than English, or they may have low literacy in both English and their primary language. Training needs to be adjusted to accommodate all the factors that are present.

Practical. Training programs should present information, ideas, and skills that participants see as directly useful in their working lives. Successful transfer of learning occurs when the participant can see how information presented in a training session can be applied in the workplace.

Overview of Best Practices for Susan Harwood Training Grants

Training providers and instructional facilitators who recognize and embrace characteristics of sound training and principles of adult education will maximize the benefits of the training for their participants.

1. Intended audience. This guidance is primarily intended for organizations that develop and provide worker safety and health training under OSHA's Susan Harwood Grant program, but may likewise prove valuable to any organization that provides similar occupational health and safety training.
2. Training techniques, methods and modes.
   1. Proven adult learning techniques should be at the core of training development and delivery.
   2. Peer-to-peer training with activity-based learning is one effective model for worker training. Effective development of peer trainers requires ongoing organizational support to the developing peer trainer.
   3. Activity-based learning should fill at least two-thirds of training hours (no more than onethird is lecture).
   4. Training must be provided in a way that workers receiving it can understand. In practical terms, this means that the training must be both in a language and vocabulary that the workers can understand.
   5. While computer-based training (CBT) can augment the effectiveness of safety and health training for workers, it should not be the sole form of training that workers receive.
3. Needs assessment. Safety and health training should be preceded by a needs assessment to ensure the training meets the needs of the participants. The Susan Harwood Training Grant Program Solicitation for Grant Applications (SGA) requires that grantees conduct a needs assessment in order to identify safety and health hazards relevant to the training programs. Needs assessments can also be used to learn more about your target population's knowledge, experience, learning styles, reading and writing skills, and interests.
4. Evaluation of Training. Evaluating your training allows you to assess whether the training is having the desired results, and informs you as to whether you need to make changes to your training program. Training evaluation is a requirement in the Susan Harwood Training Grant Program SGA.

Principles of Adult Education

The vast majority of students who attend safety and health training sessions are adults who already possess the knowledge, skills, and abilities to work in their current occupations. The objective of safety and health training is to provide additional knowledge, skills, and attitudes to assist workers in recognizing and taking action to correct hazards in their current work environments.

The following are the basic principles of how adults learn, which is directly applicable to safety and health training programs:

• Adults are voluntary learners: Most adults learn because they want to. They learn best when they have decided they need to learn for a particular reason.
• Adults learn needed information quickly: Adults need to see that the subject matter and the methods are relevant to their lives and to what they want to learn. They have a right to know why the information is important to them.
• Adults come with a good deal of life experience that needs to be acknowledged: They should be encouraged to share their experiences and knowledge.
• Adults need to be treated with respect: They resent an instructor who talks down to them or ignores their ideas and concerns.
• Adults learn more when they participate in the learning process: Adults need to be involved and actively participating in class.
• Adults learn best by doing: Adults need to "try-on" and practice what they are learning. They will retain more information when they use and practice their knowledge and skills in class.
• Adults need to know where they are heading: Learners need "route maps" with clear objectives. Each new piece of information needs to build logically on the last.
• Adults learn best when new information is reinforced and repeated: Adults need to hear things more than once. They need time to master new knowledge, skills, and attitudes. They need to have this mastery reinforced at every opportunity.
• Adults learn better when information is presented in different ways: They will learn better when an instructor uses a variety of teaching techniques.

Three kinds of "learning exchanges" should be used during training:

• Participant-to-Participant: "Participant-to-participant" learning exchange recognizes that participants can learn from one another's experiences. Participant-to-participant exchanges should be a key feature of the training.
• Participant-to-Facilitator: Facilitators can learn as much from training sessions as participants do. On many subjects, a group of participants may have more extensive knowledge and experience in certain areas than a facilitator.
• Facilitator-to-Participant: Classroom learning needs structure. A facilitator's role is to guide discussions, encourage participation, draw out and/or add information as needed, and highlight key issues and points.

The Principles of Adult Education: A Checklist

Material adapted from Teaching About Job Hazards, Nina Wallerstein and Harriet Rubenstein, American Public Health Association, 1993.

General Principles

The best training programs take advantage of the following characteristics of adult learners:

• Adults are self-motivated.
• Adults expect to gain information that has immediate application to their lives.
• Adults learn best when they are actively engaged.
• Adult learning activities are most effective when they are designed to allow students to develop both technical knowledge and general skills.
• Adults learn best when they have time to interact, not only with the instructor but also with each other.
• Adults learn best when asked to share each other's personal experiences at work and elsewhere.

Environmental and Learning Needs Assessments

1. Does the learning environment encourage active participation?
   • How are the chairs, tables, and other learning stations arranged in the classroom?
   • How does this arrangement encourage or inhibit participation and interaction?
   • Can the arrangement be changed easily to allow different kinds of interaction?
   • Is the climate of the classroom sufficiently comfortable to allow learning?
2. Does the social environment or atmosphere in the learning environment encourage people to participate?
   • Are warm-up activities or "ice breakers" needed to put people at ease?
   • Do trainers allow participants to say things in their own words, or do they translate what is said into other words or jargon?
   • Are participants encouraged to listen carefully to each other?
   • Are they encouraged to respect different points of view?
   • Are they encouraged to use humor and is the humor appropriate?
3. People learn in different ways. Do the learning activities in the training program provide participants with an opportunity to do each of the following?
   • Listen
   • Look at visuals
   • Ask questions
   • Read
   • Write
   • Practice with equipment (if applicable)
   • Discuss critical issues
   • Identify problems
   • Plan actions
   • Try out strategies in participatory ways
4. Does the program effectively promote participatory learning activities?
   • Is enough time allotted for participant interaction?
   • Have the instructors developed workable and effective interactive activities?
   • Does the physical environment encourage interaction?
   • Does the atmosphere in the classroom encourage interaction?
   • Are the learning activities sensitive to cultural differences among the participants?
   • Does the training engage participants in critical thinking and analysis about the subject being covered?
5. How effectively do the lectures in the program encourage participation?
   • Are they combined with a participatory exercise?
   • Are they brief?
   • Are they well organized?
   • Are audio-visual aids incorporated in the lecture?
   • Does the lecturer rely too heavily on his or her notes?
   • Was there enough time for questions and comments from others?
   • Does the lecturer promote challenging questions about the content being delivered?
6. How effective are the participatory activities used in the program?
   • Are the purposes of the activities clearly specified?
   • Are the tasks that people are expected to complete clearly described?
   • Are participants given enough information to complete the expected tasks?
   • Is the information accompanying the activity clearly presented and easily understood?
   • Is the information presented relevant to the task?
   • Are participants given enough time to perform the expected tasks?
   • Are participants given enough time to share what they have learned from the tasks with each other?
   • Are the participants given a clear summary of the main points they were expected to learn in the activity?
7. How effectively do the case studies and role-playing activities in the program encourage participation?
   • Is the situation being discussed familiar to the participants?
   • Does the situation evoke strong feelings in the participants?
   • Does the situation lead to an in-depth analysis of the problem?
   • Does the situation encourage people to consider a range of possible strategies for dealing with the problem?
   • Are people provided with enough information to participate in the activity in a meaningful way?
   • Are people provided with so much information that they have no room to improvise or to call on their own experience?
   • Are people provided with an opportunity to discuss the social, cultural, and historical contexts of the situations?
8. How effectively does the organization of the program encourage participation?
   • Are discussion groups small enough to ensure participation? (No more than 4 to 6 people.)
   • Is the ratio of discussion groups to instructors small enough? (A single instructor cannot effectively supervise more than three or four groups).
   • Is there enough room to enable each group to talk amongst itself without disruption?
   • Does each group have its own moderator and notetaker?
   • Does the responsibility for leading and recording the discussion rotate among those willing to do the job?
   • Are the groups supplied with guidelines about how to lead and report their discussions?
   • Do the activities make allowances for anyone in the group who may have problems reading and writing?
9. Is the program sensitive to literacy differences?
   • Do the trainers check privately with anyone having reading and writing difficulties?
   • Is reading aloud or writing in front of the group only voluntary and never mandatory?
   • Are all instructions and other required material read aloud?
   • Do the materials incorporate enough visual aids and props?
   • Do the trainers repeat out loud anything they write on a board or flip chart?
   • Are evaluations conducted to assure that the trainees comprehend the training material?
10. Do the audio-visual aids used by the training program encourage participation?
    • Do the instructors write an on-going record of what is being discussed on the board or flip charts?
    • Are participants encouraged to challenge the record if they consider it inaccurate?
    • Are approaches utilizing integrated instructional technologies effective in eliciting participation?

Program Design, Delivery, and Evaluation Elements

This document categorizes the elements into six sections: Staffing; Training Facilities and the Learning Environment; Training Course Materials and Content; Training and Overall Program Evaluation; Training Program Quality Control; and Specific Populations to Consider.

This document does not provide guidance for craft, trade, job classification or task training. All trainees should already possess the knowledge, skills and attitudes specific to their individual craft or trade prior to taking a Susan Harwood sponsored training program. A written training plan should have been previously prepared in order for an organization to receive its Susan Harwood Grant; it should be implemented, maintained, and updated as necessary.

Element One: Staffing

There are two major staffing roles: Project Director and Instructional Staff.

1. Project Director

2. Instructional Staff

Photo provided by The Construction Safety Council.

Element Two: Training Facilities and the Learning Environment

Ideally, training facilities should have sufficient resources and equipment to perform classroom and activity-based learning in a setting conducive to effective learning.

However, often such facilities are not available and instructors find themselves having to make do and adapt to training in the environment they are given. Sometimes training will be conducted in remote or non-traditional locations. For example, in the case of day laborers, training could be "in the field" where workers are waiting to be hired. In other cases, you may be in a smaller room than anticipated, or there may be no electrical outlets or flip charts in the room where you are training. Trainers should anticipate such setbacks and prepare as best as they can.

Instructor-trainee ratios. Class sizes of about 25 people (or less) work best, especially when incorporating activity-based learning into the training experience. When class size exceeds 30 people, it is advisable to provide a second instructor and divide the class into two sections during instruction.

Training Facilities and Resources

Adequate and appropriate facilities for supporting the training include the following:

• Sufficient space for all attendees to sit comfortably during instruction;
• Sufficient room set-up for participants to interact with one another;
• Enough equipment for all attendees and demonstration equipment for the instructor/facilitator (if applicable);
• Space and facilities for small group exercises or hands-on training using equipment as part of activity-based learning; and
• Equipment, technical support, and resources sufficient to support training via technology, such as during instructor presentations or web-based training used by students to enhance learning (if applicable).

Room Layouts That Encourage Participation and Improve Learning

Top left: U-Shape. Top right: Round Tables. Bottom left: Hollow Square. Bottom right: Circle.

Element Three: Training Course Materials and Content

Training Development/Instructional Design

Training courses should be developed and updated as necessary to be consistent with the recognized principles of training development/instructional design. Training development should follow a systematic process that includes: a needs assessment, learning objectives, adult learning principles, course design, and evaluation. (Reference: ANSI Z-490.1-2009.)

One such instructional design model is called "ADDIE." This stands for the main components of the ADDIE process: Analysis, Design, Development, Implementation, and Evaluation. Training materials and content are produced as the course author progresses through the instructional design cycle. First a training analysis is performed, then the structure of the course is designed, next specific content is developed, after that the course is implemented or presented, and lastly the course is evaluated.

Particular attention should be devoted to the following with respect to course design and content:

• Demographics of the training target audience and their training needs; for instance, what is the literacy level of your target audience?
• Learning objectives, including learning objectives for each training module.
• Analysis and selection of a delivery method appropriate to the training target audience and the learning objectives.
• Instructional materials including, but not limited to, an instructor's manual with lesson plans and learning objectives, a trainee manual, training aids, and learning technologies.
• Evaluation methods as noted in the OSHA Susan Harwood Grant SGA.

Training Objectives

Every instructor has objectives he/she wishes to accomplish during training. Instructional objectives should be student-focused and state the desired learning outcome. However, it is necessary to note that, while good training can be provided, workers can still face difficulty at work when raising their voices to try to get problematic conditions corrected.

When constructing objectives, the main question that objectives answer is: What should the participant be able to do differently, or more effectively, after the training is completed?

The SMART Model is one method used to construct practical objectives.

• "S" stands for Specific. Objectives should specify what they need to achieve.
• "M" stands for Measurable. You should be able to measure whether you are meeting the objectives or not.
• "A" stands for Achievable. Objectives should be attainable and achievable.
• "R" stands for Relevant. Objectives should lead to the desired results.
• "T" stands for Time-bound. When do you want to achieve the set objectives?

Other training objective construction models include the A-B-C-D Model and Roger Mager's Theory of Behavioral Objectives.

The A-B-C-D Model:

• "A" stands for audience. State the learning audience within the objective.
• "B" stands for behavior. State the behavior you wish to see exhibited following training.
• "C" stands for condition. State the conditions under which the behavior will occur.
• "D" stands for degree. To what level (or degree) will the learner be enabled to perform?

Roger Mager's Theory of Behavioral Objectives has three components:

• Behavior: The behavior should be specific and observable.
• Condition: The conditions under which the behavior is to be completed should be stated, including what tools or assistance is to be provided.
• Standard: The level of performance that is desirable should be stated.

Examples of Objectives:

• The participant will be able to describe elements on a Material Safety Data Sheet (MSDS) when provided with a copy of an MSDS that applies to a chemical at his/her worksite.
• By the end of this course, workers will be able to identify health effects information (i.e., acute/chronic, dose/time, routes of entry) about specific chemicals present in their workplace by using the NIOSH Pocket Guide.
• The participant will be able to describe how to create a workplace hazard map, where and how co-workers can be engaged in the creation of hazard maps, and how these hazard maps can be used to identify health and safety hazards in need of correction.

Note that training objectives emphasize what the participant will be able to do, not what the instructor intends to do. In each example, workers are expected to be able to accomplish specific goals by the end of the course. For instance, using the MSDS objective as an example, the learner will be able to 'describe' the elements on an MSDS; the instructor may intend to have the learner 'list' the elements as evidence of their ability to 'describe'. On the job, the learner would be expected to 'describe' not 'list' the elements on an MSDS.

When developing learning objectives, be mindful of what is in your control in the classroom and what is out of your and the participants' control in the workplace.

Addressing Literacy in Teaching Methods and Materials

Literacy can be defined as the ability to "use printed and written information to function in society, to achieve one's goals, and to develop one's knowledge and potential." (National Coalition for Literacy, 2009.)

Some workers, including those born in the U.S., have limited literacy in their primary language. More than one-third of recent immigrants have fewer than 12 years of education. Among those from Latin-America, 35% have less than a ninth grade education. Low-literacy training materials and teaching methods that are not limited to written materials should be prepared or acquired to meet the needs of this type of training audience. (Reference: Draft Immigrant Worker Safety and Health Report, NIOSH and University of Massachusetts - Lowell.)

Multiple training methods that require fewer literacy skills should be used with this population. Methods include the use of photos, pictures, short role-plays, case studies, demonstrations, hands-on practice, and small group activities with which workers can identify and easily understand.

Training Techniques to Reach All Literacy Levels

The following suggestions are designed to help grantees adapt training techniques to reach participants at all skill/literacy levels. These techniques will also be helpful in teaching participants for whom English is a second language (sometimes referred to as English Language Learners (ELL)).

• Do not assume that all participants are equally skilled or confident in speaking, reading, writing, and math.
• Plan for plenty of small group activities where participants get to work together on shared tasks – reading, discussing, integrating new information, relating to life experience, recording ideas on flip charts, and reporting back to the whole group. In small groups, participants can contribute to the tasks according to their different backgrounds and abilities.
• Try to use as many teaching techniques as possible that require little or no reading.
• At the beginning of a class mention that you are aware that people in the group may have different levels of reading and writing skills.
• Establish a positive learning situation where lack of knowledge is acceptable and where questions are expected and valued. Participants need to be able to indicate when they do not understand and to feel comfortable asking for explanations of unfamiliar terms or concepts.
• Make it clear that you will not put people on the spot. Let them know that you are available during breaks to talk about any concerns.
• Let the group know that they will not necessarily be expected to read material by themselves during the training.
• Let people know that you will not be requiring them to read aloud. Ask for volunteers when reading aloud is part of an activity. Never call on someone who does not volunteer.
• Do not rely on printed material alone. When information is important, make sure plenty of time for discussion is built into the class so participants have the opportunity to really understand.
• Read all instructions aloud. Do not rely on written instructions or checklists as the only way of explaining an activity or concept.
• If other materials must be read aloud, read them yourself or ask for a volunteer.
• Make sure your handouts are easy to read and visually appealing.
• Give out only the most important written material. Make any other materials available as an option.
• If possible, provide audio recordings of key readings so that participants have the option to listen and read along.
• Explain any special terms, jargon, or abbreviations that come up during the training. Write them on a flip chart.
• If participants have to write, post a list of key words. This can serve as a resource for people with writing or spelling difficulties.

Participatory Methods of Instruction

Participatory training methods draw on participants' own experiences. They encourage teamwork and group problem solving. Participants have the opportunity to analyze health and safety problems in a group and to develop solutions. There can also be valuable exchanges between workers and trainers about their lives and work.

Participatory methods work quite well with people who have difficulty reading and writing. They also allow the instructor to observe who may be having difficulty with the concepts and to engage with them to ensure comprehension. Participatory methods 1) draw on the participants' own knowledge and experience about health and safety issues; 2) emphasize learning through doing without relying on reading; and 3) create a comfortable learning experience for everyone.

Samples of Participatory Methods

Participatory training methods draw on the trainee's own experiences and knowledge, as well as encourage valuable exchanges between workers and trainers. The following are examples of methods to encourage trainees to participate and be actively engaged in class:

• Ice-breakers
• Risk maps
• Role-playing
• Games
• Small group exercises
• "Trigger" visuals
• Brainstorming
• Demonstrations and hands-on activities
• Participatory lectures

For more information on each method, see The Right to Understand: Linking Literacy to Health and Safety Training.

Picture provided by the American Road & Transportation Builders Association.

Training Materials

Training materials, such as handouts, PowerPoint presentations, or flip charts, are often used as visual aids that facilitate and enhance the student's learning experience. Materials should be easy-to-read and should highlight the most important messages or needs. Keep in mind that visual aids—such as PowerPoint presentations, handouts, overheads, and flip charts—play a supportive role to the main teaching technique and do not substitute for teaching.

The following are some principles for creating the text for easy-to-read materials:

• Base the content on the workers' most important needs.
• Identify the "priority message." The priority message should convey the most important information about a problem and how it could be solved. It should be short, informative, and easy to remember.
• Don't offer so much information that a reader could feel overwhelmed.
• Organize text into short, logical sections by using headings or subtitles.
• Use words that are easy to understand.
• Define technical terms or jargon.
• Keep sentences short and simple.
• Use a conversational style and active voice, such as the kind of language that the students use.

The design of the material is as important as the content. Making the material visually appealing and easier on the eye will encourage people to read it. The following are tips for the design of the materials:

• Use a large, easy-to-read font for the main text.
• Emphasize important points with underlining, bold type, italics, or boxes.
• Include plenty of white space by using wide margins.
• Use plenty of simple illustrations to explain the text.
• Use simple line drawings that are free of clutter and abstract drawings.

Using PowerPoint

PowerPoint is not a teaching technique – it is a visual aid that can be used to enhance learning, just like flip charts, overheads, and handouts. PowerPoint will not, in and of itself, improve student learning. It is the way that instructors use PowerPoint that can encourage learning. Deciding when, where, and how it can be used appropriately is the key.

Many instructors mistakenly use PowerPoint as their main teaching technique. If an instructor teaches only by showing and reading a PowerPoint presentation, there is not much opportunity for participation. In fact, use of PowerPoint can stifle participation. The teaching turns out to be "one-way", similar to the "traditional" model of education with the instructor as expert and the students just as the receivers of information. As mentioned previously, adult education is most effective when it is participatory—when students are active participants in the learning process.

Educators need to be creative in using PowerPoint. If you plan to use PowerPoint, it is critical that it be used in such a way that participants retain and use the information, as well as participate in the learning experience.

There are three main issues to consider when using PowerPoint: content, design, and delivery.

Content:

• If you are creating PowerPoint presentations, it is best to plan your workshop or class first and then write the content of the PowerPoint slides.
• Include the main points, not lots of text.

Design:

• One concept per slide.
• Use a simple design.
• Make sure you really understand how to create and design PowerPoint slides. It takes some knowledge and skill to develop a PowerPoint presentation. For instance, getting the animation correct can be tricky.
• Do not make the mistake of designing the PowerPoint with too many graphics and animation (a common error among instructors). This can result in a design that is too complicated and difficult to read. Go easy on the graphics. Simple graphics that are easily understood are best. Do not use graphics just to make a slide look good; only use them if they have some content value. Keep animation to a minimum.
• Use lots of white space.
• Use contrast: dark on light, or light on dark. In choosing colors, make sure that the text is easy to see.
• Design from top left to bottom right.
• Use large font sizes (26 point minimum). No more than two fonts on a page.
• Limit use of bolding, italics and underlining.

Delivery:

• Always remember that PowerPoint is a visual aid, not a teaching technique.
• Your slides should serve as a focal point for the issues to be discussed. Use them to help control the pace of presentation and discussion. Read a slide aloud and follow with commentary, explanation and discussion. Remember questions and discussion are part of the learning experience.
• Practice using PowerPoint before you actually use it in a class. Make sure you are comfortable moving between slides and between information on slides.

A final note on technology: using PowerPoint requires that you have all the technology you need, that it is in good working order, and that you know how to use it. The best prepared PowerPoint will fail if the technology fails, or if the instructor has trouble using it. Always have a back-up plan in place!

Using Flip Charts

Flip charts, like PowerPoints, are visual aids that are used to facilitate, enhance or bring more clarity to the learning experience. It is an interactive and flexible aid that promotes interaction and engagement between the facilitator and the participants.

Flip charts promote participation as they are interactive—the facilitator can use the flip chart to write participants' ideas or answers. They also promote flexibility in teaching and learning, since the facilitator is writing as discussion evolves and not fixed in a "set" progression. Flip charts are low-tech, inexpensive, and easily portable. They also reinforce learning because participants can see and hear what is being talked about.

Participants feel like they have contributed if the facilitator writes what the participant says. It is best to use the words the participant uses and not to paraphrase. It is necessary to remember that what gets written needs to be discussed. Filling a room with lists of things people have said without analyzing and discussing what they say does not produce real learning.

The following are tips for using the flip chart:

• Use dark marker colors, such as black, dark brown, or dark blue for the main text. Lighter colors should only be used to highlight. Using many colors on a flip chart will catch your audience's eye.
• Print in large block letters. Do a "test" flip chart page before the workshop and make sure it can be read from the back of the room.
• Be sure not to crowd the flip chart with too much information. Only key points should be written.
• Watch your spelling. If you have problems with spelling, work on memorizing the correct spelling of words you are likely to use. But do not let spelling get in the way of using a flip chart. Creating a "spell-free" zone in the class may take some pressure off both you and the participants.
• Many facilitators prepare some flip charts in advance. Be sure to proofread any flip charts prepared in advance.
• Post pages you want participants to continue to be able to see, or pages you want to refer back to.
• Tear pieces of tape ahead of time to make it easier to post flip chart pages.
• Keep prepared flip charts covered with a blank page until you are ready for the class to see the page. When finished discussing a page, "flip" it over, unless you want the class to be reminded of the information.
• If possible, use flip chart paper with light, preprinted "grid" lines to help you print more legibly.
• Do not turn your back on the group and "talk to the flip chart." Write what is needed, and then turn back to the group. A few moments of silence is okay. Do not block your audience's view of the chart—stand to the side.
• You can, in advance, lightly pencil in reminder notes to yourself on the flip chart.

Element Four: Training and Overall Program Evaluation

Photo provided by NYCOSH.

As noted in the Susan Harwood Training Grant SGA, there are three types of training evaluation that should be conducted: 1) training session reaction assessments; 2) learning assessments; and 3) training impact assessments.

Training Evaluation

The three required training evaluations are based on the Kirkpatrick Training Evaluation model—Level 1: Reaction, Level 2: Learning, and Level 3: Behavior/impact. The Kirkpatrick Training Evaluation Model is one of the most widely used models of training evaluation. (For more information see Evaluating Training Programs, by Donald Kirkpatrick, 1975.)

There are other training evaluation techniques and methods that can be used to measure the effectiveness of training programs, such as NIOSH's Training Intervention Effectiveness Research (TIER) paradigm, Ecological Momentary Assessment, Simulation Methodologies for Training and Evaluation, etc. For more information about these and other training evaluation methods, refer to CDC's "Report from the 1999 National Conference on Workplace Safety and Health Training, DHHS (NIOSH) Publication No. 2004-132."

Training Reaction Survey. Sometimes called "smile sheets", training reaction surveys measure the trainee's immediate perceptions of the quality and usefulness of the training. The results should be considered when improving the program since they provide information regarding relevancy of the information and the teaching style of the instructor. A reaction survey is a subjective evaluation of the training course by the trainees. Questions about the trainer's presentation skills, accommodations, the course's pace, and the difficulty and usefulness of content may be included in a reaction survey. (Reference: ANSI Z490.1). Reaction to training can be conducted by: using participant feedback questionnaires; gathering informal comments from participants; and by holding focus group sessions with participants.

Examples of questions to assess participant reaction include:

• In your view, what were the three most important strengths of the program?
• In your view, what were the three most important weakness of the program?
• Please evaluate the technical level of this training
  1. It was too technical for such a short time
  2. It should have been more technical
  3. It was at the right technical level
• The course material has been
  1. Totally new to me
  2. Mostly new to me
  3. Somewhat new to me
  4. I knew most of it before
• The instructor's contribution to the training process was valuable.
  1. Strongly agree
  2. Agree
  3. Neutral
  4. Disagree
  5. Strongly disagree
• Were the materials, hand-outs and/or activities useful?

  Yes No Don't know

• Were the teaching methods effective?

  Yes No Don't know

  Comments

• Please share with us your overall evaluation of this training
  1. I would recommend this training enthusiastically to others in my workplace
  2. I would recommend this training to others in my workplace
  3. I would not recommend this training to others in my workplace
  4. Other ideas and thoughts:

Learning assessments. Learning assessments measure the skills, knowledge, or attitude that the trainee retains as a result of the training. If you use pre- and post-tests in your training, then the post-test will show the knowledge gained during training. Alternately, small group activities can serve as a "post-test" to see if participants are "getting it." These can be more effective than a post-test because they give facilitators the opportunity to address any problems in understanding that may arise during the program itself. With a true post-test, the class is over and you no longer have an opportunity to help the participant understand the concept s/he missed. You may also conduct follow-up evaluations or focus groups three to six months following the training to check retention of information.

Examples of questions you can ask to assess learning include:

• I feel well informed about how workers should be best protected from hazards on the job
  1. Strongly agree
  2. Agree
  3. Neutral
  4. Disagree
  5. Strongly disagree
• The most common way that toxins enter the body is through
  1. Skin contact or absorption
  2. Eye contact or eye absorption
  3. Inhalation- breathing
  4. Ingestion- swallowing
• Administrative, engineering and work practice controls are all used to limit lead exposure. If all these controls are used and the workers' exposure to lead is still above the permissible exposure limit (PEL), then respirators MUST be used in order to protect the worker's health
  1. True
  2. False
  3. I Don't Know

Training impact assessments. Training impact evaluations are typically conducted three to six months after the training and could be conducted by written/electronic surveys or by focus groups. Measures include the level of worker involvement on safety committees, increases in the number of formal complaints filed, or increases in sharing safety and health information with co-workers who did not participant in training.

Examples of questions to ask in order to assess the impact of your training include:

• Since the training, which of the following have you done that you did not do before your training?
  1. I pay more attention to the materials I am working with (transporting, loading, or unloading)
  2. I make sure I have shipping papers, and have read them
  3. I ask for material safety data sheets
  4. I look at the placards associated with the materials I am handling
  5. I speak up if I think there is a safety and health issue
  6. I work with hazardous materials with more caution and awareness
  7. I have not done anything differently yet
• What have you been able to do since returning from training to share new safety and health knowledge with other workers?
  1. Write in my company/union newspaper
  2. Talk in safety and health/union meetings
  3. Talk informally on the job
  4. Work with the company/union to communicate safety and health priorities to management
  5. Train other workers
  6. Work on a safety and health committee
  7. Nothing yet
• Since you attended this particular training program, have you tried to make improvements in health or safety or participated in other health and safety-related activities in your workplace or in a workplace where you represent members?

  Yes No

  If you answered "Yes": go on to question #3 and answer questions #3 – #9.

  If you answered "No": what was/were the reason(s) you were not able to be active in health or safety issues with your union since attending this training program (please check all reasons that apply:

  Other people take care of safety and health
  No issue came up that needed addressing
  Local union had other pressing issues to deal with
  Not a priority for me in the work I do with the local union
  Concern about retaliation for raising health and safety issues
  Not enough time between when we took this training and now to do much
  Other:

Overall Program Evaluation

Key questions for evaluating the overall quality and appropriateness of a training program should include the following:

• Were the program objectives clearly stated?
• Was there evidence that the program accomplished its objectives?
• Were appropriate facilities and staff available and committed to the program?
• Was there an appropriate mix of classroom, demonstration, and activity-based learning?
• Did new training technologies that have been integrated impact the program being assessed?
• Is the program providing quality worker health and safety training that fully meets the intent and requirements of the applicable regulations?
• What are the program's strengths?
• What are the program's weaknesses?
• How can the program be improved?
• Are trainers using the training outline, objectives and content provided?
• Are the course materials current and the delivery methods relevant to the training target audience?

Documentation and Recordkeeping

A recordkeeping system should be established for controlling all records and documents to ensure that the records are:

• Retrievable, readily identifiable, and maintained in an orderly manner;
• Dated, current, accurate, and legible; and
• Retained for at least one year following the training.

Certain regulations require specific records be kept for proof of completion of required training. Organizations will want to keep enough documentation to help them compile quarterly reports under the grant requirements.

Student records should identify:

• The target audience and stated learning objectives;
• Sources used to develop the training materials;
• The persons designing and developing the training and their qualifications;
• All training materials developed for the course; and
• Plans for evaluation and continuous improvement of the course.

Trainer records should identify:

• Date, location and duration of the training;
• Course name;
• Name(s) of the trainer/s;
• Materials used; and
• List of trainees participating in the class.

Course evaluation records:

• Document formal or informal training evaluation results.

Element Five: Training Program Quality Control

While a written quality assessment and control plan is not required under the Susan Harwood Training Grant, having one will help ensure the overall quality of your training program. A quality control plan is not the same as training evaluation. A quality control plan can ensure that each element of your training program is being done well and achieving its goals. It provides a tool with which to review your training program. While not necessarily all-inclusive or stringent, the "Checkpoints" in this document could essentially be your quality control tool.

Quality Control Program Assessment

Maintain a written quality assessment and control plan that considers the adequacy and appropriateness of:

• Instructor performance;
• Course evaluations, including feedback, updating, and corrective action;
• The role of trainee evaluations to provide feedback for training program improvement;
• Course materials;
• Outside reviewers to provide overall technical policy guidance (if applicable);
• The competency and role of the outside reviewers (if applicable); and
• The minutes or reports of the outside reviewers' meetings or written recommendations (if applicable).

Annual Update

The Project Director should review the written quality assessment and control plan periodically and update it as appropriate. The periodic update provides an opportunity to consider how well the program has:

• Included all applicable regulatory changes;
• Implemented course updates that have occurred during the preceding year;
• Integrated new training technologies; and
• Integrated new modules within the training program.

Photo provided by The Construction Safety Council.

Element Six: Specific Populations to Consider

Specific Populations to Consider

1. Non-English speaking. A person's verbal ability often tends to exceed his or her literacy levels. For best results, trainers should communicate in the native language of the participants and should provide materials in the participants' primary language. If the trainer does not speak the trainees' primary language, interpreters may be used. However, be sure to use a translator with trusted credentials. It is not advisable to use one worker as a translator for the others. Employ approaches similar to those used for low-literacy audiences.
2. Limited English proficiency. Materials used with those who have limited English proficiency should be easy to understand or written in languages other than English. Favor those materials or curricula that encourage interaction, student input, and critical thinking. (Szudy and Gonzalez Arroyo). Consider using pictograms, visuals, and demonstrations or other methods that are non-verbal to convey information. Employ approaches similar to those used for low-literacy audiences.
3. Contingent workers, day laborers and temporary workers. Employ approaches similar to those used for low-literacy or non-English speaking audiences. This will ensure maximum communication of the training content with minimum language interference. Favor visual and verbal methods over written text.
4. Young Workers. Workers who are high school or college age and recent additions to the workforce require additional guidance. They may be fully able to intellectually comprehend training information, but they lack the experience that time in the workforce provides. Additional emphasis should be placed on safety and health precautions, experiential exercises and demonstrations that exhibit the inherent danger that lurks in the workplace.

Photo provided by United Steelworkers - Tony Mazzocchi Center.

Appendix A: Multilingual Resources

Below are resources to use when looking for (mostly) Spanish language health and safety material. Remember that simply translating English health and safety materials into Spanish or another language is not necessarily adequate for your target population to understand the material. There are many different terms and dialects in Spanish (and other languages) and you need to ensure you are using the correct ones. In addition, using the correct literacy level is just as important in other languages as it is in English. It is best to test the translated materials using a focus group made up of a subset of your target population.

English to Spanish OSHA Dictionary

• Frequently used Construction Industry terms
  http://www.osha.gov/dcsp/compliance_assistance/spanish/osha_construction_terms_ensp_freq.html
• Frequently used General Industry Terms
  http://www.osha.gov/dcsp/compliance_assistance/spanish/osha_general_terms_ensp_freq.html

OSHA Dictionaries - English-to-Spanish and Spanish-to-English

https://www.osha.gov/dcsp/compliance_assistance/spanish_dictionaries.html

National Institute for Occupational Safety and Health

http://www.cdc.gov/spanish/niosh/pubsnum-sp.html

This site includes links to NIOSH publications on a variety of construction topics, and also provides links to other agencies and organizations that have Spanish resources.

Multilingual Health and Safety Resource Guide

http://lohp.org/library/multilingual-guide/

This is an extensive collection of links to worker health and safety training materials (such as factsheets, curricula, and checklists) that are available from many sources online in languages other than English. It was prepared by LOHP and the California Commission on Health and Safety and Workers' Compensation. The Guide was updated in late 2005.

At the end of the Guide, you'll find a listing of web sites with additional links to health and safety information and resources in other languages. The Guide is designed to complement the statewide Worker Occupational Health and Safety Training and Education Program (WOSHTEP).

Electronic Library of Construction Occupational Safety and Health (eLCOSH)

http://www.cdc.gov/niosh/elcosh.html

This Electronic Library was developed and is maintained by CPWR—The Center for Construction Research and Training and is intended to provide accurate, user-friendly information about safety and health for construction workers from a wide range of sources worldwide.

Information is organized by hazard, trade and job site, and they have educational materials including tailgate guides, hazard alerts, and worker pocket cards and brochures.

They reference construction-related materials available in other languages, including: Creole, French, German, Italian, Polish, Portuguese and Spanish.

Georgia Tech Spanish Language Construction Training Website

http://www.oshainfo.gatech.edu/hispanic/empieze-aqui.html

This site provides training guides in Spanish on several construction health and safety topics—scaffolding, fall protection, electricity, handling of objects/materials, and trenches and excavations. For each topic, there are educational materials presented in various formats, including posters, pamphlets, tailgate session guides, and formal presentations.

Labor Occupational Health Program, UC Berkeley

http://lohp.org/

LOHP has produced a curriculum on construction safety, Tailgate Training for California Construction Workers, which is available in both English and Spanish. This book can help construction foremen and other trainers conduct effective safety training sessions on the job. It includes detailed Training Guides on 14 construction safety topics. There are also 14 matching Checklists on related Cal/OSHA regulations. For some topics, Case Studies (based on actual injuries and accidents) and Factsheets are also provided. Both the English and Spanish editions of Tailgate Training for California Construction Workers are available for sale and can also be read online in English and Spanish.

Occupational Health Branch, California Department of Health Services

http://www.cdph.ca.gov/programs/ohb/Pages/default.aspx

BuildSafe produced a health and safety tailgate training kit in English and Spanish. The kit consists of Safety Break cards that cover 23 general construction safety topics and are linked to information in the Cal/OSHA Pocket Guide for the Construction Industry. These cards are simple to use and designed to improve the quality of tailgates.

MI Trabajo Seguro

http://www.MiTrabajoSeguro.org

This Spanish-language website has helpful safety and health information available for construction workers. Developed in collaboration with the hit telenovela "Pecados Ajenos", this site introduces helpful construction safety information to workers and their families, parallel to a construction safety storyline on the show.

Appendix B: Program Evaluation and Quality Control Resources

Evaluation of Worker Training Programs

It is important to periodically evaluate the training program to make sure that training is effective and that programs are achieving the intended results. Evaluation should determine how well a program is implemented, how much knowledge is gained by students, and the outcomes of the training.

The following are resources and guides that can be used to create and conduct a successful evaluation and maintain quality control of the training program.

Resource Guide for Evaluating Worker Training: A Focus on Safety and Health (1997)

The Resource Guide for Evaluating Worker Training is published by the National Institute of Environmental Health Sciences (NIEHS) and its Worker Education and Training Program (WETP) in collaboration with the George Meany Center for Labor Studies. The goal of the Resource Guide is to ensure that workers receive safety and health training that works.

The Resource Guide presents and explores a range of successful evaluation ideas, techniques, and tools for:

• Identifying areas for program improvement;
• Measuring the short- and longer-term accomplishments of a worker training program; and
• Assessing whether, and to what extent, training has brought positive changes to the workplace.

Examples of instruments developed and used by NIEHS WETP awardees to evaluate their worker training programs are displayed throughout the Guide.

For copies of the Resource Guide, please contact the NIEHS National Clearinghouse for Worker Safety and Health Training at (202) 331-7733 or email wetpclear@niehs.nih.gov.

Evaluation Resources

UCLA-Center for the Study of Evaluation Program Evaluation Kit (1987)

The Program Evaluation Kit developed by the UCLA Center for the Study of Evaluation consists of 9 volumes of practical guidelines for designing and implementing evaluations.

• Volume 1, The Evaluator's Handbook (provides dozens of checklists)
• Volume 2, How to Focus an Evaluation
• Volume 3, How to Design a Program Evaluation
• Volume 4, How to Use Qualitative Methods in Evaluation
• Volume 5, How to Assess Program Implementation
• Volume 6, How to Measure Attitudes
• Volume 7, How to Measure Performance and Use Tests
• Volume 8, How to Analyze Data
• Volume 9, How to Communicate Evaluation Findings

Photo provided by NYCOSH.

Measuring and Evaluating the Outcomes of Training (1996)

This is a collection of research papers that was presented at the 1996 NIEHS Spring meeting on measures and evaluation of safety and health training programs.

Appendix C: References

• Assessing Occupational Safety and Health Training: A Literature Review, U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, DHHS (NIOSH) Publication No. 98-145, June 1998.
• Criteria for Accepted Practices in Safety, Health, and Environmental Training, American National Standards Institute, Inc. (ANSI)/American Society of Safety Engineers (ASSE), Z490.1-2009.
• Culture, Health and Literacy: A Guide to Health Education Materials for Adults with Limited English Literacy Skills, http://healthliteracy.worlded.org/docs/culture/about.html.
• Delp, L. et al., Teaching for Change: Popular Education and the Labor Movement, UCLA Center for Labor Research and Education, 2002.
• Evaluation of the Limited English Proficiency and Hispanic Worker Initiative, U.S. Department of Labor, Employment and Training Administration, Prepared by Coffey Consulting, December 2009.
• Immigrant Worker Safety and Health Report, from a conference on research needs, draft NIOSH scientific information disseminated for peer review, NIOSH and University of Massachusetts Lowell, April 2010.
• Minimum Health and Safety Training Criteria: Guidance for Hazardous Waste Operations and Emergency Response (HAZWOPER), HAZWOPER-Supporting and All-Hazards Disaster Prevention, Preparedness and Response, National Institute of Environmental Health Sciences (NIEHS) Worker Education and Training Program (WETP), January 2006, http://tools.niehs.nih.gov/wetp/.
• ODP Blended Learning Approach, version 1.0, ODP/DHS. November 27, 2003.
• OSHA Outreach Training Program Guidelines, U.S. Department of Labor, Occupational Safety and Health Administration, February 2009.
• OSHA Training Standards Policy Statement, U.S. Department of Labor, Occupational Safety and Health Administration, April 28, 2010, http://www.osha.gov/dep/standards-policy-statement-memo-04-28-10.html.
• Report from the 1999 National Conference on Workplace Safety & Health Training: Putting the Pieces Together & Planning for the Challenges Ahead, Cosponsored by the National Institute of Environmental Health Sciences and Occupational Safety and Health Administration, Department of Health and Human Services, HHS (NIOSH) Publication No. 2004-132, February 2004.
• Robson L., Stephonson C., Schulte P., Amick B., Chan S., Bielecky A., Wang A., Heidotting T., Irvin E., Eggerth D., Peters R., Clarke J., Cullen K., Boldt L., Grubb P. A systematic review of the effectiveness of training & education for the protection of workers. Toronto: Institute for Work & Health, 2010; Cincinnati, OSH: National Institute for Occupational Safety and Health, DHHS (NIOSH) Publication No. 2010-127.
• Szudy, Elizabeth and Gonzalez Arroyo, Michele, The Right to Understand: Linking Literacy to Health and Safety Training, Labor Occupational Health Program, University of California at Berkeley, 1994.
• Training Requirements in OSHA Standards and Training Guidelines, OSHA 2254, U.S. Department of Labor, Occupational Safety and Health Administration, 1998 (Revised).
• Wallerstein, N. and Rubenstein, H., Teaching about Job Hazards, A Guide for Workers and Their Health Providers, American Public Health Association, 1993.

U.S. Department of Labor

For more information:

www.osha.gov (800)321-OSHA (6742)