January 26, 1990

MEMORANDUM FOR:   REGIONAL ADMINISTRATORS

THROUGH:          LEO CAREY, Director
                 Office of Field Programs

FROM:             BRUCE HILLENBRAND, Director
                 Federal-State Operations

SUBJECT:          State Plan Monitoring and Evaluation Procedures

As you know, the Occupational Safety and Health Administration further responded on September 11, 1989 to studies on OSHA's State plan monitoring and evaluation system which were conducted by the General Accounting Office (GAO) and Department of Labor Office of the Inspector General (OIG). In doing so, OSHA agreed to undertake a comprehensive review of the current policies and procedures, considering the recommendations contained in the GAO and OIG reports as well as OSHA's own concerns with the current system, with the goal of implementing appropriate revisions by Fiscal Year 1992. In addition, the Agency has made a commitment to implement a number of measures to address the GAO and OIG recommendations on an interim basis while the comprehensive review is underway.

The purpose of this memorandum is to provide clarification of the existing State plan monitoring policies and procedures and additional, interim guidelines to be followed in carrying out those aspects of the monitoring function which are the subject of GAO and OIG recommendations. Effective immediately, each Region should take all necessary steps to observe the following guidelines.

Following up on State corrective action:

OSHA Instruction STP 2.22A addresses tracking State responses to OSHA evaluation recommendations in those sections dealing with the quarterly discussion with the State and with the annual evaluation report. As noted on page VIII-5, paragraph IV.B.5.e, the report should contain recommendations for improvement of State performance where warranted. Any recommendation should be appropriately labeled as such and be highlighted by such means as setting it off from the remainder of the report text. Paragraphs IV.B.5.a.(2) and b.(2) on pages IV-3 and 4 advise that the quarterly discussion with the State is an appropriate time for the Region and State to communicate with regard to the sufficiency of State corrective actions. In addition, paragraph VIII.B.5.c.(4) on page VIII-4 requires that each section of the annual evaluation report should discuss State responses to recent previous recommendations in that program category as well as recommendations that have not been addressed by the State and an assessment of the impact on the State's program.

In meeting the need to follow-up on State corrective actions it is essential to monitor closely their implementation and efficacy. The first step in this process is for the Region to ensure that the State meets the requirement to respond formally within 60 days of receiving an annual evaluation report that contains recommendations, and that a copy of the State's response is forwarded promptly to the National Office (see paragraph VIII.C.3. on page VIII-6). The Region should evaluate the State's response to determine whether it satisfactorily will address the evaluation findings and recommendations and, if it does not, resolve any deficiencies.

The Region should closely track State performance in program areas which are the subject of recommendations through regular review of computer reports and queries or information derived through other means, including special investigations where appropriate, as provided in Chapter IV of STP 2.22A. And, the Region periodically should discuss with the State its progress in completing corrective actions. The Region must ensure that any recommendation in a State's annual evaluation report is fully discussed in the subsequent report. The Region should notify the Director of Federal-State Operations of significant problems affecting the State's effectiveness which are encountered at any point.

In a related area, the OIG report noted that quarterly discussions with the States often were not timely or documented. Our comprehensive review of the monitoring system will address the appropriate nature and frequency of communication with the States. In the interim, Regions should maintain a written record with their other monitoring files of all significant information covered in formal discussions with the States, including State actions in response to previous recommendations.

Assuring that on-site activity occurs during the course of each State's annual evaluation:

As we indicated in our response to the GAO and OIG reports, it is expected that some degree of on-site activity will occur in the course of each State's evaluation period and be reflected in the annual evaluation report.

OSHA's current procedures for on-site State plan monitoring are discussed in Chapter IV and Appendices K-L of OSHA Instruction STP 2.22A. The use of these procedures is suggested principally where information available from the IMIS or other readily available data sources is not sufficient to evaluate a particular aspect of State performance.

In addition to the traditionally recognized on-site tools of interviews, case-file reviews, accompanied visits and spot-check monitoring visits, the Regions should be aware of, and use as appropriate for the activity being monitored, on-site sources of information such as are described in paragraph IV.C.3.b.(3)(e) on page IV-11. These include attendance at training sessions or hearings, examination of State documents other than case files, review of equipment or laboratory facilities, and evaluation of sample analyses.

As discussed in Chapter IV of STP 2.22A, the use of on-site monitoring should not be with the purpose of obtaining routine monitoring data but should be used to obtain information not otherwise available through the IMIS. This may include information on different or unique State plan components.

As you may be aware, for FY 1990 the Congress removed previous appropriations language limiting the use of Spot Check Monitoring Visits along with eliminating several other riders. The deletion of the rider was predicated on OSHA's assurance that the procedures in Chapter IV of STP 2.22A limiting the conduct of spot check visits would remain in effect. Therefore, these visits should still be conducted only on a limited basis as a last resort when other monitoring tools are not sufficient and with National Office approval as set out in Chapter IV of STP 2.22A.

Ascertaining whenever impact studies have been conducted by the States and evaluating and reporting on them in the State's annual evaluation report:

In responding to GAO's recommendation that OSHA work with the States to develop and implement plans to evaluate the impact of their various programs on specific establishments or industry groupings, OSHA expressed its belief that impact studies would be most useful and cost-effective if coordinated on a national basis, but indicated that it is not feasible presently, within our current resource limitations, to undertake such an effort. However, as an interim measure, OSHA indicated to OIG and GAO that we would ask the States to submit any impact studies they have conducted and consider them in the course of preparing annual evaluation reports.

Each Region should immediately request the State plan designees within its jurisdiction to submit any significant State-initiated impact or evaluation study completed within the last two years, particularly those that address program components which differ from the Federal program. For example, such studies might involve the impact of a particular standard or training program. The Region should review all studies submitted, use them as a source of information for evaluating the effectiveness of different or unique State plan components, and report on their findings in the State's annual evaluation report. Subsequently, any State planning such a study should notify the Regional Office in advance, report informally on the progress of the study, and provide a copy of the report when completed. In addition, any study the Region identifies as being of particular interest or as having particular relevance to the administration of the Federal or other State programs should be submitted to the Director of Federal-State Operations for consideration and possible distribution.

Strengthening OSHA's internal audits of regional and area office monitoring activities:

As indicated in the response to GAO and OIG, OSHA will develop an audit protocol which will ensure that Area Office and Regional Office monitoring efforts will be analyzed and evaluated as a whole. In the interim, the OSHA Instruction on Field Audits, PAE 2.1B, is being revised to incorporate specific guidelines to be followed in auditing Area Office State plan monitoring activities. This revision is expected to be issued by the end of January 1990.

Encouraging the innovative use of IMIS reports in the monitoring process:

The innovative use of IMIS reports will also be addressed in the course of the comprehensive review and revision of the monitoring procedures. In the interim, Regional Administrators should review and assess any innovative uses of IMIS reports proposed and/or employed by Federal or State personnel in their regions. Where the Regional Administrator believes that the approach could be utilized by other monitoring staff, a description of the data/reports used and their analysis should be submitted to the Director of Federal-State Operations. The Directorate will establish a repository for such techniques and will establish a mechanism for sharing this information among regions and States, including entry into the OSHA Computerized Information System (OCIS), if appropriate.

Reevaluating SPAM measures to provide more meaningful comparisons with Federal data where appropriate:

In response to OIG's concern, OSHA is seeking an opinion from the Solicitor's Office (SOL) to address whether State public sector performance must be compared to private sector performance or if other standards of comparison are permissible. when the SOL opinion is received, additional guidelines in this area will be considered. (Of course, as part of the review of the monitoring system, other SPAM measures will be reevaluated as well.)

Improving timeliness of evaluation reports:

The OIG study noted that evaluation reports are not always submitted in a timely manner. The time limit for submission of reports will be addressed as part of the overall revision of the monitoring system. In the interim, Regions should ensure that reports are submitted within 60 days of receipt of the last SPAM report of the evaluation period, as set out in STP 2.22A, page VIII-5. As discussed in the separate memorandum on SPAM reports, these reports are now being transmitted electronically, so that data will be available in a more timely manner. However, since the review and analysis of State performance should be ongoing, it is not necessary to wait for the receipt of the SPAM for the report's time period before beginning preparation of the report. As noted in STP 2.22A, page VIII-5, the report need not include a discussion of new outliers and other information appearing in the 6-month data on the last SPAM report of the evaluation period if this analysis could require a delay in preparation of the report.

In addition, some offices apparently have been delaying preparation of evaluation reports until after receipt of the Consultation Activities Measures (CAM) reports for the appropriate period. Past delays in running the CAM reports are not expected to continue once State consultation programs are on-line. However, if the CAM report for the evaluation period is not available, regions should use the most recent available CAM report or data from State records for preparation of the evaluation report, noting any differences in time periods in the report.

Using additional tools for analysis of State performance:

Additional clarifications and guidelines for use of the State Plan Activities Measures Report and other Integrated Management Information System Reports are detailed in my separate memorandum of this date to Regional Administrators (Subject: SPAM Analytical Handbook). That memorandum discusses the use of Micro-to-Host and special Host Reports in SPAM analysis and includes an Analytical Handbook which should provide assistance to you in the analysis of SPAM data.

In addition, as noted in that memorandum, Change 1 to OSHA Instruction ADM 1-1.21A is being issued to require States to utilize the "desired state and other micro reports" as appropriate for their operations for maintenance and verification of their databases and management of their day-to-day operations as is required of Federal Area Offices. (See Chapter I, paragraph E. - Desired State Reports, OSHA Instruction ADM 1-1.21A, CH-1. Also see Objectives 6 and 7 of the May 15, 1987 Directorate of Field Operations memorandum, "Desired State of A.D.P. Implementation by End of FY `88.") Regions should ensure that the States are using these reports as appropriate to their operations to meet these objectives.

To assist you in following the above guidelines, attached is a list summarizing the major points in this memorandum. We appreciate your attention to the interim guidelines and hope that they will help to improve the monitoring process.

If there are any questions, please contact Barbara Bryant, Director, Office of State Programs, on 523-7242.

MEMO OKed BY OIG & GAO.

Attachment



Summary of Interim Guidelines for State Plan Monitoring In Response to OIG/GAO Recommendations

Following up on State corrective action:

- Clearly identify and label recommendations in State's annual evaluation report.

- Ensure that State responds formally within 60 days of receipt of an evaluation report containing recommendations.

- Provide copy of State response to Director of Federal-State Operations.

- Track State performance in areas of recommendations, including special investigations where appropriate.

- Discuss progress and problems in responding to recommendations in the next report.

- Maintain written record of all significant information covered during formal discussions with State.

Assuring that some on-site activity occurs during the course of each State's annual evaluation:

- Conduct some degree of on-site monitoring during the course of each State's annual evaluation period.

- Determine the nature and focus of on-site effort after consideration of all available data and other off-site sources of information.

- Apply on-site procedures principally where information from the IMIS or other readily available data sources is not sufficient to evaluate a particular aspect of State performance, and where necessary to obtain otherwise unavailable information on different or unique State plan components.

- The definition of on-site monitoring is not limited to case file reviews, accompanied visits and spot-check monitoring visits. Other on-site tools include documented discussions or interviews with State officials, attendance at training sessions or hearings, examination of State documents other than case files, review of equipment or laboratory facilities, and evaluation of sample analyses.

Ascertaining whenever impact studies have been conducted by the States and evaluating and reporting on them in the State's annual evaluation report:

- Request States to submit any significant State-initiated impact or evaluation study completed within the last two years, and to notify the Region in advance of any subsequently planned studies and to submit them upon completion.

- Review studies and report on findings in evaluation report. - Submit relevant studies of interest to Federal-State Operations.

Strengthening OSHA's internal audits of regional and area office monitoring activities:

- Revised instruction incorporating specific guidelines for auditing State plan monitoring activities will be issued by January 1990. Later revisions will include specific guidelines for auditing a region's monitoring function as a whole rather than specific area offices.

Encouraging the innovative use of IMIS reports in the monitoring process:

- Review and assess any innovative uses of IMIS reports proposed and/or employed by Federal or State personnel.

- Submit a description of useful approaches to the Directorate of Federal-State Operations.

Reevaluating Public Sector SPAM measures to provide more meaningful comparisons with Federal data where appropriate:

- Solicitor's opinion on public sector comparisons has been requested; specific guidelines may be issued after receipt.

Improving timeliness of evaluation reports:

- Do not delay preparation of evaluation reports pending receipt of consultation data.

Using additional tools for analysis of State performance:

- Use standard and special IMIS reports for monitoring and data analysis.

- Use Analytical Handbook for guidance on data analysis for specific activities measures.

- Ensure that States are using desired state and other micro reports for managing their programs.