WORKS Summary

Senate Bill 430, passed by the Legislature in 1997, required the Secretary of the Department of Health and Human Resources (DHHR) to establish a methodology for evaluating the effectiveness, efficiency, and fiscal impact of the West Virginia WORKS Program. Furthermore, the bill stated that the Performance Evaluation and Research Division, within the Legislative Auditor's Office, may review and make recommendations with respect to the agency's evaluation methodology. This report provides an assessment of DHHR's methodology for evaluating the effectiveness of the State's welfare reform program.

The Legislative Auditor's review and recommendations are limited because DHHR's methodology was not finalized. DHHR indicated that:

So that our efforts will be consistent with a national strategy for evaluating welfare reform, any methodology must be designed to conform to federally prescribed specifications for sampling, data collection, reporting, and priority areas for evaluation. However, the final ruling on reporting requirements will not be forthcoming until 1998. At that time, it will be necessary to determine whether all required sources of both qualitative and quantitative data have been identified and will be available.

Establishing a Final Methodology Should Not Be Delayed

It is the Legislative Auditor's opinion that the State can immediately begin the process of finalizing an evaluation methodology. The Legislative Auditor agrees that for the sake of efficiency, developing an evaluation methodology should not make efforts to comply with federal requirements more difficult or create unnecessary duplication. However, the State mandated evaluation should not be put on hold until the federal government releases its final rules on reporting requirements. There are aspects of the evaluation that can proceed immediately without interfering with meeting federal requirements. There are clearly certain data that can be identified and must be available to determine the effectiveness of the WORKS program and its fiscal impact. Some variables would include:

Furthermore, part of a good evaluation process will require the ability to track welfare recipients and former welfare recipients on an individual basis. This will also be essential management information. Management will need to identify individuals who are approaching the limit for receiving benefits in order to assist them in becoming employed. DHHR needs to be in a
position to answer certain questions not only for evaluation purposes, but also for management control. For example, it will eventually be important for DHHR and the Legislature to know the following information at any point in time:

DHHR's Draft Methodology is Too Broad and Lacks Details

The Legislative Auditor's review of DHHR's draft evaluation methodology finds that it is broad and lacks details, although conceptually it would comply with the goals of Senate Bill 430. The methodology is divided into four parts consisting of: 1) baseline forecasting, 2) impact evaluation, 3) process study, and 4) cost-benefit analysis. The Department's evaluation design proposes to answer the following five questions:

The variables described in the methodology would attempt to use variables that will be difficult to measure and also would require an enormous amount of data to be collected which would be time consuming, and expensive. Although some of this information may be needed for federal reporting purposes, it is not critical to providing an adequate evaluation of WORKS for the State Legislature in compliance with SB 430. If federal reports provide interesting results, these should be made available to the State Legislature. Overall, the methodology is over-ambitious and needs to be scaled back to answer basic questions such as:


Recommendations

The Legislative Auditor recommends that DHHR begin the process of identifying the necessary data that will be necessary to evaluate the effectiveness and fiscal impact of the WORKS program. The methodology should be scaled back to address the basic issues previously mentioned. DHHR should also submit to the legislature an assessment of the RAPIDS system's ability to track, report, or calculate at least the following information:

DHHR's assessment of the RAPIDS system should also include a needs-assessment which should specify any necessary modifications of RAPIDS to produce reports and track the essential information previously discussed regardless of the release of federal regulations.