The Legislative Auditor's review and recommendations are limited because DHHR's methodology was not finalized. DHHR indicated that:
So that our efforts will be consistent with a national strategy for evaluating welfare
reform, any methodology must be designed to conform to federally prescribed
specifications for sampling, data collection, reporting, and priority areas for
evaluation. However, the final ruling on reporting requirements will not be
forthcoming until 1998. At that time, it will be necessary to determine whether all
required sources of both qualitative and quantitative data have been identified and
will be available.
Establishing a Final Methodology Should Not Be Delayed
It is the Legislative Auditor's opinion that the State can immediately begin the process of finalizing an evaluation methodology. The Legislative Auditor agrees that for the sake of efficiency, developing an evaluation methodology should not make efforts to comply with federal requirements more difficult or create unnecessary duplication. However, the State mandated evaluation should not be put on hold until the federal government releases its final rules on reporting requirements. There are aspects of the evaluation that can proceed immediately without interfering with meeting federal requirements. There are clearly certain data that can be identified and must be available to determine the effectiveness of the WORKS program and its fiscal impact. Some variables would include:
Furthermore, part of a good evaluation process will require the ability to track
welfare recipients and former welfare recipients on an individual basis. This will also be
essential management information. Management will need to identify individuals who are
approaching the limit for receiving benefits in order to assist them in becoming employed.
DHHR needs to be in a
position to answer certain questions not only for evaluation purposes, but also for
management control. For example, it will eventually be important for DHHR and the
Legislature to know the following information at any point in time:
DHHR's Draft Methodology is Too Broad and Lacks Details
The Legislative Auditor's review of DHHR's draft evaluation methodology finds
that it is broad and lacks details, although conceptually it would comply with the goals of
Senate Bill 430. The methodology is divided into four parts consisting of: 1) baseline
forecasting, 2) impact evaluation, 3) process study, and 4) cost-benefit analysis. The
Department's evaluation design proposes to answer the following five questions:
The variables described in the methodology would attempt to use variables that will be difficult to measure and also would require an enormous amount of data to be collected which would be time consuming, and expensive. Although some of this information may be needed for federal reporting purposes, it is not critical to providing an adequate evaluation of WORKS for the State Legislature in compliance with SB 430. If federal reports provide interesting results, these should be made available to the State Legislature. Overall, the methodology is over-ambitious and needs to be scaled back to answer basic questions such as:
Recommendations
The Legislative Auditor recommends that DHHR begin the process of identifying the necessary data that will be necessary to evaluate the effectiveness and fiscal impact of the WORKS program. The methodology should be scaled back to address the basic issues previously mentioned. DHHR should also submit to the legislature an assessment of the RAPIDS system's ability to track, report, or calculate at least the following information:
DHHR's assessment of the RAPIDS system should also include a needs-assessment which should specify any necessary modifications of RAPIDS to produce reports and track the essential information previously discussed regardless of the release of federal regulations.