Senate Bill 430, passed by the Legislature in 1997, required the Secretary of the Department of Health and Human Resources (DHHR) to establish a methodology for evaluating the effectiveness, efficiency, and fiscal impact of the West Virginia WORKS Program. Furthermore, the bill stated that the Performance Evaluation and Research Division, within the Legislative Auditor's Office, may review and make recommendations with respect to the agency's evaluation methodology to evaluate the impact of the pilot project prior to it being implemented statewide. However, the statewide implementation of the WORKS project began on October 1, 1997, prior to the agency's evaluation of the pilot project. Consequently, the evaluation methodology submitted to the Legislative Auditor's Office is designed to analyze the WORKS program after statewide implementation.
The Legislative Auditor's review and recommendations are limited because DHHR's methodology was not finalized. A draft copy of the methodology is contained in Appendix A. DHHR has federal reporting requirements as well as the State's evaluation mandate. In order to be consistent in state and federal evaluation and reporting, DHHR indicated that "any methodology must be designed to conform to federally prescribed specifications for sampling, data collection, reporting, and priority areas for evaluation." However, Federal specifications are expected to be released sometime in 1998. DHHR indicated that once federal specifications are available, the agency will determine if all sources of data have been identified or will be available. Additional reporting capabilities may need to be developed for current information systems.
It is the Legislative Auditor's opinion that the State mandate does not have to comply with federal specifications because they are two separate requirements. The Legislative Auditor agrees that for the sake of efficiency, procedures established for the State evaluation should not make efforts to comply with federal requirements more difficult or create unnecessary duplication. However, the State mandated evaluation should not be put on hold until the federal government releases its final rules on reporting requirements. There are aspects of the State methodology that can proceed immediately without interfering with meeting federal requirements.
Senate Bill 430 outlines the goals of the WORKS program.
The first is "to achieve more efficient and effective use of public assistance funds."
The second is "to reduce dependency on public programs by promoting self-sufficiency."
The program should be structured "to emphasize employment and personal responsibility."
After five years, there is expected to be a decrease in the number of persons receiving public assistance, and a decrease in the amount of time an individual remains on public assistance.
Finally, the agency's evaluation should include measures for "the increase in employment rates in program areas; the completion of educational and training programs; the increased compliance in preventive health activities, including immunizations; and a decrease in the case-load of division of personnel."
DHHR's Evaluation Methodology is Too Broad and Lacks Details
The Legislative Auditor's review of DHHR's draft evaluation methodology finds that it is broad and lacks details, although conceptually it would comply with the goals of Senate Bill 430. The methodology is divided into four parts consisting of: 1) baseline forecasting, 2) impact evaluation, 3) process study, and 4) cost-benefit analysis. The Department's evaluation design proposes to answer the following five questions:
The variables described in the methodology would attempt to use variables that will be difficult to measure, difficult to develop an adequate time-series, and also would require an enormous amount of data to be collected which would be time consuming, and expensive. Although some of this information may be needed for federal reporting purposes, it is not critical to providing an adequate evaluation of WORKS for the State Legislature in compliance with SB 430. If federal reports provide interesting results, these should be made available to the State Legislature. Overall, the methodology is over-ambitious and needs to be scaled back to answer basic questions such as:
Baseline Data Analysis
The baseline analysis is similar to the approach used by the Legislative Auditor in
his study required by SB 430. A major difference is that DHHR intends to use 1996 data
for the baseline, while the Legislative Auditor's Office used the 1993-1996 period as the
baseline. The baseline data will be collected from 1993 to 1995 and will use 1996 as a
base year for comparison purposes. A forecasting mechanism will be used to examine
trends and determine which variables may have been affected by program-specific
influences in future periods under the WORKS program; The analysis will be conducted
by county. DHHR does not indicate what program variables will be evaluated in the
baseline analysis. This is a serious problem because defining the following variables is a
formidable task: self-sufficiency; Food Stamp and Medicaid participation; family
structure and stability; "well being" of children and long-term "prospects" for
self-sufficiency; application rate and caseload. Furthermore, DHHR's baseline does not
indicate whether and how economic conditions will be accounted for in its analysis.
A cost-benefit analysis is planned by the Department of Health and Human Resources in order to weigh the costs of the program against its benefits. Such factors as administrative costs, transportation costs, Food Stamp payments, and education and training expenses will be examined. This will include an analysis of the impact of the program on communities, day care costs, among other factors. The cost-benefit analysis should be used to obtain a complete picture of the impact that WORKS has, not only on cash assistance payments, but also on other services and the resulting fiscal impact.
DHHR intends to complete a process study to "document the implementation of WORKS and its operations and practices." The study will focus on all aspects of services provided including:
The goal of the process study is to examine any differences in the services provided, the sequence of services offered to clients, and relationships with other agencies. The study will serve as a management tool for allocating resources and to expose aspects of the program requiring management attention.
The agency's evaluation design includes the development of a model to determine the factors leading to successful outcomes. Its major objectives are to establish baseline data and to conduct impact evaluations. The impact evaluations will include, but not be limited to the following factors:
The long-term effectiveness of WORKS should largely be measured by the amount of time individuals remain employed after leaving public assistance. In order to remain employed and become self-sufficient, recipients must be placed in jobs which will keep them from returning to public assistance.
Proposed Federal TANF Regulations
Proposed TANF Regulations are currently available for public comment. They describe the Administration for Children and Families' (ACF) interpretation of the PWORA as it applies to the states. One of the Department of Health and Human Resources' most immediate concerns is meeting federal data reporting requirements. In order to minimize data reporting as required by Section 411 of the PWORA, the ACF proposes to limit reports to a quarterly TANF Data Report, a quarterly TANF Financial Report, an annual program and performance report, and an annual addendum to the fourth quarter Financial Report. Quarterly aggregated caseload data will be required. The ACF also proposes to require States to report only on the demographic and financial characteristics of families who successfully apply for assistance. The proposed disaggregated case data collection requirements include data on both families receiving TANF assistance and families no longer receiving TANF assistance. The ACF proposed regulations have outlined the possible fiscal penalties against States. These penalties include a penalty of four percent of the adjusted State Family Assistance Grant (SFAG) for "failure to submit an accurate, complete and timely required report."
Data must also be maintained on specific services received by the family: subsidized housing, medical assistance, Food Stamps or subsidized child care. The number of months the family spends on each form of assistance is another important data category. The number of hours per week adults participated in various work activities is a key measure required by the federal legislation for the purposes of determining participation rates. These work activities include education, subsidized employment, unsubsidized employment, work experience, job search, job skills training or on-the-job training, and vocational education. A sample of closed cases is to be examined and the reason for case closure is to be determined according to three categories. These are employment, marriage, and the five-year limit on benefit eligibility.
In addition to the data that must be collected and is specified in the federal legislation, the ACF's proposed TANF regulations describes new data elements that must be maintained by States. These include data on clients relating to Zip Codes, family affiliation, Social Security Numbers, and gender. Five data elements related to the administration of a data collection system are proposed, including State FIPS Code, the reporting month, sampling stratum, family case number, and sample case disposition. "Break-outs" of existing data elements are proposed as well. These include data on earned income, unearned income, subsidized housing, and Food Stamp assistance type. Other data elements are proposed because they are necessary to implement the requirements of other statutes. These include cooperation with child support, time limits for receipt of assistance, new applicants, and the amount of a family's cash resources. The ACF also proposes new data elements relating to child care.
Sampling may be used by States to meet disaggregated data collection and
reporting requirements. It may not be used to collect aggregated data. A sampling and
statistical manual will be issued at a later date and will contain instructions on procedures
and specifications for sampling methods.
It is the Legislative Auditor's opinion that the State can immediately begin the process of finalizing an evaluation methodology. The Legislative Auditor agrees that for the sake of efficiency, developing an evaluation methodology should not make efforts to comply with federal requirements more difficult or create unnecessary duplication. However, the State mandated evaluation should not be put on hold until the federal government releases its final rules on reporting requirements. There are aspects of the evaluation that can proceed immediately without interfering with meeting federal requirements. There are clearly certain data that can be identified and must be available to determine the effectiveness of the WORKS program and its fiscal impact. Some variables would include:
The agency's plan assumes that RAPIDS will be sufficient for the data collection requirements of WORKS. Most WORKS cases have recently been converted to the RAPIDS system, which was not originally designed for WORKS, but the system is being upgraded to accommodate WORKS cases. The current data collection methods used by the Department of Health and Human Resources utilize multiple data systems which are in different locations. Employment placement and financial data for different benefit programs are currently maintained separately. This creates difficulties in accessing all the data that are relevant for evaluating the net fiscal impact and effectiveness of the program. The agency's goal is to incorporate data that have been stored in older data systems into RAPIDS. The establishment of one central location for the storage of WORKS-related data would facilitate future analysis. The ability of RAPIDS to collect all of the data required has yet to be developed.
The following recommendations are made.
The Legislative Auditor recommends that DHHR begin the process of identifying the necessary data that will be necessary to evaluate the effectiveness and fiscal impact of the WORKS program. The draft methodology should be scaled back to address the basic issues previously mentioned.
DHHR should submit to the Joint Committee on Government Operations an assessment of the RAPIDS system's ability to track, report, or calculate at least the following information: