STATE OF WEST VIRGINIA

FULL PERFORMANCE EVALUATION OF THE

Department of Administration

PURCHASING DIVISION

OFFICE OF LEGISLATIVE AUDITOR
Performance Evaluation & Research Division
Building 1, Room 314-W
State Capitol Complex

CHARLESTON, WEST VIRGINIA 25305
(304) 347-4890



January 1999

ISSUE AREA 1: Two out of Nine RFP Contracts Reviewed Were Found to be In Error

The Purchasing Division, within the Department of Administration, is the central purchasing agent for state agencies. During fiscal year 1998, Purchasing awarded 2,189 contracts to vendors for a variety of goods and services. These contracts were awarded using one of three procedures. One procedure is a Request for Quote (RFQ), another is a Request for Proposal (RFP), and the third is a sole-source.

In this issue, the Legislative Auditor's Office examined the RFQ and the RFP process to determine if these processes are timely, competitive, and providing cost savings to the state. The Legislative Auditor found that in general both procedures are timely, competitive, and cost-effective. However, the RFP process has an unnecessarily high risk of awarding a contract to the wrong vendor because of agencies that do not follow the procedure properly, and because the Purchasing Division does not oversee the RFP process adequately.


The RFP Process
According to the Purchasing Manual, RFP's are used to acquire complex, high dollar purchases, which require a high-level of expertise on the part of the vendor. Tangible goods should not be bid using an RFP. The RFQ process is used to acquire tangible property. An estimate from a sample of contracts suggests that of the 2,189 contracts for 1998, approximately 12 were RFP's. However, the dollar value of RFP's can be worth millions of dollars.

The main difference between an RFP and RFQ is that the RFQ is based strictly on lowest bid. Generally, the vendor that matches the agency's specifications and has the lowest quoted price will be awarded a RFQ contract. However, the RFP process is broken down into two parts: 1) technical; and 2) cost. The evaluation of each part is done separately and both parts combine to total 100 points. The evaluation criteria for an RFP must consider cost in the criteria at a minimum of 30% (or 30 points) of the decision. The remaining points are assigned to the technical proposal. The technical proposal is determined by the agency and it must include weighted values for vendor characteristics that the agency feels are important, but greater weight should be placed on those characteristics of greater importance. An example of a RFP evaluation criteria is comprised of the following:

Example of a RFP Evaluation Criteria

Technical Proposal
€ Method of Performance = 25 points
Total Technical Points Possible =70 points

Cost Proposal = 30 points
Total Points = 100 points

The Purchasing Division uses a process called "Minimum Acceptable Score" (MAS) in the evaluation process. According to Purchasing's Standard Format for RFP's, dated 10-1-98,

The MAS process requires that a vendor score a minimum of 70% of the total points possible on the written technical portion of the proposal in order to qualify to continue to the next level of the evaluation process....Any vendor failing to attain the "MAS" would be disqualified and removed from any further consideration in the award process. (The cost proposal would be destroyed unopened).

The objective in using the MAS system is stated by Purchasing as follows:

Use of the "mas" technique insures that quality is maintained in that any vendor who cannot attain a minimum of at least 70% of the total points possible is not qualified to obtain the award at any cost regardless how low.

In the example given above, a vendor would have to receive at least the MAS of 49 (or 70%) of the 70 total possible technical points in order to continue to the next level, which is the opening of its sealed cost proposal for consideration. Anything less than 49 points in the technical evaluation would result in the vendor being disqualified for further consideration (its cost proposal would be destroyed unopened). If the total technical points were 60 and the cost proposal equaled 40 points, the vendor would have to score at least 42 (or 70%) of the 60 total possible technical points in order to have its cost proposal considered.

Each vendor's technical proposal is evaluated independently by a committee. This committee is comprised of a minimum of three and a maximum of seven qualified and knowledgeable members who are willing to participate. Any state government employee who meets the qualifications is eligible to represent an evaluation committee. Purchasing approves the one individual who is selected as chairperson. This individual will have extensive experience or will have received training in the evaluation process of RFP's from the Purchasing Division. The chairperson is also a voting committee member who is responsible for all communication between the committee and the Buyer in the Purchasing Division.

The chairperson will calculate the cost score of all vendors who obtained the MAS by giving the maximum points possible (usually 30 or 40 points) to the lowest bidder, and will provide a ratio score to any other qualifying bidder based on the following formula:

(Low Bid / Bid Being Evaluated) * Weighted Cost Score = Raw Cost Score

Finally, the chairperson adds the raw cost score to the technical score. The vendor with the highest total points is the apparent successful vendor. The chairperson prepares the final consensus evaluation document along with a written justification for reduction of technical points from each vendor under consideration. All committee members of record must sign the final evaluation.


Analysis of RFP Contracts

The Legislative Auditor's Office conducted a sample of contracts with a value of over $100,000. There were 799 such contracts in fiscal year 1998. The sample size was 67 randomly selected contracts awarded during FY 1998. These contracts were comprised of RFP's, RFQ's, and sole source purchases. The random selection resulted in only one RFP selected. This suggests that Purchasing handled an estimated 12 RFP's for FY98, assuming the sample is representative of the total population of contracts.

In order to have a larger number of RFP's to review, the Legislative Auditor's Office requested five additional RFP contracts. Purchasing provided the Legislative Auditor with six. In addition, the Legislative Auditor performed a pre-sample test review of two RFP's, which brings the total number of RFP's reviewed to nine.


RFP's had Proper Documentation, were Competitive and Timely

The Legislative Auditor's Office found each RFP to have a Bid Award Justification Sheet that explained what deficiencies were present in each vendor's technical proposal that lead to the committee's consensus on technical points for each vendor. The Bid Award Justification Sheets were also signed by each member of the evaluation committee. The evaluation process appears fair if the committee documents the negative comments which led to the consensus scores for each vendor. The Legislative Auditor found adequate documentation on how the consensus scores were determined.

The median number of vendors for RFP's was three. The number of vendors bidding for a contract is important in terms of achieving cost savings. The correlation between the number of vendors and the difference between the high and low bid as a percentage of the low bid was 0.96. This is a high correlation and it indicates that the more vendors bidding the greater the difference in cost proposals. The vendor with the lowest bid will not always be selected for a RFP contract, however, with more vendors bidding, the probability of being limited to selecting a vendor with the highest bid is decreased.

The median number of days it takes to award a RFP contract was 94, or roughly three months. The steps involved in awarding a RFP contract includes announcing the contracts, receiving bids, pre-bid conferences, evaluation of bid. Considering the time involved from start to finish, the Legislative Auditor find this to be timely.


RFP's have Unnecessarily High Risk of Making Incorrect Award

Of the nine RFP's reviewed, the Legislative Auditor found two (one for FY97 and one for FY98) that violated proper procedure. These violations created the possibility of the wrong vendor being awarded a contract. If 1 out of 7 RFP's for 1998 were found to be evaluated incorrectly, this suggests that 2 RFP contracts per year are at-risk of being awarded to the wrong vendor.

Finding two RFP's evaluated incorrectly out of nine is unacceptably high, simply because there should be no risk at all of any RFP being awarded incorrectly. This assertion is based on three factors, one is that Purchasing is required to review the evaluation committee's final evaluation document according to its procedure, second, the calculations involve basic arithmetic, and third, the number of RFP's to review is small. Given these three factors, there is no reason why any errors should occur. Below is a description of the two RFP's that were evaluated incorrectly.


Case #1 - Vendor Incorrectly Disqualified

In the first RFP that was incorrectly evaluated, the Legislative Auditor discovered that a vendor was disqualified after the technical evaluation when it should not have been. Consequently, the vendor's cost proposal was not considered. The technical portion was worth 60 points in this case and the cost portion was worth 40 points. Therefore, a vendor had to have a MAS of at least 70% of 60 points or 42 points to qualify for its cost proposal to be considered. This particular vendor scored 45 points on the technical portion and should have been given further consideration after the technical portion, but was not.

The Legislative Auditor received permission to open the sealed bid of this vendor and compared it with the cost proposal of the only vendor who received further consideration. After reviewing the cost proposals, it was determined that the vendor who was awarded the contract would have still been awarded the contract because it had a large lead in technical points, making it difficult for the other vendor to make up the difference in the cost proposal. The winning vendor received all 60 technical points and it also had an in-state residence preference.

Although the correct vendor was awarded the contract, the incorrect disqualification created the possibility of an incorrect award. The disqualified vendor could have had a cost proposal low enough to be awarded the contract. The incorrect disqualification created the risk of the state entering into a contract that was more expensive than it needed to pay, and awarding the contract to the wrong vendor.


Case #2 - Vendors should have been Disqualified but were not

The second RFP contract that was found to be inaccurately evaluated involved vendors who were given further consideration after the technical portion but should not have been. The technical portion was worth 60 points and the cost portion was worth 40 points. Again, a vendor had to have a score of at least 42 points or 70% of the 60 technical points. Three vendors were bidding for the contract. One vendor had a MAS of 57, another vendor scored 38, and the third vendor scored a 30. Obviously, only the vendor with the MAS of 57 should have had its cost proposal considered. However, the sealed cost proposals of all three vendors were opened for evaluation by the evaluation committee.

After cost proposals were evaluated and added to the technical points, the vendor with the MAS of 57 was awarded the contract with 93 total points. The vendor with the MAS of 38 made up some ground because it had the lowest bid, however, it was not enough. It ended up with 78 total points. Again, the risk of awarding the contract to the wrong vendor existed. Either of the two vendors with technical scores of 30 and 38 could have had costs low enough to be awarded the contract. Besides the potential of awarding the contract to the wrong vendor, the other potential effect in this cases would have been awarding the contract to a vendor that was unqualified according to the agency's technical criteria. This indicates that the state potentially could have contracted with a vendor that might have performed unsatisfactory.

The purpose for the MAS is to insure a quality vendor is selected. If for some reason costs are considered for vendors who do not qualify after the technical evaluation, the basic principle of insuring quality vendors is violated, and becomes a meaningless exercise. This is magnified when the vendors in this case were as much as 27 points behind after the technical evaluation. If vendors are allowed to make up ground on the cost evaluation, then the technical evaluation does not serve its purpose as stated in the Standard Evaluation Process.

Reasons for Incorrect Evaluations

There are three reasons that RFP procedures are not being followed on a consistent basis. First, some evaluation committees are not following the RFP procedures properly. It could be that the chairpersons of the committees are not knowledgeable enough in the process. In the first RFP evaluated incorrectly, it appears that the chairperson used the MAS for a 70-30 ratio when it should have used the MAS for a 60-40 ratio. This could lead to a vendor being disqualified incorrectly, which did happen. This indicates a need for Purchasing to ensure the qualifications of evaluation committee members, and to improve its training in the RFP process.

Secondly, the RFP procedures should be reviewed for its clarity, and it should be determined if agencies have current procedures. The Legislative Auditor has copies of two Purchasing Manuals, one was dated May of 1996 and the other went into effect July 1, 1998. Within these two manuals there is an explanation of the RFP process. Both state that cost shall represent at least 30% of the total award criteria. However, there is a section in the May of 1996 manual that also states cost should be 40% or more of the total award criteria. The 40% figure is not mentioned in the July of 1998 manual. This inconsistency can lead to state agencies becoming confused and uncertain of exactly what the RFP procedures are.

Finally, the third reason for the risk of making an incorrect award is that Purchasing is not adequately reviewing the final consensus evaluation document for accuracy. After the chairperson prepares the final consensus evaluation document along with the written justification for reduction of technical points, the Buyer reviews the scoring and justification. According to the Standard Evaluation Process, the Buyer is responsible for the following three things:

) The Buyer will review the justification narrative to insure that the points deducted have been properly documented;
) The Buyer will review the evaluation to determine if any of the vendors were stated to not have attained the MAS, and if proper justification is attached to disqualify the vendor; and
) The Buyer will verify the cost calculations by the committee for accuracy.

In the two RFP contracts in which inaccuracies were found, it is apparent that the Buyer did not adequately review the final consensus evaluation documents.

The RFQ Process

A Request for Quotation (RFQ) is used by the Purchasing Division to acquire all tangible goods such as equipment and supplies. RFQ's include a description of the item being purchased, the bid price of the unit or units being sought, delivery date, maintenance and quantity. The lowest responsible bidder who meets the required specifications is usually granted the contract. In cases where there were multiple bidders and the lowest bidder was not chosen, the Purchasing Division is required to provide a Bid Award Justification Letter. In the Legislative Auditor's sample of 67 contracts, 62 were RFQ's for FY 1998. Obviously, most contracts awarded are via RFQ's.

RFQ's are Competitively Bid Resulting in Savings to the State

The Legislative Auditor found that the spread between the high and low bids of the RFQ's in the sample was $1.7 million. This is a potential cost savings. Since some costs are based on costs per unit, a better reflection of the potential cost savings is the spread between the high and low bid as a percentage of the low bid. In the sample, the median percentage was 57%. This represents a potential savings of $57,000 per $100,000 contract.

A correlation analysis of the RFQ contracts in the sample with specified dollar amounts shows a positive and relatively high correlation of 0.677. This means that the more bidders there are for a contract, the larger the spread between high and low bids, resulting in savings to the state. In 90% of the cases, the low bidder is awarded the contract.

Having a large number of bidders per contract is important to increase the probability of receiving the lowest price. The sample of RFQ contracts showed a median of six bidders per contract. The correlation analysis shows that the smaller the number of vendors, the smaller the spread between the high and low bid. Purchasing's method of registering and notifying vendors of contracts, contributes to achieving a large enough number of bidders per contract to save the state millions.

Vendor Preference Does not Affect Many Decisions

In cases where the contract price being bid on is in excess of $50,000, resident vendors who meet the following conditions can be awarded vendor preference.

€An individual resident vendor who has resided in West Virginia continuously for four years prior to the bid being submitted.

€For the purpose of producing or distributing commodities or completing a project, an average of 75% of the vendor's employees are residents of West Virginia and will continue to be so for the duration of the project. These employees must have resided in West Virginia for at least two years prior to the bid being submitted.

By meeting one or both of these criteria, a vendor can be awarded a maximum of 5% preference points (2.5% per). If the bid with vendor preference exceeds the bid of the lowest, non preferred bidder by less than 5% (2.5% for meeting one criteria), then the vendor with the preference will be awarded the contract.

A maximum of 2.5% vendor preference points can also be awarded to nonresident vendors. In order to qualify, they must employ a minimum of 100 state residents, or if the nonresident vendor has an affiliate or subsidiary which maintains its headquarters or principle place of business within West Virginia and which employs a minimum of 100 residents. At least 75% of the vendor's employees or the vendor's affiliates or subsidiary's employees must have resided in West Virginia continuously for two years immediately preceding the bid.

It is not often that a vendor receives a contract based on the resident vendor preference. The Executive Director of Purchasing indicated that in FY 1996, only 4 West Virginia vendors received the resident preference out of a total of 2,100 contracts. This is due to the fact that most in-state vendors are competing against other in-state vendors, and that when there is competition with out-of-state vendors, the preference ratio is not large enough to make a difference. Our sample of 62 RFQ's showed that more than 45% of the contracts were bid on by West Virginia vendors only (See Table 1). Because it was in-state versus in-state vendors, vendor preference would have no relevance in keeping the contract in the state.


Table 1
Bidders per Contract by Location
Number of BiddersPercentage of Total Bidders
In-State Only2345.10%
Out-of-State Only611.76%
Both in and Out of State2243.14%
Total*51
*Total is less than sample number because of multiple bidders for one contract


Purchasing Processes Bids Timely

State law requires that all vendors must register with the Purchasing Division. Registered vendors receive a West Virginia Purchasing Bulletin through the mail bimonthly. This bulletin lists all contracts that will be bid on and the bid opening date. The Legislative Auditor found that in the sample of RFQ's, it took an average of 59 days for Purchasing to determine who would be awarded the contract. This was from the date that the bids were officially opened until the contract was awarded. Purchasing is able to process all RFQ bids, which includes making sure that specifications are met, notify bidders, and awarding the contracts within two months. Considering the amount of work that must go into each contract, from start to finish, this appears to be a satisfactory time frame.

Conclusion

Overall, the Purchasing Division has achieved a competitive and timely procurement system for the state. Documentation of the RFP process is adequate. However, the RFP process has an unnecessarily high risk of making an incorrect award. Given the relatively simple procedure, and Purchasing's requirement to oversee the evaluations of these contract awards, there should be no risk at all of awarding a contract to the wrong vendor. The potential effects of this can be costly to the state and to vendors.

Recommendation 1
Purchasing should write the RFP procedures in a clear and concise manner that enables state agencies to easily follow them, and Purchasing should improve its training to ensure state agencies follow the RFP procedures.

Recommendation 2

Purchasing should improve its review of the final consensus evaluation document of each apparent successful vendor of an RFP contract to ensure accuracy and completeness.

ISSUE AREA 2: Purchasing Division Should Improve the Collection of Certain Management Information

Certain management information is not collected by the Purchasing Division that is useful to improve the procurement process, and identify areas that may need improvement. There is a significant amount of purchases that do not go through Purchasing. This suggests a need to ensure that state agencies are complying with proper purchasing procedures to ensure purchases are competitive, ethical, and economical.

Purchasing Does Not Perform Quality Assistance Reviews on a Regular Basis

According to the Purchasing's Agency Purchasing Manual,

Periodic visits will be made by Purchasing Division representatives to review purchasing activities, the delivery of commodities and services, and records of the agencies. They will assist and advise state agencies in matters pertaining to the rules and regulations, policies and procedures of the Purchasing Division. Agency personnel should always take the opportunity to offer suggestions and request any other assistance that will assure a more efficient and effective purchasing system.

However, this is difficult to accomplish when Quality Assistance (QA) reviews are not taking place on a formal basis, just on an as needed basis. The Legislative Auditor requested from Purchasing the outcomes of Quality Assistance reviews for the past three fiscal years. No QA reports were provided.

State agencies are allowed to make purchases without going through the Purchasing Division, if the purchases are under $10,000. Purchasing requires that even in these cases, the agency is required to ensure competitive purchases. Purchases between $1,001 and $5,000 require a minimum of three verbal bids when possible. The agency is required to document and record all bids for public record on forms prescribed by Purchasing. Awards can only be made to vendors who are registered with the Purchasing Division. Agency purchases between $5,001 to $10,000 require at least three written bids on forms prescribed by Purchasing.

Having QA reviews performed on a regular basis could possibly indicate whether or not agencies are conforming to proper purchasing procedures. For example, these reviews could accomplish at least three things:

) indicate if agencies are by-passing purchasing requirements by stating there is only one source for a product, when really there is not;
) indicate if agencies are splitting total orders so that the individual amounts are below the $10,000 limit in order for agencies to avoid going through the Purchasing Division; and,
) indicate if agency purchases are competitive and ethical as required by Purchasing.

There are instances in which state agencies make purchases exceeding $10,000 that Purchasing does not know who will be awarded the contract. For example, according to a Purchasing Buyer, many of the Division of Highways maintenance contracts are multiple award contracts. This means that all vendors who submit a bid are awarded a pricing contract. However, the low bidder for an individual order, placed against these contracts, is determined by the Division of Highways at the time the order is placed. If the low bidder has prior commitment at that time, the next lowest bidder is then selected. The sums of money in these contracts can be sizable. These types of occurrences warrant some oversight by Purchasing to ensure that proper purchasing procedures are being followed, and nothing unethical is taking place. The same holds true for purchases under $10,000.

The Executive Director indicated that QA reviews are not conducted on a formal level because Purchasing does not have an appropriate number of staff. He also indicated that the basis for QA inspections is found in West Virginia Code §5A-3-9 which states, "Within the limit of funds available, the director, or some person appointed by the director, shall determine whether commodities delivered or services performed conform to contractual requirements...."

Most Vendor Performance Complaints are Verbal

According to the Purchasing Manual, Section 4.8 on Vendor Performance, it states:

Vendor performance and product quality are crucial in the state purchasing process. When these factors do not meet expectations, agencies are encouraged to contact the vendor and voice their concerns. Complaints should be resolved expeditiously and courteously, preferably by the state agency. If the complaint cannot be resolved at the agency level, it should immediately be referred to the Purchasing Division. The Vendor Performance Form, WV-82 is to be used to report all incidents of vendor performance even if a resolution has been reached. When assistance is requested, a Purchasing Division official will notify the vendor directly in writing of the problem.

According to the Executive Director of Purchasing, "After the problem is resolved, a copy of the documentation is kept in the specific purchase order file for future reference." The Director does not monitor vendor performance complaints, but he indicated that written vendor performance reports occur in less than 1% of all transactions. However, there are many more verbal complaints by state agencies to Purchasing Buyers or to the Executive Director.

Since Purchasing does not monitor formal or verbal vendor performance complaints, it is not known the extent of poor performance by vendors. There is a benefit in requiring agencies to submit vendor performance forms on good or bad performance on contracts with a value greater than a certain amount. If vendors know ahead of time that their performance will come under a certain amount of scrutiny, then these vendors would be more apt to deliver their goods and/or services in a timely and efficient manner. Furthermore, if Purchasing monitored these performance evaluations of vendors, it could use the information to identify vendors with frequent complaints against them. Overall, this process would improve vendor performance and reduce contracting with vendors with a history of poor performance. Therefore, the Legislative Auditor contends that Purchasing should require state agencies to submit vendor performance forms for transactions with vendors, good experiences or bad.

Purchasing Does not Keep Records of Vendor Protests

Vendors can file a protest to Purchasing if they feel that they have been unfairly disqualified or denied a contract. This can occur in the RFP and RFQ process. For example, if a vendor with the lowest bid for a RFQ contract was not awarded a contract because it did not meet the specifications, the vendor could file a complaint opposing the decision. All protests slow up the purchasing process because Purchasing cannot proceed until the protest has been settled. Agency will then be delayed in receiving their goods or services. RFP contracts have been known to be delayed by more than a year until a court decision is made on the complaint.

There are basically three different levels of protests that a vendor can pursue. A 1st level protest goes to the Director of Acquisition and Contract Administration. If the protest is not settled at this level, then a vendor can take it to the 2nd level, which is forwarded to the Executive Director of Purchasing. If the vendor is not satisfied at the 2nd level, then the protest can be taken to circuit court. Of all the contracts that are awarded each year, the Executive Director estimates that approximately less than 1% result in any type of protest. Most of the protests do not reach the 2nd level, with approximately only four protests a year making it that far. According to the Executive Director, only two or three protests a year make it to circuit court.

Currently, Purchasing does not keep records of the number of protests that reach the 1st, 2nd, or court level. Keeping a complete and accurate record of all protests can be a good management tool used to gauge the performance of the procurement system. For instance, if records of protests were kept, then Purchasing could monitor whether or not there was an increase in the number of protests from year to year. If there happened to be an increase in 1st and/or 2nd level protests, then this could indicate that the purchasing process is breaking down at both the agency level and the Purchasing Division level, and the appropriate measures could be taken to alleviate the problem. Therefore, Purchasing should consider monitoring all levels of vendor protests.

Recommendation 3

The Purchasing Division should keep written records of all vendor protests that occur on the 1st, 2nd, and circuit court levels.

Recommendation 4

The Purchasing Division should consider rule changes that require all state agencies to submit a prescribed vendor performance form for transactions of a certain dollar value for good or bad performance. Purchasing should also monitor vendor performance forms as a means to possibly deny contracts to vendors with frequent complaints against them.
Recommendation 5

The Purchasing Division should conduct Quality Assistance reviews as stated in the Purchasing Manual.