In this issue, the Legislative Auditor's Office examined the RFQ and the RFP process to determine if these processes are timely, competitive, and providing cost savings to the state. The Legislative Auditor found that in general both procedures are timely, competitive, and cost-effective. However, the RFP process has an unnecessarily high risk of awarding a contract to the wrong vendor because of agencies that do not follow the procedure properly, and because the Purchasing Division does not oversee the RFP process adequately.
The RFP Process
According to the Purchasing Manual, RFP's are used to acquire complex, high dollar
purchases, which require a high-level of expertise on the part of the vendor. Tangible goods
should not be bid using an RFP. The RFQ process is used to acquire tangible property. An
estimate from a sample of contracts suggests that of the 2,189 contracts for 1998, approximately
12 were RFP's. However, the dollar value of RFP's can be worth millions of dollars.
The main difference between an RFP and RFQ is that the RFQ is based strictly on lowest bid. Generally, the vendor that matches the agency's specifications and has the lowest quoted price will be awarded a RFQ contract. However, the RFP process is broken down into two parts: 1) technical; and 2) cost. The evaluation of each part is done separately and both parts combine to total 100 points. The evaluation criteria for an RFP must consider cost in the criteria at a minimum of 30% (or 30 points) of the decision. The remaining points are assigned to the technical proposal. The technical proposal is determined by the agency and it must include weighted values for vendor characteristics that the agency feels are important, but greater weight should be placed on those characteristics of greater importance. An example of a RFP evaluation criteria is comprised of the following:
The Purchasing Division uses a process called "Minimum Acceptable Score" (MAS) in the evaluation process. According to Purchasing's Standard Format for RFP's, dated 10-1-98,
The MAS process requires that a vendor score a minimum of 70% of the total
points possible on the written technical portion of the proposal in order to qualify to
continue to the next level of the evaluation process....Any vendor failing to attain
the "MAS" would be disqualified and removed from any further consideration in
the award process. (The cost proposal would be destroyed unopened).
The objective in using the MAS system is stated by Purchasing as follows:
Use of the "mas" technique insures that quality is maintained in that any vendor
who cannot attain a minimum of at least 70% of the total points possible is not
qualified to obtain the award at any cost regardless how low.
In the example given above, a vendor would have to receive at least the MAS of 49 (or 70%) of the 70 total possible technical points in order to continue to the next level, which is the opening of its sealed cost proposal for consideration. Anything less than 49 points in the technical evaluation would result in the vendor being disqualified for further consideration (its cost proposal would be destroyed unopened). If the total technical points were 60 and the cost proposal equaled 40 points, the vendor would have to score at least 42 (or 70%) of the 60 total possible technical points in order to have its cost proposal considered.
Each vendor's technical proposal is evaluated independently by a committee. This committee is comprised of a minimum of three and a maximum of seven qualified and knowledgeable members who are willing to participate. Any state government employee who meets the qualifications is eligible to represent an evaluation committee. Purchasing approves the one individual who is selected as chairperson. This individual will have extensive experience or will have received training in the evaluation process of RFP's from the Purchasing Division. The chairperson is also a voting committee member who is responsible for all communication between the committee and the Buyer in the Purchasing Division.
The chairperson will calculate the cost score of all vendors who obtained the MAS by giving the maximum points possible (usually 30 or 40 points) to the lowest bidder, and will provide a ratio score to any other qualifying bidder based on the following formula:
(Low Bid / Bid Being Evaluated) * Weighted Cost Score = Raw
Cost Score
Finally, the chairperson adds the raw cost score to the technical score. The vendor with the highest total points is the apparent successful vendor. The chairperson prepares the final consensus evaluation document along with a written justification for reduction of technical points from each vendor under consideration. All committee members of record must sign the final evaluation.
Analysis of RFP Contracts
The Legislative Auditor's Office conducted a sample of contracts with a value of over $100,000. There were 799 such contracts in fiscal year 1998. The sample size was 67 randomly selected contracts awarded during FY 1998. These contracts were comprised of RFP's, RFQ's, and sole source purchases. The random selection resulted in only one RFP selected. This suggests that Purchasing handled an estimated 12 RFP's for FY98, assuming the sample is representative of the total population of contracts.
In order to have a larger number of RFP's to review, the Legislative Auditor's Office requested five additional RFP contracts. Purchasing provided the Legislative Auditor with six. In addition, the Legislative Auditor performed a pre-sample test review of two RFP's, which brings the total number of RFP's reviewed to nine.
RFP's had Proper Documentation, were Competitive and Timely
The Legislative Auditor's Office found each RFP to have a Bid Award Justification Sheet that explained what deficiencies were present in each vendor's technical proposal that lead to the committee's consensus on technical points for each vendor. The Bid Award Justification Sheets were also signed by each member of the evaluation committee. The evaluation process appears fair if the committee documents the negative comments which led to the consensus scores for each vendor. The Legislative Auditor found adequate documentation on how the consensus scores were determined.
The median number of vendors for RFP's was three. The number of vendors bidding for a contract is important in terms of achieving cost savings. The correlation between the number of vendors and the difference between the high and low bid as a percentage of the low bid was 0.96. This is a high correlation and it indicates that the more vendors bidding the greater the difference in cost proposals. The vendor with the lowest bid will not always be selected for a RFP contract, however, with more vendors bidding, the probability of being limited to selecting a vendor with the highest bid is decreased.
The median number of days it takes to award a RFP contract was 94, or roughly three months. The steps involved in awarding a RFP contract includes announcing the contracts, receiving bids, pre-bid conferences, evaluation of bid. Considering the time involved from start to finish, the Legislative Auditor find this to be timely.
RFP's have Unnecessarily High Risk of Making Incorrect Award
Of the nine RFP's reviewed, the Legislative Auditor found two (one for FY97 and one for FY98) that violated proper procedure. These violations created the possibility of the wrong vendor being awarded a contract. If 1 out of 7 RFP's for 1998 were found to be evaluated incorrectly, this suggests that 2 RFP contracts per year are at-risk of being awarded to the wrong vendor.
Finding two RFP's evaluated incorrectly out of nine is unacceptably high, simply because there should be no risk at all of any RFP being awarded incorrectly. This assertion is based on three factors, one is that Purchasing is required to review the evaluation committee's final evaluation document according to its procedure, second, the calculations involve basic arithmetic, and third, the number of RFP's to review is small. Given these three factors, there is no reason why any errors should occur. Below is a description of the two RFP's that were evaluated incorrectly.
Case #1 - Vendor Incorrectly Disqualified
In the first RFP that was incorrectly evaluated, the Legislative Auditor discovered that a vendor was disqualified after the technical evaluation when it should not have been. Consequently, the vendor's cost proposal was not considered. The technical portion was worth 60 points in this case and the cost portion was worth 40 points. Therefore, a vendor had to have a MAS of at least 70% of 60 points or 42 points to qualify for its cost proposal to be considered. This particular vendor scored 45 points on the technical portion and should have been given further consideration after the technical portion, but was not.
The Legislative Auditor received permission to open the sealed bid of this vendor and compared it with the cost proposal of the only vendor who received further consideration. After reviewing the cost proposals, it was determined that the vendor who was awarded the contract would have still been awarded the contract because it had a large lead in technical points, making it difficult for the other vendor to make up the difference in the cost proposal. The winning vendor received all 60 technical points and it also had an in-state residence preference.
Although the correct vendor was awarded the contract, the incorrect disqualification created the possibility of an incorrect award. The disqualified vendor could have had a cost proposal low enough to be awarded the contract. The incorrect disqualification created the risk of the state entering into a contract that was more expensive than it needed to pay, and awarding the contract to the wrong vendor.
Case #2 - Vendors should have been Disqualified but were not
The second RFP contract that was found to be inaccurately evaluated involved vendors who were given further consideration after the technical portion but should not have been. The technical portion was worth 60 points and the cost portion was worth 40 points. Again, a vendor had to have a score of at least 42 points or 70% of the 60 technical points. Three vendors were bidding for the contract. One vendor had a MAS of 57, another vendor scored 38, and the third vendor scored a 30. Obviously, only the vendor with the MAS of 57 should have had its cost proposal considered. However, the sealed cost proposals of all three vendors were opened for evaluation by the evaluation committee.
After cost proposals were evaluated and added to the technical points, the vendor with the MAS of 57 was awarded the contract with 93 total points. The vendor with the MAS of 38 made up some ground because it had the lowest bid, however, it was not enough. It ended up with 78 total points. Again, the risk of awarding the contract to the wrong vendor existed. Either of the two vendors with technical scores of 30 and 38 could have had costs low enough to be awarded the contract. Besides the potential of awarding the contract to the wrong vendor, the other potential effect in this cases would have been awarding the contract to a vendor that was unqualified according to the agency's technical criteria. This indicates that the state potentially could have contracted with a vendor that might have performed unsatisfactory.
The purpose for the MAS is to insure a quality vendor is selected. If for some reason costs are considered for vendors who do not qualify after the technical evaluation, the basic principle of insuring quality vendors is violated, and becomes a meaningless exercise. This is magnified when the vendors in this case were as much as 27 points behind after the technical evaluation. If vendors are allowed to make up ground on the cost evaluation, then the technical evaluation does not serve its purpose as stated in the Standard Evaluation Process.
Reasons for Incorrect Evaluations
There are three reasons that RFP procedures are not being followed on a consistent basis.
First, some evaluation committees are not following the RFP procedures properly. It could be that
the chairpersons of the committees are not knowledgeable enough in the process. In the first RFP
evaluated incorrectly, it appears that the chairperson used the MAS for a 70-30 ratio when it should
have used the MAS for a 60-40 ratio. This could lead to a vendor being disqualified incorrectly,
which did happen. This indicates a need for Purchasing to ensure the qualifications of evaluation
committee members, and to improve its training in the RFP process.
Secondly, the RFP procedures should be reviewed for its clarity, and it should be determined if agencies have current procedures. The Legislative Auditor has copies of two Purchasing Manuals, one was dated May of 1996 and the other went into effect July 1, 1998. Within these two manuals there is an explanation of the RFP process. Both state that cost shall represent at least 30% of the total award criteria. However, there is a section in the May of 1996 manual that also states cost should be 40% or more of the total award criteria. The 40% figure is not mentioned in the July of 1998 manual. This inconsistency can lead to state agencies becoming confused and uncertain of exactly what the RFP procedures are.
Finally, the third reason for the risk of making an incorrect award is that Purchasing is not adequately reviewing the final consensus evaluation document for accuracy. After the chairperson prepares the final consensus evaluation document along with the written justification for reduction of technical points, the Buyer reviews the scoring and justification. According to the Standard Evaluation Process, the Buyer is responsible for the following three things:
) The Buyer will review the justification narrative to insure that the points deducted
have been properly documented;
) The Buyer will review the evaluation to determine if any of the vendors
were stated to not have attained the MAS, and if proper justification is attached to
disqualify the vendor; and
) The Buyer will verify the cost calculations by the committee for accuracy.
In the two RFP contracts in which inaccuracies were found, it is apparent that the Buyer did not
adequately review the final consensus evaluation documents.
The RFQ Process
A Request for Quotation (RFQ) is used by the Purchasing Division to acquire all tangible
goods such as equipment and supplies. RFQ's include a description of the item being purchased,
the bid price of the unit or units being sought, delivery date, maintenance and quantity. The lowest
responsible bidder who meets the required specifications is usually granted the contract. In cases
where there were multiple bidders and the lowest bidder was not chosen, the Purchasing Division
is required to provide a Bid Award Justification Letter. In the Legislative Auditor's sample of 67
contracts, 62 were RFQ's for FY 1998. Obviously, most contracts awarded are via RFQ's.
RFQ's are Competitively Bid Resulting in Savings to the State
The Legislative Auditor found that the spread between the high and low bids of the RFQ's in the sample was $1.7 million. This is a potential cost savings. Since some costs are based on costs per unit, a better reflection of the potential cost savings is the spread between the high and low bid as a percentage of the low bid. In the sample, the median percentage was 57%. This represents a potential savings of $57,000 per $100,000 contract.
A correlation analysis of the RFQ contracts in the sample with specified dollar amounts shows a positive and relatively high correlation of 0.677. This means that the more bidders there are for a contract, the larger the spread between high and low bids, resulting in savings to the state. In 90% of the cases, the low bidder is awarded the contract.
Having a large number of bidders per contract is important to increase the probability of receiving the lowest price. The sample of RFQ contracts showed a median of six bidders per contract. The correlation analysis shows that the smaller the number of vendors, the smaller the spread between the high and low bid. Purchasing's method of registering and notifying vendors of contracts, contributes to achieving a large enough number of bidders per contract to save the state millions.
Vendor Preference Does not Affect Many Decisions
In cases where the contract price being bid on is in excess of $50,000, resident vendors who meet the following conditions can be awarded vendor preference.
An individual resident vendor who has resided in West Virginia continuously for
four years prior to the bid being submitted.
For the purpose of producing or distributing commodities or completing a project,
an average of 75% of the vendor's employees are residents of West Virginia and
will continue to be so for the duration of the project. These employees must have
resided in West Virginia for at least two years prior to the bid being submitted.
By meeting one or both of these criteria, a vendor can be awarded a maximum of 5%
preference points (2.5% per). If the bid with vendor preference exceeds the bid of the lowest, non
preferred bidder by less than 5% (2.5% for meeting one criteria), then the vendor with the
preference will be awarded the contract.
A maximum of 2.5% vendor preference points can also be awarded to nonresident
vendors. In order to qualify, they must employ a minimum of 100 state residents, or if the
nonresident vendor has an affiliate or subsidiary which maintains its headquarters or principle place
of business within West Virginia and which employs a minimum of 100 residents. At least 75% of
the vendor's employees or the vendor's affiliates or subsidiary's employees must have resided in
West Virginia continuously for two years immediately preceding the bid.
It is not often that a vendor receives a contract based on the resident vendor preference. The Executive Director of Purchasing indicated that in FY 1996, only 4 West Virginia vendors received the resident preference out of a total of 2,100 contracts. This is due to the fact that most in-state vendors are competing against other in-state vendors, and that when there is competition with out-of-state vendors, the preference ratio is not large enough to make a difference. Our sample of 62 RFQ's showed that more than 45% of the contracts were bid on by West Virginia vendors only (See Table 1). Because it was in-state versus in-state vendors, vendor preference would have no relevance in keeping the contract in the state.
Table 1 Bidders per Contract by Location | ||
Number of Bidders | Percentage of Total Bidders | |
In-State Only | 23 | 45.10% |
Out-of-State Only | 6 | 11.76% |
Both in and Out of State | 22 | 43.14% |
Total* | 51 |
Purchasing Processes Bids Timely
State law requires that all vendors must register with the Purchasing Division. Registered vendors receive a West Virginia Purchasing Bulletin through the mail bimonthly. This bulletin lists all contracts that will be bid on and the bid opening date. The Legislative Auditor found that in the sample of RFQ's, it took an average of 59 days for Purchasing to determine who would be awarded the contract. This was from the date that the bids were officially opened until the contract was awarded. Purchasing is able to process all RFQ bids, which includes making sure that specifications are met, notify bidders, and awarding the contracts within two months. Considering the amount of work that must go into each contract, from start to finish, this appears to be a satisfactory time frame.
Conclusion
Overall, the Purchasing Division has achieved a competitive and timely procurement system for the state. Documentation of the RFP process is adequate. However, the RFP process has an unnecessarily high risk of making an incorrect award. Given the relatively simple procedure, and Purchasing's requirement to oversee the evaluations of these contract awards, there should be no risk at all of awarding a contract to the wrong vendor. The potential effects of this can be costly to the state and to vendors.
Recommendation 1
Purchasing should write the RFP procedures in a clear and concise manner that enables
state agencies to easily follow them, and Purchasing should improve its training to ensure state
agencies follow the RFP procedures.
Recommendation 2
Purchasing should improve its review of the final consensus evaluation document of each
apparent successful vendor of an RFP contract to ensure accuracy and completeness.
ISSUE AREA 2: Purchasing Division Should Improve the Collection of Certain
Management Information
Certain management information is not collected by the Purchasing Division that is useful to
improve the procurement process, and identify areas that may need improvement. There is a
significant amount of purchases that do not go through Purchasing. This suggests a need to ensure
that state agencies are complying with proper purchasing procedures to ensure purchases are
competitive, ethical, and economical.
Purchasing Does Not Perform Quality Assistance Reviews on a Regular Basis
According to the Purchasing's Agency Purchasing Manual,
Periodic visits will be made by Purchasing Division representatives to review
purchasing activities, the delivery of commodities and services, and records of the
agencies. They will assist and advise state agencies in matters pertaining to the rules
and regulations, policies and procedures of the Purchasing Division. Agency
personnel should always take the opportunity to offer suggestions and request any
other assistance that will assure a more efficient and effective purchasing system.
However, this is difficult to accomplish when Quality Assistance (QA) reviews are not taking place
on a formal basis, just on an as needed basis. The Legislative Auditor requested from Purchasing
the outcomes of Quality Assistance reviews for the past three fiscal years. No QA reports were
provided.
State agencies are allowed to make purchases without going through the Purchasing Division, if the purchases are under $10,000. Purchasing requires that even in these cases, the agency is required to ensure competitive purchases. Purchases between $1,001 and $5,000 require a minimum of three verbal bids when possible. The agency is required to document and record all bids for public record on forms prescribed by Purchasing. Awards can only be made to vendors who are registered with the Purchasing Division. Agency purchases between $5,001 to $10,000 require at least three written bids on forms prescribed by Purchasing.
Having QA reviews performed on a regular basis could possibly indicate whether or not agencies are conforming to proper purchasing procedures. For example, these reviews could accomplish at least three things:
) indicate if agencies are by-passing purchasing requirements by stating there is only
one source for a product, when really there is not;
) indicate if agencies are splitting total orders so that the individual amounts are below
the $10,000 limit in order for agencies to avoid going through the Purchasing
Division; and,
) indicate if agency purchases are competitive and ethical as required by
Purchasing.
There are instances in which state agencies make purchases exceeding $10,000 that
Purchasing does not know who will be awarded the contract. For example, according to a
Purchasing Buyer, many of the Division of Highways maintenance contracts are multiple award
contracts. This means that all vendors who submit a bid are awarded a pricing contract. However,
the low bidder for an individual order, placed against these contracts, is determined by the
Division of Highways at the time the order is placed. If the low bidder has prior commitment
at that time, the next lowest bidder is then selected. The sums of money in these contracts can be
sizable. These types of occurrences warrant some oversight by Purchasing to ensure that proper
purchasing procedures are being followed, and nothing unethical is taking place. The same holds
true for purchases under $10,000.
The Executive Director indicated that QA reviews are not conducted on a formal level because Purchasing does not have an appropriate number of staff. He also indicated that the basis for QA inspections is found in West Virginia Code §5A-3-9 which states, "Within the limit of funds available, the director, or some person appointed by the director, shall determine whether commodities delivered or services performed conform to contractual requirements...."
Most Vendor Performance Complaints are Verbal
According to the Purchasing Manual, Section 4.8 on Vendor Performance, it states:
Vendor performance and product quality are crucial in the state purchasing process.
When these factors do not meet expectations, agencies are encouraged to contact the
vendor and voice their concerns. Complaints should be resolved expeditiously and
courteously, preferably by the state agency. If the complaint cannot be resolved at
the agency level, it should immediately be referred to the Purchasing Division. The
Vendor Performance Form, WV-82 is to be used to report all incidents of
vendor performance even if a resolution has been reached. When assistance is
requested, a Purchasing Division official will notify the vendor directly in writing
of the problem.
According to the Executive Director of Purchasing, "After the problem is resolved, a copy
of the documentation is kept in the specific purchase order file for future reference." The Director
does not monitor vendor performance complaints, but he indicated that written vendor performance
reports occur in less than 1% of all transactions. However, there are many more verbal complaints
by state agencies to Purchasing Buyers or to the Executive Director.
Since Purchasing does not monitor formal or verbal vendor performance complaints, it is not known the extent of poor performance by vendors. There is a benefit in requiring agencies to submit vendor performance forms on good or bad performance on contracts with a value greater than a certain amount. If vendors know ahead of time that their performance will come under a certain amount of scrutiny, then these vendors would be more apt to deliver their goods and/or services in a timely and efficient manner. Furthermore, if Purchasing monitored these performance evaluations of vendors, it could use the information to identify vendors with frequent complaints against them. Overall, this process would improve vendor performance and reduce contracting with vendors with a history of poor performance. Therefore, the Legislative Auditor contends that Purchasing should require state agencies to submit vendor performance forms for transactions with vendors, good experiences or bad.
Purchasing Does not Keep Records of Vendor Protests
Vendors can file a protest to Purchasing if they feel that they have been unfairly disqualified or denied a contract. This can occur in the RFP and RFQ process. For example, if a vendor with the lowest bid for a RFQ contract was not awarded a contract because it did not meet the specifications, the vendor could file a complaint opposing the decision. All protests slow up the purchasing process because Purchasing cannot proceed until the protest has been settled. Agency will then be delayed in receiving their goods or services. RFP contracts have been known to be delayed by more than a year until a court decision is made on the complaint.
There are basically three different levels of protests that a vendor can pursue. A 1st level protest goes to the Director of Acquisition and Contract Administration. If the protest is not settled at this level, then a vendor can take it to the 2nd level, which is forwarded to the Executive Director of Purchasing. If the vendor is not satisfied at the 2nd level, then the protest can be taken to circuit court. Of all the contracts that are awarded each year, the Executive Director estimates that approximately less than 1% result in any type of protest. Most of the protests do not reach the 2nd level, with approximately only four protests a year making it that far. According to the Executive Director, only two or three protests a year make it to circuit court.
Currently, Purchasing does not keep records of the number of protests that reach the 1st, 2nd, or court level. Keeping a complete and accurate record of all protests can be a good management tool used to gauge the performance of the procurement system. For instance, if records of protests were kept, then Purchasing could monitor whether or not there was an increase in the number of protests from year to year. If there happened to be an increase in 1st and/or 2nd level protests, then this could indicate that the purchasing process is breaking down at both the agency level and the Purchasing Division level, and the appropriate measures could be taken to alleviate the problem. Therefore, Purchasing should consider monitoring all levels of vendor protests.
Recommendation 3
The Purchasing Division should keep written records of all vendor protests that occur on
the 1st, 2nd, and circuit court levels.
Recommendation 4
The Purchasing Division should consider rule changes that require all state agencies to
submit a prescribed vendor performance form for transactions of a certain dollar value for good or
bad performance. Purchasing should also monitor vendor performance forms as a means to
possibly deny contracts to vendors with frequent complaints against them.
Recommendation 5
The Purchasing Division should conduct Quality Assistance reviews as stated in the
Purchasing Manual.