Case Study of the Implementation of the Performance Information Procurement System at the Dallas Independent School District

Introduction

The Dallas Independent School District (DISD) has had a history of problems with delivering performing construction.  DISD was using the design-bid-build process, which is typically referred to as the low-bid process, and it is the most common procurement method throughout the construction industry.  However, this process has been plagued with non-performance, such as not finishing on time, not finishing within budget, and providing a product that did not meet the owner’s quality expectations (Illia 2001, ENR 1997, Post 2001, ENR 2001, ENR 2001, Rosta 2001, Rosenbaum 2001, ENR 2002). Agencies nationwide have been searching for alternate delivery systems in order to improve their end products (Angelo 1997 & 2001, ENR 2001, Illia 2001). 

The Performance Based Studies Research Group (PBSRG) at Arizona State University (ASU) proposed to the DISD, that it was not only the delivery process, but also the selection method that was causing the poor performance.  PBSRG also proposed that the DISD could achieve the following based on a theoretical foundation of information theory and past tests of the information theory:

1. Minimize design, management of construction, and maintenance of facility systems.
2. Increase competition.
3. Increase performance of constructed systems (on time, minimize contractor change orders, and meet quality expectations.)
4. Maintain or minimize construction costs.

Industry Structural Stability

The authors propose that industry structural instability is the major cause of construction nonperformance.  The instability is caused by poor business practices, which are encumbered by non-value added functions.  Figure 1 shows the industry structure based on competition and performance.  Worldwide price pressures and the owner’s motivation to get the best price, has moved the industry into Quadrants I and II.  Quadrant I is the design-bid-build or low bid environment.  Quadrant II is the best value (performance and price) environment.  Figure 2 shows that Quadrant I is controlled by the design, construction management, and inspection, while Quadrant II is controlled by the performing contractor who has been selected by best value.  The performance environment moves the risk from the owner or buyer to the contractor or vendor.  This by definition is “performance contracting,” the identification of the requirement but not how to do the requirement (DUSD 2001, OAM 2002). 

Quadrant I has the following poor business practices:

1. Attempt to increases value by lowering the price (win-lose.)
2. Solves performance issues by management and control instead of more skilled contractors and craftspeople.
3. Does not use performance information to get best value.
4. Uses minimum standards, directing contractors to produce the minimal quality of work.
5. Minimizes competition by subjectively selecting materials and means and methods.

The authors propose that by using information technology (performance information, Information Measurement Theory (IMT),) and minimizing subjective bias, management, and control, a user can move from Quadrant I to Quadrant II.  This move would increase performance, increase competition, maintain price, and minimize management, control, and the need for subjective expertise.  The authors propose that by transforming engineers into “information workers,” owners and buyers of construction can solve the industry instability problem.  One of the difficulties in making the movement is the large number of current participants in the construction industry who have built their business based on the functions required in Quadrant I.  These companies would lose their competitive advantage (or be forced to change their functions) if the use of IMT and performance information changes the environment from Quadrant I to Quadrant II.  A Quadrant I environment, because it lacks information, has the following conditions:

1. Confuses liability of problems between the contractor, manufacturers, designers, and inspectors.
2. Allows bureaucracy to exist, and oftentimes results in “finger pointing”, forcing the owner to solve the problem. 
3. Results in practices that are inefficient and which oftentimes are not resolved. 

Performance Information Procurement System (PIPS)

PIPS was developed by Dean Kashiwagi at ASU in 1991, and modified and improved from 1994 to the present through over 350 tests.  PIPS has been used successfully in the private sector (Motorola, Honeywell, Intel, IBM, and United Airlines) and on all levels of government (Federal Aviation Administration (FAA), State (Wyoming, Hawaii, Utah, and Georgia) and local levels (University of Hawaii and the Dallas Independent School District.)  Results include (Hawaii 2002):

1. 98% on time, on budget with no contractor generated cost change orders, and meeting quality expectations.
2. Projects from $20K to $53M.
3. 100% customer satisfaction.
4. Better value and lower overall delivery cost.

PIPS is schematically shown in Figure 3.  PIPS has a past information phase where contractors submit a reference list and the owner collects performance information from the references.  A management plan was required from the alternatives to identify risks, how they will minimize the risk, and how they will add value to the project.  A prioritization phase included the use of an artificial intelligent processor (Zeleny 1982).  The final part included the pre-award phase, which requires the contractor to make any clarifications.  The contract is then awarded and the contractor is responsible for meeting the owner’s expectations.

Figure 3: Minimization of Risk using PIPS

The differences from other best value or alternate delivery systems is the PIPS includes all the following characteristics:

1. Minimizes the use of minimum standards.
2. Increases competition.
3. Forces continuous improvement.
4. Ties performance on past jobs, performance on the current job, and future competitiveness.
5. Allows a contractor to regulate each other by the identification and minimization of risks during the management plan phase. 

History of the Dallas Independent School District (DISD)

The Dallas Independent School District is the 10^th largest school system in the nation (Keller 1997).  With an approximate $980 Million in their general operating budget, the District is responsible for 218 schools located within a 351-square-mile area (DISD 2001).  Unfortunately, within the last six years, the DISD has been plagued with corruption charges.  The FBI has investigated various public officials for extortion, bribery, massive overtime fraud, mail fraud, intentional misapplication of funds, and other charges (Keller 1997, Schutze 1997& 2000, Kopenec 1997).  Several employees have been found guilty, including a former Dallas School District Superintendent (Keller 1997).  The challenge to DISD and PBSRG was to see if PIPS could be implemented in an environment that has been vulnerable to corruption and mismanagement. They have also been hampered by the low quality of poor performing contractors.  DISD managers were seeking a process that minimized management, increased value and performance within the budgets of the DISD.   

Objectives of DISD

In the summer of 2001, the DISD partnered with the PBSRG to run a prototype test on 9 different roofing projects.  The objective was to implement and evaluate the PIPS process to see if it could produce the following goals:

1.        Change the attitude of contractors and manufacturers servicing DISD to be more service oriented. 

2.        Increase the quality of work that is performed on the DISD roofs.

3.        Increase manufacturer participation

4.        Increase contractor participation

5.        Complete the projects on time, within budget, and with no contractor-cost change orders.

6.        Ensure that the roofs had no leaks.

7.        Shift risk from the DISD to the contractors that were responsible for the work.

8.        Provide longer and/or better warranties (less exclusions).

9.        Provide justified documentation on using best-value to select a contractor / system.

The scope of work for the prototype PIPS test was to use the process to re-roof 9 school buildings.  The total size of these buildings was approximately 500,000 square feet, and the estimated budget was $5 Million. 

DISD / PIPS Process Modification

 Being a public entity, the DISD had certain legal constraints.  DISD used the Design-Build delivery system to implement PIPS.  The process required the following modifications:

1. Changed PIPS from a one-step to a two-step process.  The first step reduced the number of alternatives to five, and the second step picked the best value.  A separate model was run to prioritize each school. 
2. The DISD limited the type of roofing systems to modified bitumen, built up roofs (BUR,) and sprayed polyurethane foam roof systems, which met the BUR or equal performance tests for durability.
3. Included a minority or women business enterprise (MWBE) rating.
4. Included a risk minimization factor that no contractor could receive more than 33% of the work and no manufacturer’s system could receive more than 67% of the work.  This factor was used to ensure that the roofs would get done in a timely fashion and to increase competition between systems. 

The detailed steps for DISD are listed below:

1. Owner sets the performance criteria and weights the criteria, which are included in the RFP.
2. Contractors and manufacturers submit references. DISD requested a maximum 50 references within 150 miles of the center of DISD, and inspected as many as 20% of the roofs from both the contractors and the manufacturers.  All references were sent a reference sheet for owner ratings by DISD. 
3. Owner collects performance information.
4. Contractors submit a management plan.
5. Model prioritizes top five contractors with partnering manufacturers and management plans based on the preset owner weights.
6. The processor identifies the top five options. 
7. The top five options then submit management plans and prices.
8. The processor identifies the relative performance, and then compares the relative performance and prices to prioritize the alternatives based on the user’s weights.
9. The owner identifies a priority of roofs to be done, and identifies the top rated option.  The owner then offers the projects to the top rated contractor.  Some contractors are skipped due to the contractor’s 33% constraint of the roofs (by roof area,) or the 66% of the requirement per system.  This process is done to all the roofs.  This is possible because some of the roof “best values” may be lower than the budgets. 
10. The top prioritized contractor is required to go through a technical review and responsible for any clarifications. 

Difficulties with Implementation

 The following problems were encountered in the implementation:

1. Contractors and manufacturers had difficulty understanding the importance of their references. 
2. Contractors and manufacturers had difficulty following instructions.
3. DISD procurement agent did not fully understand their own procurement rules and constraints due to a lack of experience of using alternate delivery systems.
4. DISD project manager was more accommodating than required, making decisions to allow contractors to meet the information requirements.
5. DISD had difficulty transferring all the risk to the contractor.

The DISD test verified that the roofing sector is in Quadrant I.  The majority of the roofing contractors and manufacturers do not know the performance of their systems.  In general, roofing contractors and manufacturers are not experienced in checking back with customers and getting customer satisfaction and performance information.   They do not regularly check their roof systems, and do not know the owner’s representatives who are tracking the roof performance.  Warranties minimize the liability of the manufacturers through the use of exclusions and are not determined by proven performance.

DISD was used to getting two to three roofing contractors and two to three manufacturers to bid the re-roofing projects.  Due to the motivation to get more contractors and manufacturers involved, the DISD project manager, expended excessive time and effort to assist the contractors and manufacturers.  Experience has shown that assisting the contractors and manufacturers does not result in either higher performance by the contractors and manufacturers or more understanding of performance.  However, most users when they first implement the process will tend to take this approach. 

The DISD project manager was not accustomed to turning the risks over to the contractors and allowing them to perform and get rated based on their performance.  The PIPS system identifies “turning the risk over to the contractor” as not worrying about their performance.  If problems occur, they must solve their problems, minimizing risk and ensuring that the user and owner recognize their performance.  One of the most difficult tasks is to allow the performing contractors to solve their own problems and for the owner’s representative not to take any responsibility for the performance of the contractor.

Analysis of Results

Table 1 shows an analysis of the past performance references that were submitted.  The number of contractors increased from the usual 3 to 21, and the number of manufacturers from 3 to 11.  The number of surveys returned was low for the manufacturers. 

Table 1

Analysis of the Company References Lists

The results from Table 2 show that the contractors and manufacturers had information on roofs with an average age of 4 years.  The average warranty periods were 15 to 25 years.  This is troubling because owner’s perceive that the warranty period is related to performance information.  The performance information also shows that 53% of the roofs never leaked, which means 43% leaked for one reason or another.  This is higher than previously seen in other areas.  It is also important to note that the contractors and manufacturers are directed to only submit roofs that show optimal performance or capability.   

Table 2

Analysis of Various Performance Criteria

Other results included the following:

1.                    Out of 48 submittals (some companies submitted multiple systems), 34 had missing or incorrect information. 

2.                    At least one company had 0% of roofs that never leaked (they all leaked), and 25% that currently leak, 0% completed on time, and an average service period of 0.4 years.  It is clear to see that some companies did not spend enough time in selecting proper references and did not follow up with them (resulting in poor performance numbers and poor response rates). 

The areas that impacted the manufacturers included:

1.                    Not enough returned surveys

2.                    Not enough older roofs that did not leak

The data shows that manufacturers do not track the performance of their roof system in terms that can assure the buyers of long-term performance. The single most important criteria in the second phase prioritization were the contractor’s management plan.  The management plan score was composed up of the first phase management plan score, the second phase detailed proposal evaluation, and the MWBE evaluation.  The reason that the management plan category had such a strong impact was due to the MWBE evaluations (see Table 3).  The average rating was 4.1 out of 10 with a standard deviation of 3.2, which caused a substantial difference in scores. 

Before the award of any of the nine projects, the DISD began to notice a change in attitude from various contractors/manufacturers.  Roofs that the DISD were having problems with, due to finger pointing between the contractor and manufacturer, were suddenly being repaired without any pressure of the DISD personnel. School Districts around DISD whom the competing contractors serviced, were also receiving an upgraded service.  Activities like this are not uncommon when a PIPS project is run, since the contractors and manufacturers realize that their past performance does make a difference on their capability to get future projects. 

Table 3

Analysis of the Management Plan Category

The following are the preliminary results of the prototype DISD-PIPS tests:

 Total Number of Jobs:  9

Total Number of Different Contractors Awarded a Job: 4

Total Number of Different Systems Awarded: 2 (BUR and SPF)

Total Budget:  $4.8 Million

Total Awards:  $4.2 Million

Percent Under Budget: -15%

Average Length of Warranty:  19 years

The roofs have not been completed as to date.  So far, the roofs that are constructed are above the quality of the normal DISD roofs.  In analyzing the value that the DISD purchased, the high performance did not cost the DISD any additional funding.  To understand why, the results of Table 4 must be analyzed to understand the hypothesis of the PIPS system:

Table 4

Award Results of the DISD Best Value Implementation

Performance does not always cost more.  Contractor #6, a very highly rated contractor who did not bid previously under DISD, won the Johnston School with a bid over 10% under the next lowest bid, and 38% below the budget.  However, on Mt Auburn School, they were awarded the project on the highest bid due to the better values not being available.  The process forces a “win-win” between the contractors who are minimizing risk and attempting to meet the expectations of the owner, and the owner who has a limited budget. 

Conclusion

The testing of the PIPS process met the following objectives of the DISD:

1.        Changed the attitude of contractors and manufacturers servicing DISD to a more service oriented approach. 

2.        Increase the quality of work that is performed on the DISD roofs.  The quality of work increased from just a roofing project to waterproofing the building roof structure.

3.        Increase manufacturer participation.  The number of manufacturers went from 3 to 11.

4.        Increase contractor participation.  The number of contractors went from 3 to 21.

5.        Complete the projects on time, within budget, and with no contractor-cost change orders.  The projects are not yet completed, but there have been no contractor generated change orders.

6.        Ensure that the roofs had no leaks.  Contractors are not only fixing the roof leaks, but also penetration and fascia leaks.

7.        Shifted risk from the DISD to the contractors that were responsible for the work.  The contractors took full responsibility for all leaking during the construction.  They are required to inspect their own work and fix any leaking attributed to the roofs during the length of the warranty. 

8.        Provide longer and/or better warranties (less exclusions).  The warranties are full service warranties by the manufacturers.

Although the construction is not completed yet, the results are similar to the results at the State of Hawaii and other owners who have tested PIPS.  The results also sustain the hypothesis of industry structural stability that performing contractors can perform without means and methods direction, and best value does not always mean higher costs.  The tests also sustain the hypothesis that a user information system that uses performance information minimizes the risk of failure and nonperformance.   

Recommendations

Based on the DISD test, PBSRG is modifying PIPS to meet the following objectives:

1. Make PIPS easier to manage by minimizing work without increasing risk.
2. Eliminate the confusion with some of the contractors that felt that their references were not contacted.
3. Motivate the contractors to check with their own references.
4. Reinforce the need for the project manager representing the owner to not make decisions.
5. Allow the contractors and manufacturers to determine their own performance. 

 The following changes are being made to the PIPS process and recommended to any party running performance-based procurement:

1. Allow the contractors or vendors to send the rating sheets to their references, and the references will send the ratings directly to the owner. 
2. Only inspect the roofs to minimize risk.  If risk is identified, then inspect only those roofs.
3. Minimize the criteria to customer satisfaction, ability to identify risk, performance in terms of leaking, age, and ability to fix leaking.  

These changes reduce the management of the procurement by over 50% and the authors propose does not increase risk.  The change to allow the vendor to send the forms to their references motivates the vendors to know their references and also to identify their performance.  An analysis of over 100 roof procurements shows that the 80% of the inspections do not minimize risk, and the inspections are required only to confirm very high or low performance.  Because this is a performance based system, only the high past performance becomes a risk if not validated by inspection.   Recommend testing of PIPS continues with the optimized version. 

References

Angelo, W. J. (1997, June).  Court Clears Way For New Methods.  Engineering News Record, 238[24], 10.

Angelo, W.J. (2001, June).  Web Voters Seek Alternatives.  Engineering News Record, 246[25], 107. 

 Dallas Independent School District (DISD) (2001).  http://www.dallasisd.org/

Decadence in Big D. (1997, April 25).   Houston Chronicle Dallas Bureau, (Jim Schutze, author).

Deputy Under Secretary of Defense (2001) “A Guide to Collection and Use of Past Performance Information.” http://www.acq.osd.mil/ar/

If They Only Had A Brain.  (2000, July 6).  The Dallas Observer. (Jim Schutze, author).  

Illia, T. (2001, April). CM Says Local Preference Lifts Nevada Building Costs.  Engineering News Record, 246[13], 12. 

Illia, T.  (2001, July).  Late, Overbudget State Job Sparks Contracting Changes.  Engineering News Record, 247[1], 17.

Illia, T. (2001, September).   Liens Pass $56 Million Mark on Completed Intel Project.  Engineering News Record, 247[11], 12.

Keller, B. (1997, October 15).  In Plea Deal, Dalls Supt. Admits Theft.  Education Week. 

Kopenec, S.  (1997, October 8).  Superintendent Agrees to Plead Guilty to Embezzlement.  Abilene Reporter News. 

Office of Acquisition Management, “Seven Steps to Performance Based Services Acquisition.” 2002.

Post, N. M. (2001, February).  Beach Development Awash in Delays.  Engineering News Record, 246[5], 56.

Rosenbaum, D. B.  (2001, February).  Contractor Says Hoover Dam Ruling Would Change All Rules.  Engineering News Record, 246[5], 12.

Rosta, P. and Korman, R. (2001, July).  Jury Deliberates Damages as Tutor’s Methods are Debated.  Engineering News Record, 247[5], 11.

 Staff Writer (1997, June).  A Defining Moment.  Engineering News Record, 238[25], 58.

 Staff Writer (2001, August).  Hong Kong’s Contractors are Failing at a Growing Rate.  August 6, 2001.  Engineering News Record, 247[6], 16. 

Staff Writer (2001, August).  Nevada Removes Contractor. Engineering News Record, 247[9], 15.

Staff Writer (2001, August) San Diego Wants to Tighten Focus on Responsible Bidders.  Engineering News Record, 247[9], 15.

Staff Writer (2002, February). California Court Limits Change Order Claims.  Engineering News Record, 248[5], 7. 

State of Hawaii, “Report for Senate Concurrent Resolution NO.39 Requesting A Review of the Performance Information Procurement System (PIPS)”.  PIPS Advisory Committee.  November, 2002.

Zeleny, M. (1982).  Multiple Criteria Decision Making.  New York: McGraw Hill.