Continuing the Ranking Game: Using ASC Publication as One Criteria for the Ranking of C-schools

Rationale

The public construction program (C-schools) ranking game can trace its meager beginnings to Engineering News Record’s (ENR) and a select group of peer-construction program leader’s efforts beginning in 2000. ENR attended the Program Leaders Meeting at the 2001 Annual Conference of the Associated Schools of Construction (ASC) to vainly attempt to garner appropriate C-school top-ranking criteria from the attending program representatives. It is the opinion of the authors of this paper that ENR failed in the attempt, and that they discovered that construction programs, as a general rule, do not desire a national C-school top-ranking system. In October of that year ENR published the results of their effort, which did not afford a systematic top-ranking of C-schools, but did report student enrollment as a 20 largest C-schools rank along with profiling fourteen other data points across 88 programs (ENR, 2001).

At the 2005 Annual ASC Conference held at the University of Cincinnati the debate began again on top-ranking construction programs (Badger & Smith, 2005). These authors proposed a straw man ranking system, developed in a non-public workshop at Estes Park Colorado, to rank construction programs across five groups over nine categories thereby creating their “vision” of World Class C-school. This current article’s authors find it interesting to note that the private Estes Park workshop recommended that once a standard metric was established that ASC programs would be able benchmark themselves against these standards. Further, that this visionary information would be used for self evaluation purposes, not rankings, and would theoretically lead to improved program performance. Counter to the workshop’s recommendation not to rank their article is a rank article with overtones of competition by selected program size.

Profiling Data

The ASC represents is the only academic organization exclusively representing construction higher education programs. There are others who dabble in construction education, i.e. American Society of Engineer Education (ASEE), but these organizations have agendas different than the member programs of the ASC which is the reason that the ASC exists. The ASC began profiling its membership and publishing the results early in its history through annual institutional program, curriculum, student, textbook and salary surveys. The earliest profiler identified is a 1974 Course and Curriculum Information publication of the sixteen member programs (ASC, 1974). In 1982 James Rodger, Memphis State University, published the Minutes of the 18^th General Meeting profiled student enrollments of the fifty members (ASC, 1982). 1986 Dr. Kent Davis, John Brown University conducted and published the first identified ASC Textbook Survey (ASC, 1988). 1988-89 data was published from the ASC Institutional Survey including student enrollment at both undergraduate and graduate levels, and program faculty counts (ASC, 1989). In 1989 the second of two topics identified by the ASC Board as an essential organizational function was to serve as an information source to members, industry and the public (Badger, 1989). Steve Schuette, Purdue University, conducted the first published faculty salary survey (Schuette, 1994). In 1995 the ASC went electronic through the ASC Web Site, forms were created and annual data on the ASC annual surveys were called for, collected, archived and reported until 2002. At the current time only the salary survey has been maintained. One could easily say that the ASC has lost the mission of the early 1980’s in sharing program profiles. Without question, it is the responsibility of the ASC to profile its membership and publish that data. Without cross-program data it will be almost impossible for the ASC to advance the mission of its membership. To that end it has failed its membership.

Any profile data collected may be incorrect and as such any ranking system is precarious because the underlying data may not be reliable. Much of the data used is supplied by the program leaders themselves and is often a reflection not of reality but what they really want to report. Different programs may have different ways of counting courses, faculty, students, or it is entirely possible that the agent reporting data simply made a mistake that is not caught. Quantitative data also strips away information about the qualitative differences in programs. In some cases quantitative profile data would make programs look more similar than they really are, but in most cases it makes programs look more different in quality than they really are. Programs probably vary more in character and tradition than they do in quantity and size. The problem with a totally quantitative ranking system will be that many closely ranked programs will have very similar total profile criteria scores, and even programs with quite different rankings can have similar profiles. With an insignificant change in the profile data, such as two additional students passing the AIC exam, a program could leap over competitors with similar total scores. So, top-rankings are treacherous because they mask the fact that the differences between C-schools are negligible on many profile criteria. In reality, for C-schools that have close profile scores and ranks, whether the university has a good football team is as important as the difference in rank.

Ranking System

Profiling is integral to program advancement and any systematic approach to C-school ranking, but the C-school ranking game is a very dangerous adventure. This discussion on ranking is drawn from the works of Jeffrey E. Stake, Indiana University School of Law.

“There are lies, damned lies, statistics, and rankings. Rankings create hidden problems for people who are choosing a school, people who are affiliated with a school, and for society at large. They can discourage applicants, students, and teachers. They can cause applicants to choose the wrong school and schools to admit the wrong applicants. Rankings can even cause schools to teach less well by deflecting a curriculum from what matters to what will improve rankings (Stake, 2005).”

Three principal end users are generally identified within the ranking literature, 1) students, 2) programs, and 3) employers. Rankings are highly dependant on the choice of factors and the equity weights applied to the various factors. There are no correct weights that fit all users. Ranking systems with arbitrarily fixed weights will potentially mislead students, programs and employers. The nine categories and embedded sub-categories proposed by Badger & Smith are quite extensive and may be appropriate to their expectations. However, their assignment of a value system is considered by this paper’s authors as inappropriate and exemplifies their program leader and engineering bias. Each ranking user is looking for completely different rank based on unique value systems, problematically from the same data set. Students and their parents are most likely interested in searching for and finding a program that fits with their unique expectations. These expectations might include the quality of teaching, the accessibility of faculty, the availability of internship programs, the potential for employment during and after their educational experience, the strength of the alumni network, as well as, housing, educational costs, and local leisure opportunities. Not found within these search expectations might be faculty research and scholarly publication, donations, institutional funding, advisory boards, continuing education (which they will not need until after they graduate) and above all, faculty offices. One could probably effectively argue that a potential student or even a potential employer would have no value in the ranking criteria proposed by this current paper; however construction programs should value the criteria.

Badger and Smith identify five program groups by significant levels of size and maturity between programs. They propose a stratification of the ASC, in the order of the controversial “peer-six” collaboration. A better solution would be to normalize the data prior to any ranking to adjust for or negate the effects of program size and maturity. For some criteria, higher numbers are better and for other criteria such as faculty-to-student ratios and tuition prices, lower scores are better. However, these scores could be normalized by subtracting them from a constant to reverse their order, making them the higher score within the ranking. Following this iteration, all scores on one criterion could be divided by the standard deviation of those scores. The end result would be that, the criteria are equally distributed, and therefore have equal weight irrespective of the criterion factors. The size and maturity of a program has little to do with the quality of teaching or program content.

Rankings may erroneously make some programs look bad. Users of the rankings may not know much about construction education and be lead to believe that the programs at the bottom of the list are inferior programs, when they are in fact excellent academic programs that produce superior results. A rank’s meaning is totally dependent upon the values of the end user and any rank system must provide for adjustable value weights. Ranking equity weighting will most likely be defined by minute differences in composite C-school scores. As with all games there are winners and losers and there will be bottom-ranked programs as well top-ranked programs. Construction education, program profiling and even top-ranking systems has many potential winners. The negative side effect of ranking programs, as we do other competitions, may be to cause users to forget that unlike games, one program’s losing does not give their program meaning, educational programs do not need to defeat other programs to be winners.

Problem Statement

None of the nine categories proposed by the Badger and Smith paper mentioned research in the categorical title. Research is included in the model under the category Faculty which accounted for 150 points out of a total of 1000 points. Of these 150 points 40 points are allotted to Publications. This paper’s authors maintain that publications within this academic construction organization are one of the most visible indicators of construction research activity, to some extent the most easily and factually quantified, and as such demands significant weight in any proposed ranking or assessment scheme. The Badger and Smith model’s publication criteria we consider flawed by two aspects. First it is an annual snapshot of faculty publications, as we all know there is significant turnover in faculty within construction programs and the push for publication is related to tenure seeking faculty. There is a significant probability that programs with the youngest and most tenure seeking faculty will skew the criterion scoring positively in favor of that program. It is probably safe to say that this young faculty is not yet representative of a program’s academic quality as are their more mature faculty. Finally, we also are aware of faculty who are not aligned with construction while teaching within construction programs. Most engineers and architects teaching within construction programs, as a general rule, do not make use of the ASC publication opportunities. They support the discipline’s publication effort from whence they came. To these, construction remains a second-hand resource.

This paper continues the ranking debate by proposing ranking criteria for C-schools based on a research output profiles exemplified by “long-term” and “current” ASC publication. This first attempt at ranking research output will use the publications database of the ASC. This research will profile and evaluate the publication record of construction education programs in the (International) Journal of Construction Education (JCE) and the Annual Proceedings of the ASC in order to propose an effective research output criterion set and measurements to be included in future models for C-school top-ranking.

Method

Sample Description

The populations under analysis are the publications of the ASC; the Annual Proceedings and the (International) Journal of Construction Education. The ASC website at http://www.ascweb.org/ holds an archive of the proceedings dating back to the 23^rd conference held at Purdue in 1987 with the 22^nd conference proceedings held at the University of Florida in 1986 representing the first published proceedings; however it is not archived by the ASC. All Journal articles dating back to volume one published in 1996 are also in the archive. The total number of proceedings papers is 601 and the total number of journal articles is 137.

Procedure

This paper will rank the research output of construction programs by analyzing four samples from the population:

Scoring

Each article or paper in the population will be assigned one point. If the authors are all from the same university the whole point will go to that university. If there are authors from different universities the one point will be apportioned equally between the universities. For example the paper entitled Distance Education with Internet2 Audio/Video Technology published in Volume 1.1 (Summer 2004) of the International Journal of Construction Education has three authors from the University of Nebraska, Lincoln and one author from Colorado State University, therefore 0.75 points would be assigned to the University of Nebraska, Lincoln and 0.25 points to Colorado State University. C-schools will be initially ranked by publication counts within the sampled grouping of the total population.

There are three threats identified that could have a direct affect upon the calculation of the final ranking. These threat’s scorings are treated as cumulative step scores in achieving a final score and rank. Each weighting step utilizes an assigned value and calculated score leading to the next threat category. The first, membership starts and stops date data was gathered from the 2001 ASC Database and the current programs listed on the ASC Website. Length of membership was then adjusted to the length of time of the period being investigated. If they are not an active member of the ASC their score is zero and if they have not been a participant for at least 75% of the evaluation period their score is calculated using a 0.75 multiplier. This iteration assures that all active member program articles or papers, given that they have an established publication track record, are equally measured by increasing the count to match the full time-period members, e.g., IF(Membership Years=0,0,(IF(Membership Years<75% of Sample Period,(Publication Count*0.75),(((1-(Membership Years/Years of Sample Period))+1)*Publication Count)))).

The Carnegie classifications were taken from the ASC Graduate Education web site. It is the opinion of these authors that a Tier 1 institution’s faculty are expected to publish, that a Tier 5 faculty are not necessarily required to publish, but all faculty should be publishing. If a Tier 5 faculty publishes they are exceeding their employment requirements and therefore should be rewarded above a Tier 1 faculty. A multiplier was therefore applied to the article or paper count to affect this reward. There are seven tiers within the Carnegie classification system. Tier 7 was given a 1.0 multiplier and the multiplier was reduced 0.05 per tier thereafter. This gave a Tier 1 school a 0.75 multiplier, e.g. (Carnegie Classification Value*Membership Score).

The faculty count data was gathered from the 1988-1989 ASC Institutional Survey, the 1995 (Williamson 1995) and 2001 (Williamson 2001) ASC Databases. To establish the 2005 tenured and tenure-track faculty counts an Internet search of all represented programs. In that the ASC web site does not provide program faculty rank data the Internet search was necessary. Prior ASC surveys only included tenured and tenure-track faculty counts and the search provided consistency in faculty counts. Faculty counts were averaged during the time period being investigated. This equally weights publication production to program faculty counts, e.g. (Carnegie Score/Average Faculty Count for Sample Period).

In that there is evidence of marked non-normality we are able to remedy this problem by applying suitable data transformation such as a smoothing of the criteria by a using statistical weighting scheme to calculate the final equalized score. An equal weight adjustment factor for article or paper productivity scoring can be calculated directly by dividing estimated article or paper production by the criteria’s mean standard deviation (NIST/SEMATECH, 2005), e.g. (Faculty Score/STDEV(Total Score)). This method is similar to the Daniell window which is a weighted moving average transformation used to smooth the criteria values. This transformation amounts to a simple (equal weight) moving average transformation of the criteria values, that is, each criterion estimate is computed as the mean of the preceding and subsequent criteria values as suggested by Bloomfield (2000).

Results

The (International) Journal of Construction Education since 1996

Since its inception in 1996 there have been 137 articles published in the (International) Journal of Construction Education. The 137 articles are accounted for by 38 institutions of these 33 are ASC members. The current ASC web site identifies 105 member institutions. So 31% of the ASC membership is represented in the Journal. Table 1 shows member institutions by article counts, years of membership, Carnegie classification, average faculty counts, the total equalized points scored and the rank for the JCE since 1996.

The (International) Journal of Construction Education from 2001-2005

Over the past five years 42 articles were published in the (International) Journal of Construction Education. The 42 articles are accounted for by 22 institutions of these 19 are currently ASC members. Therefore 18% of the ASC membership is represented in the Journal. Table 2 shows member institutions by article counts, years of membership, Carnegie classification, average faculty counts, the total equalized points scored and the rank for the JCE from 2001-2005.

The Proceedings of the Annual Conference of the Associated Schools of Construction from 1986-2005

The Proceedings Archive on the ASC website contains all the proceedings from the 23^rd Annual Conference onwards. The authors also obtained a paper copy of the proceedings of the 22^nd Conference. A total of 601 papers were presented at the 20 conferences. The 601 papers are accounted for by 76 institutions of which 60 of these are current ASC members. Therefore 57% of the ASC membership is represented in the Proceedings. Table 3 shows member institutions by article counts, years of membership, Carnegie classification, average faculty counts, the total equalized points scored and the rank for the Proceedings of the Annual Conference since 1986.

The Proceedings of the Annual Conference of the Associated Schools of Construction from 2001-2005

A total of 191 papers were presented at the last five ASC conferences. The 191 papers are accounted for by 47 institutions of which 40 of these are current ASC members. Therefore, 38% of the ASC membership is represented in the Proceedings for the last five conferences. Table 3 shows member institutions by article counts, years of membership, Carnegie classification, average faculty counts, and the total equalized points scored and the rank for the Proceedings of the Annual Conference for the last five conferences.

Discussion

The authors have proposed a more scientific and statistically valid method of measuring research output using the publications of the ASC. Unlike the conclusion drawn by Badger and Smith, "The program ranking system must be complicated enough to seem [or “appear”] scientific and the results must match, more or less, people's nonscientific prejudices." We believe that the top-ranking system can be designed scientifically to eliminate end user prejudices. The initial article or paper counts result show that Texas A&M University is ranked number one in all four samples. They would clearly represent the top ranked institution in the Journal since its inception and in the past five years and they would be the highest ranked in the Proceedings since 1986. However, when the equating weights are systematically applied it reaction to the validity threats a different picture arises. We indeed found that in all samples we were splitting hairs between programs as illustrated in Table 5. Across all samples differences only totaled 1.68 (SD 0.22) articles or papers between the top five, only 3.32 (SD 0.43) articles or papers separated the top and bottom programs, and the standard distributions were exceptionally small.

The University of North Florida (Average rank 1) is the highest ranked ‘long-term’ Journal C-school and Arizona State University (Average rank 1.5) is the highest ranked current Proceedings C-school. However, the long-term publication winner is Texas A&M University (Average rank 3) in that it is the only C-school which is represented in both the Journal and Proceedings top-five rankings which indicate a consistency in long-term Journal and Proceedings publication effort.

The University of North Florida is the highest ranked ‘current’ Journal C-school and Brigham Young University is the highest ranked current Proceedings C-school. However, the long-term publication winners are Brigham Young University (Average rank 2.5) and Texas A&M University (Average rank 3) in that they are the only C-schools which are represented in both the Journal and Proceedings top-five rankings which indicate a consistency in current Journal and Proceedings publication effort.

Which C-school deserves the highest ranking across both long-term and current sample groups? Texas A&M University (Average rank 3) ranked the highest by consistently appearing in all of the sample groupings. Brigham Young University (Average rank 2) is the only other C-school that was ranked in two of the sampling groups. If the analysis holds true then Texas A&M University has achieved the merit of being the top ranked C-school. But remember this is a rank game.

The use of the ASC publications as the source of data for ranking the research output of C-schools has arguments for and against. The authors believe the main argument for using ASC publications is that there would be an increase in faculty submitting articles or papers to both the Journal and Conference Proceedings if there was a formal C-school ranking system sponsored by the ASC. Those institutions that are competitive and seek to improve or maintain their ranking will encourage their faculty to publish in ASC publications. This would surely benefit the ASC and construction education as a whole. The main argument against is that the ASC publications are not the only venue for publication for ASC members and it is possible that other publications could and should be included. The authors maintain that if “Construction” is to continue to solidify its discipline it needs to have a core set of publications that are anchored in that discipline and not in the associated disciplines of Engineering and Architecture.

The authors also acknowledge that there are limitations to the method for measuring research output and that this current proposed publication system would only be of interest to programs. That is not to say that students or employers would sometimes not find this publication measure interesting but we believe it will not be of much significance to their unique value systems. Other or additional criteria should and could be utilized. For example, no score is awarded for the number of times that a paper is cited. This could be included if the archive was fully searchable and could be used to give extra weight to those papers that had made a significant contribution to the discipline and would be a positive weight criterion. An additional criterion could be if the construction program had a true construction Master’s and/or Ph.D. as defined by the ASC. The assumption would be that faculty within these programs would have improved opportunities for publication. The equality weight for this measure would naturally need to be negative to normalize all programs.

The authors believe that the academic debate regarding C-school top-ranking is good and healthy for a construction discipline such as ours. This paper continues that debate and hopefully will promote similar scholarship. As with any profile or rank schema there will have to be a champion, we believe that the ASC it up to the task and its membership can be that champion but it must redefine its mission to re-include profiling its members.

References

ASC (1974). Construction education programs: Course and curriculum information. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

ASC (1989). ASC institutional survey. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Badger, W. W. (1989). ASC future direction paper. Unpublished manuscript, Associated Schools of Construction, University of Nebraska – Lincoln. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Badger, W. W., & Smith, J. C. (2005). Ranking construction programs: The academic debate begins. Proceedings of the Associated Schools of Construction's Annual Conference, 41, Cincinnati, Ohio. Retrieved December 27, 2005, from the ASC Proceedings web site: http://www.asceditor.usm.edu/archives/2005/CEGT16_5800_Badger05.htm

Bloomfield, P. (2000). Fourier analysis of time series: an introduction (2^nd ed.). New York: John Wiley & Sons, Inc.

Davis, K (1988). ASC textbook survey. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

NIST/SEMATECH (2006). Engineering statistics handbook of statistical methods. Retrieved December 27, 2005, from the SEMATECH web site: http://www.itl.nist.gov/div898/handbook/

Rodger J. A. (1982). Minutes of the 18^th general meeting of the Associated Schools of Construction. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Rosenbaum D. B., Rubin, D. K. & Powers, M. B. (2001). The nation’s c-schools. Engineering News Record: New York , McGraw-Hill. 26-37.

Rounds, J. L. (1986). Proceedings of the 22^nd Annual Conference of the Associated Schools of Construction. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Schuette, S. D. (1995). ASC salary survey. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Stake, J. E., (2005). How to protect from ranking-mania. Retrieved December 29, 2005, from Indiana University School of Law - Bloomington’s web site: http://monoborg.law.indiana.edu/LawRank/rankingmania.shtml

Williamson III, K. C. (1995). [ASC electronic membership database]. Unpublished raw data. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)

Williamson III, K. C. (2001). [ASC electronic membership database]. Unpublished raw data. (Available from K. C. Williamson III, Ph.D., Department of Construction Science, Texas A&M University, College Station, Texas, 77845-3137)