PROGRAM ASSESSMENT GOOD MANAGEMENT PRACTICE

Bruce Yoakum

Department of Construction Management

California State University-Chico

Chico, California

Introduction

Program assessment, or outcome assessment has become a requirement for nearly all levels of endeavor in academic institutions. The six regional associations of schools and colleges such as the Western Association of Schools and Colleges (WASC) are the institutional accrediting agencies of the senior post secondary institutions throughout the United States. They require assessment of outcomes as part of their overall basis for granting or renewing accreditation. As a result, universities are required to demonstrate they have established an outcome assessment program campus wide. Thus each college or school as well as all of the subdivisions within those colleges or schools must also have outcome assessment programs in place. In addition, the discipline based accrediting agencies such as American Council for Construction Education (ACCE), Accrediting Board for Engineering and Technology (ABET), etc. are requiring assessment programs.

The initial struggle is establishing what "outcome assessment" is. W ASC's broad definition of assessment is "...the collection and analysis of evidence of effectiveness for all parts of the institution, including but not limited to evidence of student learning and achievement ...." It is a process to determine if what we are striving for is indeed occurring. In the case of Construction Management programs, first we define how we intend the graduates of our programs should function. A follow-up question would be are the students who receive our degrees able to perform in the construction industry as well as or better than intended?

The outcome assessment programs that we establish for our academic programs should serve two purposes: (1) to objectively satisfy all of our constituencies that we are in fact doing an acceptable job of living up to our stated mission and (2) provide mechanisms to allow us to continuously monitor and modify our programs as indicated to meet the needs of the markets) we serve. For some programs, thoughtful assessment of outcomes may be instrumental in assuring program survival. The following discussion gives a brief description of how the California State University, Chico Construction Management Department's outcome assessment program has been set up to satisfy these two purposes.

The Basis of the Program

The Construction Management Department's assessment program was organized to address the requirements of our program's accrediting agency, ACCE, and those of our University as established by the University Assessment Committee (UAC).

ACCE's Standards and Criteria for Baccalaureate Programs (Form 103) in Section VIII - Program Planning and

Outcome Assessment sets out the basic requirements for a construction program's assessment of outcomes. Those standards and criteria are reproduced in Appendix A. In addition, in 1991 ACCE's Outcome Assessment Committee produced a document titled "Implementation Guidelines for Outcome Assessment" which helps to identify the steps required to create a program and gives a general idea of what the content of the program should be.

In 1991 our University formed the UAC to increase university-wide awareness of assessment, its necessity, and to assist the campus in the continued development of its assessment programs. In the Spring of 1993 the committee published a set of guidelines which are to be used in the development of program and classroom assessment programs. The UAC's guidelines area response to the W ASC requirements and the essence of the guidelines are simply to develop assessment programs which answer three questions: (1) What are we trying to do?, (2) How well are we doing it?, and (3) How can we use this information to improve what we are doing?

For the Construction Management program the answer to question one in the preceding paragraph came from our department's five year strategic plan which is part of the College of Engineering, Computer Science and Technology's five year strategic plan. The strategic plan states the mission of the program as well as the program's goals and objectives to accomplish that mission. We were careful to establish quantifiable objectives in the strategic plan in order to adequately assess outcomes. The answers to questions two and three in the preceding paragraph are contained in the assessment program we've established.

The Assessment Program

The assessment program essentially is a restatement of the mission, goals, and objectives as contained in the departmental strategic plan. For each goal stated there is associated with it one or more objectives. The objectives are measurable and most of them are verifiable without a great deal of administrative effort.

A key element of the program is the Assessment Procedures which add specificity to the objectives. They give target dates and names of responsible parties where appropriate. These procedures are the part of the program that requires the most consideration and acceptance by the faculty. They spell out how and when the assessment of the objectives will be done and by whom. A copy of the Assessment Program is included in Appendix B. As an example, one of the goals is "Provide for and encourage the continual up-dating and development of faculty expertise." The objective associated with that goal is "Professional development plan for each faculty member in place by Spring 1994. Each plan to address, as a minimum, industry experience, publication, academics, and professional activities." The assessment procedure to accomplish and assess this objective is "A faculty professional development input form will be created by December 1993 and each faculty member will complete this form for inclusion in the Department file by February 1994. The form will include each individual's experience, plan for the year, and actual accomplishments." That form has been created and a copy is in Appendix C. These farms can also serve to assist faculty in meeting requirements for retention, tenure and promotion.

The assessment program was created with the thought that it will be treated as a dynamic tool requiring regular updating as well as continual monitoring to make sure that it is not ignored. The format allows for ease of editing including adding new items and deleting items which no longer fit. At the very least, the program will be reviewed and updated annually.

The criteria used in setting up the assessment program, in addition to trying to satisfy the requirements of the University and the accrediting agencies, was to create a plan that would be reasonably simple and would not require resources that are not realistically available. This means trying wherever possible to make use of information and resources that are already existing or in place. As an example, it was felt that trying to create some type of standardized test to assess the competency of graduating students would be very time consuming to both create and administer, and would probably not tell much about how well the graduates are prepared to function managing construction operations. Instead, we can obtain a good assessment of the graduates' preparation by the use of a relatively simple questionnaire sent out to a sampling of alumni and their employers on a biennial basis. A copy of the questionnaires we will use in the Spring of 1994 are contained in Appendices D and E. Given our limited resources, the questionnaires need to be fairly short and easy to fill out in order to maximize the percentage of responses and to make the compilation of the data manageable . To emphasize the importance of keeping the assessment program simple the following quote firm WASC's "Achieving Institutional Effectiveness Through Assessment" demonstrates the accrediting bodies recognition of today's limited resources in academia.

A special note about the costs of assessment is warranted. The Commission is concerned that institutions build assessment activities into existing processes rather than create costly new structures. Additional costs may be minimized by working with existing data, using sampling techniques, and attempting small pilot assessment projects until greater expertise is established. Comprehensive assessment projects should be initiated only after great care and planning to ensure that benefits will be maximized.

Our University's guidelines recommend assessment at the classroom as well as the program level. Accordingly, as a part of the goal to provide the Construction Management students an appropriate blend of quality courses, the program assessment includes a commitment to set up course assessments for all construction management courses by the end of the Spring 1994 term. Most courses have built-in assessment elements, such as student evaluations, exams, and projects, but the assessment is usually not documented and frequently lacks the mechanism to determine if what the students are learning really meets the objective of the course and if not how to remedy the deficiency. Classroom assessment will be geared to overcome those shortcomings. At a later date assessment of the other construction management major requirements may be considered.

Conclusions

There are as many ways to set up an assessment program as there are individuals willing to tackle it, but the common threads should be:

Keep it simple. Setting reasonable goals and measurable objectives is absolutely essential. Assessment should work for us, not the other way around.

Don't create a program that will require more than available resources can support.

The elements and the way they are assessed will change over time. This is an ongoing process, not something done every once in a while.

The assessment program should not only provide the mechanism to report assessment activity, but must also show evidence of the changes, if any, that have resulted.

At Chico State, the administration is adamant about starting with a program that is attainable and fits the needs of the department or program that it is applied to. They concede that it is unlikely that any two assessment programs will look the same since no two academic programs are the same. Program assessment does not need to be difficult, but it will be if done all at once rather than starting small and working up to what we'd ultimately like to have in place.

Summary

Our experience with the process of trying to develop an assessment program has not been totally free of resistance, but once we made up our minds to do it and make it a useable tool, we have found that it is not all that difficult. The key thing to remember is that assessment should not be viewed as an end in itself, but a means to an end. It's true that we are required to do the assessment by the accrediting agencies, but the real benefit should be in providing the programs with valuable feedback to improve the programs and the students those programs turn out. In some cases, a thoughtful, well focused assessment program could spell the difference between survival and extinction for a program.

An important thing to remember while developing an assessment program is that it is a dynamic tool; it is part of an ongoing process. Everything around us is continually changing and we must be in the mode of addressing and responding to those changes. As we get some things "fixed" we must move on to the other areas that may need attention. ACCE currently reviews the Baccalaureate program every six years, but it seems it is of very little value to initiate an assessment program, i.e. do a survey of the alumni once every six years. The assessment program needs to be in place on a continuing basis to be worth the effort it takes to set it up and to provide any long term benefits.

A reason assessment has not been a favorite place to put our efforts is it has somehow been viewed as a mysterious, time consuming and complex issue to deal with. It does not have to be complex, in fact if it is it will probably die of neglect. We have tried to keep our program simple and manageable. A good example is the questionnaire for alumni and their employers. Open-ended questions have been kept to a minimum. Written comments can be included, but they are usually difficult to deal with objectively. We have set it up to make it easy to reduce the data received and also to make sure that the conclusions from the data will be something we can act on.

Assessment programs should start small and grow as time, resources, and understanding of the process dictates. Our current assessment program does not include everything that is part of the strategic plan. It would have been too much to deal with in the beginning, but over time the assessment program and the strategic plan should be and will be parallel. After all, if we can't measure (assess) how we're doing against our goals and objectives why bother to have them in the first place?

References

"Achieving Institutional Effectiveness Though Assessment". (June 1991). Accrediting Commission for Senior Colleges and Universities of the Western Association of Schools and Colleges.

"Guidelines for Assessment at California State University, Chico". (Spring 1993)

Slobojan, Joe. (July25,1991). "Implementation Guidelines for Outcome Assessment for Construction Education Programs Accredited by the American Council for Construction Education".

Appendix A

ACCE Form 103 (Partial) Standards and Criteria for Baccalaureate Programs

VIII. Program Planning

And Outcome Assessment

The construction education unit must have a definitive plan which includes a mission statement, program goals, and measurable objectives. This plan will serve as a tool for continual development and improvement of the program.

The construction education unit's plan will form the basis for assessing outcomes of the program. Assessment input should be obtained from all program constituencies, such as students, including the documentation of their educational achievements in verifiable and consistent ways, such as evaluation of senior theses, reviews of student portfolios, and standardized test results; graduates, including job placement rates and employer evaluations; benefactors; employers; industry; faculty; and administration.

The planning and evaluation process must contain measurable objectives; a systematic means for collecting, quantifying, and analyzing data relevant to those objectives; development of conclusions based on the data collected; and program modifications, if indicated.

The conclusions and inferences drawn from this process should then be incorporated into the program plan in such a manner as to foster enhanced student achievement with respect to the construction education degree program. After each comprehensive outcome assessment, both short and long term goals and objectives should be restated and plans for implementation should be redefined, if necessary.

Adequate resources must be available so that the program may structure a mission statement, program goals and measurable objectives which will serve to ensure continual improvement of the program. Resources must support a systematic means for collecting, quantifying, and analyzing data relative to these objectives; formulation of conclusions based on this data, and appropriate program modifications.

The program planning and assessment schedule must be defined by each institution, and be consistent with the program's accreditation schedule for ACCE.

Appendix B

Department of Construction Management Program Outcomes Assessment

Mission

The Construction Management program's mission is to prepare graduates to manage construction operations and companies.

1.                     Provide a Construction Management curriculum em­phasizing an appropriate blend of

General Education

Mathematics

Science

Construction and Engineering Technology

General Business Management

Construction Management

Computer Science

2.                     Seek acceptable levels of program resources from both within and without the university community.

3.                     Prepare students for rewarding employment at the management level within the construction industry.

4.                     Provide for, and encourage the continual up-dating and  development of faculty expertise.

5.                     Maintain a high degree of program visibility at the campus, state and national levels.

Objectives

1.         a. .. Maintain a balance of ACCE unit requirements within their standards for G.E., Math, Science, Technology, Business, CM, and Computer Science.

b. .. Assure that requirements of the Departmental Writing Proficiency Policy are maintained.

c. .. Develop outcomes assessment program for each CM course by May 1994.

d. .. Modify 1991 alumni and employer questionarre and utilize every other AY starting Spring 1994.

2.         Obtain outside funding of approximately $50/alumni ($30,000) for AY 93-94. Funds may be obtained from alumni, corporate donors or others.

3.         Increase pool of committed coop/internship employers to 400+ by end of AY 93-94. This will provide the foundation for development of a mandatory coop/ internship program involving all CM students.

4.         Professional development plan for each faculty member in place by Spring 1994. Each plan to address, as a minimum, industry experience, publication, academics, and professional activities.

5.         a. .. Student organization shall complete two high visibility project each year and continue to participate in the Western Regional Associated Schools of Construction CM competition.

b. .. CM program to produce at least one article in a major industry publication per year.

c. .. Develop and distribute two brochures by May 1994. Direct one to the prospective student and one to the construction executive.

Assessment Procedures

1.         a. .. Verified during ACCE accreditation visit on six-year cycle. If ACCE requirements change, the necessary adjustments will be made.

b. .. Faculty members will be asked to complete a Course Writing Proficiency Assessment form for every Category 1 & 2 course, as delineated in the Department Writing Policy, taught during the academic year. This will be done in May of 1994. The Department Chair will review the forms and based on that review, will determine if some improvements/changes are in order for the 1994-95 academic year.

c. .. Course Outcomes Assessment Chair will verify that an Outcomes Assessment Program for each CM course has been reviewed and accepted by CM faculty by May 27, 1994.

d. .. Alumni and Employer questionnaires developed in 1990 and sent out in Spring of 1991, are to be updated/refined and sent out to a random selection of 100 alumni and their employers. Results are to be compiled and retained by the Department Chair. Department faculty and the Industry Advisory Board will review the results and make suggested changes as indicated by their review.

2.         a. ... Department Secretary will maintain a revenue log showing donor name, date, and amount for all funds received from outside sources. Department Chair and faculty will review this log in May, 1994, to compare actual to goal, determine need and develop a plan to meet that need for the 1994-95 academic year.

3.         a. ... Department Secretary will develop a file and log to document all committed co-op/internship employers and will keep it continuously updated. Inquiries to potential employers is an integral part of the current CM Co-op course.

4.         a. ... A faculty professional development input form will be created by December 1993 and each faculty member will complete this form for inclusion in the Department file by February 1994. The form will include each individual's experience, plan for the year, and actual accomplishments.

5.         a. ...1. CMA faculty advisor to encourage and assist CMA organization in identifying and executing two community projects each academic year.

a....2. CM faculty to assist student teams in the ASC competition. Both are verifiable at year-end.

b. .. Department Chair to assign to a CM faculty member the writing of one article for submittal to and publication in a major construction industry publication. Funding wild be provided to assist in article preparation. Publications might include, but not be limited to, any of the following:

Constructor (AGC)

Civil Engineering Magazine

ASCE Construction Eng and Management Journal

Project Management Assoc Magazine or Journal

Assoc of Cost Engineers Magazine or Journal

Specifier

Daily Pacific Builder

c. Prospective student and prospective employer (donor) brochures will be created using the existing brochure as a guideline. Department Chair will be responsible for getting this done.