The Challenges and Rewards of Outcome Assessment

Amir W. Al-Khafaji and James G. Seckler

Bradley University

Peoria, Illinois

Overview

The assessment of quality in higher education includes the need to have a clear awareness of institutional mission, resources, accreditation criteria, new technologies, and global competition (Al-Khafaji, et al, 1998). It is well known that quality is defined as meeting stated standards and objectives. Consequently, it is critical that a mission statement be the first order of business when developing an assessment program. Subsequently, a set of objectives with appropriate standards and norms must be established to help assess the degree of success in meeting program mission and objectives.

A major thrust of this article is to provide educational institutions with an overview of the assessment process and encourage faculty to improve. However, it is the responsibility of administrators in higher education to provide the proper environment and needed resources to stimulate and energize the faculty in their quest to achieve quality and excellence in meeting stated goals and objectives. Institutions need to place less emphasis on definitions and more on generation of a substantive rewards system for excellence in all areas of faculty work. It is important to note that it is now mandated by accrediting agencies that programs produce evidence of quality through an established assessment program.

Accreditation agencies and well-respected educators have taken the lead to demand greater accountability from educational institutions. While engineers and constructors enjoy admirable reputations, the education of future engineers and constructors must take into account the following factors:

1. Declining admissions standards.

2. Low salaries for construction and engineering graduates compared to business.

3. Global competition.

4. Changing technologies and lack of resources to adjust.

5. Poorly prepared high school graduates.

6. Lack of quality educators.

7. Lack of assessment standards to reward good teaching and service.

8. Grade inflation.

9. Inadequately prepared graduates.

10. Poor performance on standardized tests.

11. Poor communication skills.

12. High attrition rate of faculty in construction and civil engineering.

Although the validity of these and other factors may vary from one institution to another, there is no doubt as to their relevancy and legitimacy (Western Association of Schools and Colleges, 1992). Failure to institute needed changes will invite external regulation and pressures. Several states have already enacted requirements for mandatory institutional assessment. It is prudent and sensible to develop needed instruments to shape the process (Engineering News Record, 1996). It is suggested that each civil engineering and construction program build a program of routine data gathering and analysis that could be used for curricular improvement, strategic planning and resource allocations.

The ACCE Assessment Criteria

Over the years, the American Council for Construction Education (ACCE, 1998) has taken the lead in developing substantive accreditation criteria to insure quality. These criteria have evolved into an assessment-oriented set of requirements to help construction programs achieve stated missions and objectives. That is, ACCE requires a comprehensive and well-defined assessment program that relates to administration, curriculum, faculty and staff, students, facilities, services, and relations with industry (American Council for Construction Education). The specific requirements of the assessment process are summarized below.

1. A description of how outcome assessment results are correlated with program content, mission, goals and objectives to implement change where needed,

2. Provision of copies of all forms used in the program assessment process,

3. Provision of a summary of the most recent assessment cycle, including a description of the process used to evaluate both inputs and outputs, and a summary of the results,

4. A description of programs strengths, weaknesses, and opportunities identified in the assessment cycle, and

5. A statement of the specific plans, including a schedule, for overcoming identified weaknesses and incorporating identified opportunities into the program.

As part of the ACCE accreditation process the program must provide a discussion of future plans which include a description of the change in short and long term goals and objectives of the construction program as a result of program assessment (American Council for Construction Education). Furthermore, the ACCE accreditation process requires a discussion of specific plans for implementation of program changes identified through the assessment process.

The ACCE accreditation criteria for construction programs include varied sets of requirements. Important issues that must be addressed relate to mission, goals, current size, organizational structure, listing of near and long-term objectives, and how progress or achievement are to be measured. The Construction program seeking accreditation must provide information regarding intra-campus and multi-campus relationships with allied disciplines and summaries of the institutional and construction unit budget.

With regards to faculty, the institution must provide data pertaining to current staff and faculty and their assignments, faculty compensation, evaluation and promotion policies, and professional development activities. Specific items with respect to students include admission standards, quality of new students, enrollment data, grading system, academic success and failure, record keeping and academic advisement, student activities and graduate and placement data. Descriptions of laboratories, classrooms, staff offices, library, audiovisual services, computer facilities, and placement services are also required.

The self-study must cover relations with industry and a description of the advisory committee including their corporate affiliations and the type of construction activity in which they are involved, the advisory committee procedures, and the ways in which the advisory committee has assisted the construction unit. Furthermore, a description of work experience programs including cooperative education and summer job programs with an indication of the number of students and companies involved. A description of the placement assistance activities of the construction unit and number of companies recruiting are required. Finally, ACCE also requires a discussion of student-industry interaction including national construction association interaction, major field trips taken, and guest speakers.

It is clear that for a construction program to be accredited, it must meet many of the metrics and norms established by the ACCE. Consequently, for an assessment program to be useful and relevant, it must consider the specific areas identified in the ACCE guidelines. For example, placement data should be used as a measure of quality in the assessment program because it is also required by ACCE. Additional measures and norms can be developed using input from alumni, students, faculty, parents, employers, and advisory boards.  

The ABET 2000 Assessment Criteria

The Accreditation Board for Engineering and Technology (ABET) has followed in the footsteps of the American Council for Construction Education and developed specific metrics for assessing program quality. The ABET 2000 criteria requires that assessment programs be established and implemented. Specifically, ABET requires a total of 8 criteria (Accreditation Board for Engineering and Technology, 1997). Each program must have an assessment process with documented results. Evidence must be given that the results are applied to the further development and improvement of the program. The assessment process must demonstrate that the outcomes important to the mission of the institution and the objectives of the program are being measured. The specific criteria are described below (Accreditation Board for Engineering and Technology, 1997).

Criterion 1. Students

The institution must evaluate, advise, and monitor students to determine its success in meeting program objectives.

Criterion 2. Program Educational Objectives

Detailed published educational objectives that are consistent with the mission of the institution. Additionally, a process based on the needs of the program's constituencies in which the objectives are determined and periodically evaluated. Finally, a system of ongoing evaluation that demonstrates achievement of stated objectives and uses the results to improve the program.

Criterion 3. Program Outcomes and Assessment

Engineering programs must demonstrate that their graduates have an ability to apply knowledge of mathematics, science and engineering, and demonstrate competence to function effectively in a modern society. Specifically, the graduate must be able to conduct experiments and interpret data, be able to design a system to meet desired needs. Furthermore, the graduate must be able to function on multi-disciplinary teams, exhibit an ability to identify, formulate, and solve problems. Student is expected to have an understanding of professional and ethical responsibility, be able to communicate effectively, and have an understanding of the global dimensions of the profession. More importantly, future graduates must possess an awareness of the needs and importance of life-long learning and contemporary issues.

Criterion 4. Professional Component

Students must be prepared for engineering practice through the curriculum culminating in a major design experience. This may involve a senior design course incorporating engineering standards and realistic constraints as well as economic, environmental, ethical, safety, social, and political issues.

Criterion 5. Faculty

The faculty must be sufficient in number and must have the competencies to cover all of the curricular areas of the program. There must be sufficient faculty to provide appropriate levels of student-faculty interaction, student advising and counseling, university service activities, and interactions with industrial and professional practitioners, as well as employers of students. The faculty must have sufficient qualifications and must ensure the proper guidance of the program and its evaluation and development.

 Criterion 6. Facilities

Classrooms, laboratories, and associated equipment must be adequate to accomplish the program objectives and provide an atmosphere conducive to learning. Appropriate facilities must be available to foster faculty-student interactions and to create a climate that encourages professional development and professional activities. Programs must provide opportunities for students to learn the use of modern engineering tools. Computing and information infrastructures must be in place to support the scholarly activities of the students and faculty and the educational objectives of the institution.

Criterion 7. Institutional Support and Financial Resources

Institutional support, financial resources, and constructive leadership must be adequate to assure the quality and continuity of the engineering program. Resources must be sufficient to attract, retain, and provide for the continued professional development of a well-qualified faculty. Resources also must be sufficient to acquire, maintain, and operate facilities and equipment appropriate for the engineering program.

Criterion 8. Program Criteria

Those members of civil engineering faculty responsible for the upper-level professional program must provide evidence that they understand current professional practice in their specialty areas. The program must demonstrate that its graduates have the ability to apply advanced mathematics through calculus and differential equations. They must exhibit familiarity with statistics and linear algebra, knowledge of computational practices; competence in experimental design, data collection, and data analysis; and knowledge of chemistry and calculus-based physics with depth in at least one of them.

It is evident that for a civil engineering program to be accredited, it must meet many of the same metrics and norms identified by ACCE. Although the ABET criteria for accreditation varies considerably from the ACCE criteria, both criteria share the common requirement for all programs to develop substantive assessment plans. Consequently, it is critical that any assessment program consider the specific areas identified in the ABET guidelines.

The Philosophy of Outcome Assessment

The goal of any university is to have an exceptional reputation and produce graduates that are worthy of that reputation. How a university achieves recognition for being an excellent institution of higher learning is subject to debate. However, a menu for achieving prominence may include superior external relations, quality programs, quality faculty, quality facilities, quality students and an optimum enrollment. These elements are not mutually exclusive, but are interrelated as shown in Figure 1. Also, each component lends itself to being assessed against accepted norms and measures. The purposes of assessment should be to improve, to inform, and/or to prove. The assessment process should help determine whether specific objectives are being met. The process should also provide information and identify issues that affect a program and its future. Generally, the development of an outcome assessment program involves the following steps:

1. Develop a mission statement, identify goals, and what needs to be achieved.

2. Review ACCE and ABET guidelines and requirements.

3. For each goal, specific objectives should be identified.

4. A performance criterion (norm) for each objective should be established.

5. Identify outcomes for an accomplished goal.

6. Activities to be implemented to achieve the identified goals and objectives.

7. Appropriate measures for the achievement of goals and objectives.

8. Well-defined assessment and data collection plan.

9. Identification of appropriate and relevant national and university norms.

10. Mechanisms to modify practices and activities based on outcomes.

Feedback channels are an important component of the assessment plan (Sheehan and White, 1990). The feedback channels provide timely information to facilitate continuous improvement of practices and provide input for decision making. The final aspect of an assessment plan is the evaluation of whether or not the performance criteria were met and the objectives were achieved.

Since evaluation is the process of ascribing value to the assessment results, it usually occurs during the continuous improvement phase (formative evaluation) and at the end of assessment phase (summative evaluation), (American Council for Construction Education, 1998).

In developing a philosophy of outcome assessment, the constituencies served by a program must also be identified. These are employers, alumni, students, parents, faculty and staff, advisory boards, university administration, and appropriate accrediting agencies. Each of these constituencies has its own expectations, needs and desires.

Employers of the graduates of a program are excellent measures of the success of a program. Employers who repeatedly return to hire graduates of a program consider the program relevant to their needs. Some of the characteristics employers seek in graduates are work experience, leadership skills, oral and written communication skills, ability to use state-of-the art software relevant to the discipline, and a design experience (Engineering News Record, 1996). In engineering and construction, the employer would also consider passing the Fundamentals of Engineering or the Certified Professional Constructor examinations an important criterion for the hiring of a student.

One of the most important influences on a student’s selection of a university is the parent of the student. Considerations a parent will make include available financial assistance, opportunities for work experience, placement rates of the graduates of the program and their associated starting salaries, the availability of personal attention (measured by class size, tutoring availability, and faculty availability), the quality of the faculty and the quality of facilities.

Figure 1. Interrelationships between constituencies found in higher education.

The success of the alumni is another important measure of the success of a program. Successful alumni enhance the reputation of a program and, therefore, the reputation of the university. Consequently, alumni should be given opportunities for involvement after they graduate. Their input could certainly help improve the quality of the program and help insure that the program has quality facilities through donations and fund raising activities. Alumni involvement can be encouraged through recognition (honors and awards), service on advisory committees, and providing opportunities for visits to the campus.

The students who matriculate at an institution also have a variety of expectations. They want a good education (a quality program) and have an expectation of earning good grades. They want work related experience while they are in school and the guarantee of being employed when they graduate. They expect quality laboratories, quality computer facilities and the opportunity to interact with quality faculty. They seek scholarship funds and other forms of financial support and want to be recognized for their achievements. Finally, they desire safety, stimulating environment, and to have fun. It is the opinion of the authors that a significant portion of the college experience occurs outside the classroom. Involvement in professional societies, joining a fraternity or sorority, participating in intercollegiate athletics, etc. are extracurricular activities which help students develop needed leadership and people skills.

For any assessment program to be successful, it must have the support and active participation of the faculty. It is assumed that faculty members are typically very independent and want to be left alone so they can pursue their research and other professional interests. Preferably, they would like a low teaching load accompanied by a high salary. They also would like to be appreciated by receiving recognition and, perhaps, honors for their achievements. These factors must be considered when developing an assessment plan to insure that quality faculty are retained and the appropriate environment is maintained.

Each program should have an advisory board to help provide external perspectives on the quality and substance of the program. This is one way to involve alumni, employers of the program’s students, and other leaders in the workplace related to the discipline. It is important the advisory board be asked to provide input and play a role in achieving program objectives and stated assessment standards. Members of the advisory board also like to be recognized and appreciated for their participation. This expectation can be met through honors banquets and special reunions of alumni and friends of the department.

The final constituency of a program is the administration of the university. They desire to have the program have national prominence. They want high enrollment in the program and expect the faculty to be willing to accept low salaries. They also expect the program to raise funds to support its activities. The administration expects the consensus of the faculty in response to their directives. In all cases, they expect a program to meet the minimum standards required to receive accreditation. Furthermore, they expect an assessment of objectives and implementation of needed improvements.

Establishing Assessment Plan and Norms

Institutions in higher education need to develop assessment plans consistent with their missions and objectives while meeting accreditation requirements. As educators look to a future of change and increasing internal and external pressures, new paradigms for assessing quality will be necessary. Failure to reform and redefine from within will invite unnecessary change from beyond (Al-Khafaji, et al, 1998). Regardless, change is on the horizon. The performance of construction and civil engineering programs will not and should not be exempt from change. In fact, assessment programs are now mandated by accreditation agencies. Generally, an assessment plan may include a variety of measures and norms to evaluate the degree of success in meeting stated objectives. These measures may include

1. Number of students seeking admission to graduate or professional schools

2. Success on the Fundamental of Engineering Examination (FE).

3. Success on Professional license Examination (PE).

4. Placement Rate and ease of finding appropriate employment.

5. Salaries of graduates.

6. Student performance on standardized exams.

7. Student performance on senior comprehensive exams.

8. Senior theses.

9. Awards, honors, and fellowships received by students.

10. Awards, honors, and fellowships received by faculty.

11. Number of publications and research dollars generated.

12. Student to faculty ratio.

13. Student polls and questionnaires.

14. Employer polls and questionnaires.

15. Alumni polls and questionnaires.

16. Accreditation results.

17. Expenditure on equipment.

18. Faculty salaries.

19. Student course evaluations.

20. Faculty annual evaluations.

21. Success on the Certified Professional Constructors Exam (CPC).

Clearly these measures must be assessed relative to accepted standards and norms. Although many of these measures may be valid, a program of assessment must reflect the objectives and mission of the department for which it is intended. It is not necessary to measure all of the above nor measure them annually. The assessment program should be a systematic plan that links program goals and objectives to the mission statement (Western Association of Schools and Colleges, 1992). It should articulate clear statements of intended outcomes and describe the procedures to be used to assess whether goals are being met. More importantly, the assessment program must demonstrate how assessment findings are used for instituting needed improvements. Feedback should be should be an ongoing process and provide useful insights to the institution and student with faculty participation and support.

The department of civil engineering and construction (CEC) at Bradley University has developed and implemented an active assessment program as shown in Figure 2.

The CEC assessment program is viewed as comprehensive and has been used as a model for others to emulate. More importantly, it has impacted the CEC department profoundly by making it one of the largest departments on the Bradley campus. Enrollment increased from the lowest to the highest in the college of engineering. The degree of support received from industry as reflected by number of endowed and annual scholarships which has increased from 5 to more than 35 since the implementation of the assessment program. The CEC assessment program involves the following components:

1. Questionnaires to entering freshman.

2. Questionnaires to entering seniors.

3. Questionnaires to graduate students.

4. Questionnaires to alumni.

5. Questionnaires to advisory board.

6. Questionnaires to employers.

Additionally, the department holds an annual retreat to which the officers of the four student organizations in the department are invited. During this one-day meeting, each student and faculty is asked to raise issues and concerns that the department may need to address. The retreat continues to be a valuable instrument for faculty and students to solve problems that may otherwise go unnoticed. Over the years, students have become agents of change and the propulsion needed for continuous improvement.

Conclusions

Although it is difficult to develop a single model that fits all needs and requirements, it is helpful to consider the issues, norms, and metrics required for an effective assessment program. The most important first step is to develop a mission statement and a related set of realistic objectives taking into account needed resources for successful implementation. An effective assessment program must have a clear set of norms by which outcomes are measured and define how the conclusions are to be utilized or implemented.

Programs in construction and civil engineering will need more than policy manuals and clearly defined objectives. The programs must embody the concept of quality and excellence in all of its forms and at all levels. Hence, guidelines and clearly defined assessment plans are needed to foster the development of excellence among faculty, alumni, and students. More importantly, departments must develop appropriate instruments to implement needed changes identified through the assessment process. The academic community must broaden its thinking, examine changing technologies, consider global issues, define its mission, and establish an appropriate vision. The academic community must recognize that the demands and needs of external and internal constituencies are varied. It is time to move beyond the discussion and procrastination phase and ask what does it take to achieve excellence and maintain quality.

Figure 2. The Civil Engineering and Construction Assessment Program at Bradley University.

 

References

Al-Khafaji, A. W., et al., (September 1998). The Scholarship Landscape in Civil Engineering: A Bridge Between Rhetoric and Reality, Report of the ASCE Task Force on Redefining Scholarly Work, Reston, VA, 46p.

The American Council for Construction Education, Guidelines for a Self-Evaluation Study, form 102, 1998.

The Engineering Accreditation Commission of the Accreditation Board for Engineering and Technology, Engineering Criteria 2000, Third Edition, December, 1997.

Engineering News-Record (September, 1996). Shaking Up Education. Publication of the McGraw-Hill Companies.

Sheehan, B.H and J.A. White. (December 1990).Quo Vadis, Undergraduate Engineering Education, Engineering Education, 80, 8. ASEE.