A New Paradigm For Assessing Teaching Effectiveness

 

Amir W. Al-Khafaji and F. Eugene Rebholz

Bradley University

Peoria, Illinois

Introduction

A unique model is proposed to link teaching with the equally important values of excellence, integrity, and leadership. It is the pursuit of excellence that drives institutions and faculty alike (Diamond and Bonwyn, 1993). It is the responsibility of leading educators to provide a useful contemporary guide for faculty reward and recognition in this important area (Al-Khafaji, et al, 1998). In turn, institutions need to place less emphasis on definitions and more upon creative means of rewarding substantive teaching achievements.

For the purpose of assessing current practice relative to instruments used in measuring teaching effectiveness, a select and varied number of Civil Engineering departments submitted documentation relative to their policies and procedures (Al-Khafaji, et al, 1998). While the submitted documents vary significantly, the main instrument used in assessing teaching performance tends to be based solely upon past student course evaluations. In all cases, a sustained and solid performance in teaching is expected. The question is how to measure it fairly and equitably? Clearly, several other fundamental questions can be raised relative to the assessment process including:

1. How to assess easy graders relative to tough graders?

2. How to assess teaching graduate courses versus undergraduate courses?

3. How to reward those incorporating new technologies?

4. How to deal with those who consider teaching as popularity contest versus those who consider it a sacred obligation?

5. How to inspire faculty to give their best in teaching?

6. How to take advantage of the senior faculty experience in helping new faculty?

7. How to use outcomes in bringing about required changes?

Although there are other valid questions that could be raised (Al-Khafaji, et al, 1998), the seven questions listed above are fundamental to any quality program of assessing teaching. Over the years, our department has established itself as a model of excellence for others to emulate by addressing these fundamental questions. Hence, adopting new approaches and discarding flawed ones has become the hallmark that defines its faculty.

A Unique Model for Assessing Teaching Performance

The development of sound policies and procedures, however they may vary from one institution to another, must be applied equitably and with consistency (Boyer, 1990, Diamond and Bronwyn, 1993). The inference is that such policies and procedures exist which is not always the case. More than 10 years ago, the faculty and the new chair in our department developed a unique model for assessing teaching effectiveness. The new approach is focused more on "trends" and "continuous improvement" and less on just "bean counting." The most provocative features of this model includes

4. Formation of a Teaching Excellence Committee.

5. Development of a Systematic Approach of Assessment.

(1)

(2)

Finally, using just the normalized score may be unfair to those who are truly good teachers and have good students. Therefore, a composite score is determined by multiplying the average students course evaluation by normalized score. Thus,

(3)

Although the proposed model requires more effort in assessing teaching performance inside the classroom, much of the work can be computerized using spreadsheets. Also, there is more to teaching effectiveness than just measurements based on student course evaluations. Additional activities used to assess overall teaching are discussed in the next section. It should be noted that the proposed model for assessing teaching performance is a part of a larger model dealing with faculty work that includes scholarship, service, and professional development (Al-Khafaji, et al, 1998). However, these areas are beyond the scope of this article.

Teaching Activities

In the CEC department at Bradley, teaching varies from 45 to 55 points from the total of 100 points that can be achieved by faculty. The remaining 45 to 55 points are awarded for scholarly activities, service, and professional development. The new model defines specific activities that are deemed appropriate and consistent with the institutional and CEC departmental missions. These activities are described below.

Laboratory/Course Development

New courses or laboratories introduced as well as improvements made to existing courses are considered. One may incorporate microcomputer applications in a course, improve experiments for a laboratory, develop expertise and a special topics course in a field which might ultimately enhance the curriculum departmentally or college wide, etc. For example, the use of Internet instruction is considered vital contributions provided they are approved ahead of time by the chairperson.

Student Evaluation

This is limited to the feedback the chairman receives from the students through classroom student evaluations and does not include comments made directly to him. The basic factor considered herein is the ability of a faculty member to respond positively to particular problem areas identified and discussed with him by the chairman the previous year. 'Trend' is the key word and not the absolute average number received on student evaluations. The long-term objective is to make each faculty member valued and unique in their own way.

Teaching load

Faculty members who have release time will be penalized under this category. That is, the lower the teaching load, the lower the grade will be. The reason is that a faculty member with release time will be better able to develop professionally than one whose teaching load is excessive. However, class size, number of sections taught, and number of graduate courses are all-important factors that are considered in assigning points.

Commitment

The chairperson will consider the currency of the materials being taught, ability of a faculty member to address specific deficiencies identified. These may include course outlines, grading schemes, office hours, absences, etc.

Assignments

Each faculty member is expected to provide copies of his homework assignments and exams to the departmental secretary. This is important for future accreditation visits and the work of the teaching excellence committee. Homework problems are to be changed periodically.

Future Plan

Faculty members are asked to provide their future plan annually. Based on the merit of the plan, faculty will be graded. This will be true even if the faculty member is an outstanding researcher. The plan must meet specific meaningful objectives consistent with the mission of the CEC Department.

Other activities may also include teaching proposals submitted, funding received for teaching, and invited lectures (Boyer, 1990, Engineering News-Record, September 1996, Hall, Focht, Michael, Paulson, Saville, and Lowe, 1988). Additionally, the chair considers awards received by faculty members from the university, student organizations, and alumni organizations when assessing teaching effectiveness. The most important aspect of the evaluation process is the flexibility given to faculty to actually decide the total number of points to be used in assessing their performance, provided it is within the 45 to 55 points range. Additionally, faculty may decide to include at their own discretion additional activities such as reviewing textbooks, serving on a review panel, offering special courses and seminars on Professional License Examinations, and reviews for Fundamental of Engineering exams. Such activities are called interface activities and are critical to any department that is wedded to the idea of pursuing excellence in all of its forms irrespective of what that activity is called. The inherent flexibility in the model presented makes it possible to avoid the rigid format that is often dictated to the faculty. Rigid criteria tend to stifle faculty innovation and creativity.

The Teaching Excellence Committee

The department requires that all untenured faculty members attend special committee meetings on regular basis. An outstanding faculty member with a proven teaching record chairs the committee. This committee provides the senior faculty with the opportunity to interact directly with junior faculty and to help them become productive much faster than normal. Unquestionably, it is an extraordinary instrument for sharing the wealth of experience embodied by the senior faculty to help students receive the quality education they need. The committee also helps create a positive environment where junior faculty members realize that their success and accomplishment in teaching is a matter of departmental policy. The committee holds regular meetings to address issues pertaining to

1. Grading practices.

2. Course outline development.

3. Textbook adoption.

4. Issues pertaining to deficiencies and required corrective measures.

5. Student course evaluation issues and concerns.

The chair of the committee invites excellent faculty members to share their experiences and answer questions on regular basis. Additionally, the committee often has open discussions relative to the suitability and frequency of homework assignments.

Self Reporting

With the goal of seeking continual improvement and excellence in classroom teaching, the Department of Civil Engineering and Construction conducts student course evaluations for each class offered. The chair of the department then prepares a summary sheet outlining faculty performance in each course. The summary sheet provides overall averages relative to each student and to each question asked. That is, each question has an average representing the entire class and each student produces an average. A sample course evaluation form is shown in Table 1.

The student course evaluations are conducted by the departmental secretary and are sealed until after the final grades are submitted. The specific questions used in this evaluation are summarized below.

Table 1.

Q1. Organization

Q2. Preparation

Q3. Presentation

Q4. Student - Instructor Interaction

Q5. Interest in Students

Q6. Assignments, Tests and Grading

Q7. General Evaluation of Instructor

Q8. Instructional Materials

Each faculty is given copies of the evaluations with student written comments to help them develop a course of action to improve identified weak areas. The chair also provides average scores, normalized scores, composite scores, and rank of faculty within the department. The top three faculty members are recognized in a departmental meeting as a reward for their effort. Additionally, the announcement helps weak faculty identify colleagues whose guidance and experience is of benefit.

Historical Trends and Feedback

It is clear that for the evaluation process to be fair and meaningful, it must provide insights and not just numbers. Consequently, issues and concerns raised by students is a continuing process of renewal and change. This process includes faculty initiatives in addressing deficiencies with supervision from the teaching excellence committee and the chair. Also, officers from each of the four student organizations in the department are invited to an annual retreat to address specific concerns pertaining to specific courses and other pertinent curricular matters. The annual CEC retreat provides a wonderful forum for students and faculty to deal with problems as a team with total objectivity and candor. As a consequence of these deliberations, many positive developments have occurred including the addition of new courses, different course offerings, and several course content changes. This process has sent an unmistakable message to faculty and students alike that the assessment process is taken seriously. The consequence is that students and faculty members understand that the focus is definitely not the popularity of the instructor but the quality of his instruction.

In order to develop a suitable perspective on each faculty teaching performance relative to the entire department, an annual summary sheet is given to each faculty indicating their rank. Additionally, during the annual review process, the chair prepares several graphs showing all of the relevant measures versus the departmental average for the past five years. A graph for a senior faculty average score versus departmental average is shown in Figure 1.

Figure 1. Average student course evaluation for a senior faculty versus departmental average for five years.

The graph clearly shows a negative trend during the 1995-96 academic year. In this particular case, the faculty member was experiencing difficult problems due to heavy teaching loads and added responsibilities. These problems were related to the sudden resignations of several valued faculty members. Consequently, the chair took immediate steps to relieve this faculty’s burden producing a positive move for the better during 1996-97 and beyond. Note that in this case all measures were down during 1995-96. That is, the normalized and composite scores were also lower than the norm for this faculty member. Clearly, the evaluation should not be focused on individual courses, a given semester, or even a specific year but must reflect the contributions over an extended period of time. The key word to describe faculty teaching performance is "TREND". If the trend line for a faculty member points downward over a period of several years, then drastic action is required and is normally taken including termination.

Reward System

The department of civil engineering and construction has adopted a numerically based policy for determining the merit salary increase each year. The salary increase is composed of merit and adjustment for salary compression. The merit is determined in terms of the average faculty evaluation achieved on the annual assessment while the gap raise (to account for salary compression) is determined by the chairperson and the dean of the college.

The merit raise for a given faculty is based on faculty performance as described by the faculty assessment policy described in a CEC departmental policy manual. The student course evaluation score as reflected by the composite score has a direct impact on the merit salary increase and overall evaluation. A total of 35 points out of 100 are assigned to this category. Furthermore, it has a profound impact on tenure and retention decisions. Finally, the top three faculty performers are nominated by the chair to the CEC Advisory Board with the purpose of selecting one as the winner of the outstanding faculty award winner for that year. The award winner receives a plaque and $500 presented at the annual CEC Honors Banquet held before more than 250 alumni and supporters.

Conclusion

Teaching effectiveness is often considered in a very subjective manner. Student evaluation rankings are often utilized in programs, but there are some obvious weaknesses in using only this method. Common criticisms are that a faculty member may be awarding high grades or using other popularity factors to influence the rankings. These abuses make it difficult to get meaningful quantifiable data for assessment.

This paper offers a method to more accurately assess teaching effectiveness. To measure teaching ability, a model has been introduced that normalizes the student course evaluation scores. This compensates for the grades assigned in a course and lessens the problem of grade inflation. The method is adjusted for the obvious difference in grading expected for graduate level courses when compared to undergraduate courses. Teaching evaluations also need to consider other areas in addition to student evaluations. They include development of new courses or labs, teaching load, commitment, follow through with assignments and exams, as well as written future plans in the area of instruction.

It is especially important to help new faculty with the necessity of good teaching, as they are often hired immediately after attaining the Ph.D. Such faculty members have an immediate interest in research because of their most recent educational experience, and have seldom had total responsibility for teaching a course. For faculty who are not tenured, required participation in a teaching excellence committee has proven to be very profitable. A senior faculty member who can be a role model for teaching excellence chairs the committee and meets regularly with the group to address common issues such as course outline development, grading practices, homework expectations, and other concerns in teaching. This can be an enormous help to new faculty who are suddenly thrust into the role of being a teacher.

This paper also offers suggestions to benefit the experienced faculty member in their quest to excel. Each faculty member makes a self-reporting critique after examining the student evaluations. The faculty makes a response for each course taught. The response, as a minimum, includes addressing the lowest rated categories as well as improvement plans for the future. Comparisons are provided for each faculty member on their evaluation scores versus the department as a whole.

The issue is obviously improving quality rather than just popularity. Students sense this and appreciate it. Junior faculty can use it as a tool to get guidance, since the highest rated teachers are publicized. Finally, senior faculty also receive encouragement to realistically examine their accomplishments and needs by comparison to others as well as what the trend in their teaching is.

By joining an assessment of all these teaching areas together with the impact of recognition, incentives, and rewards, teaching methods are bound to improve for the benefit of all. It is important that the method used will help improve teaching rather than merely playing with numbers or intuition. This search for excellence in teaching then becomes the driving force for the faculty and the institution alike.

References

American Society of Civil Engineers, (1995). Summary Report of the Civil Engineering Education Conference, ASCE (Sponsored in cooperation with the National Science Foundation, Denver, Colorado.

Boyer, Ernest L., (1990) Scholarship Reconsidered: Priorities of the Professorate. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.

Diamond, Robert M. and Bronwyn E. Adam, (1993) Recognizing Faculty Work: Reward Systems for the Year 2000. San Francisco: Jossey-Bass.

Engineering News-Record (September, 1996). Shaking Up Education. Publication of the McGraw-Hill Companies.

Hall W. J., Focht J. Jr, Michael H., Paulson B., Saville T., and Lowe, (1988). Civil Engineering in the 21st Century: A vision and a challenge for the Profession. Prepared by the Task Committee to plan conference on Civil Engineering Research Needs. Produced by the American Society of Civil Engineers.