In the fall of 2000 Auburn University made the transition from the quarter system to the semester system. In the preparation for this change the university administration mandated that all programs be limited to 120 semester hours in order that students could realistically expect to complete an undergraduate degree in four years. This caused many problems for the technical programs of the university, particularly for the construction management unit.

The construction management unit was required to reduce the equivalent of 136 semester units to the mandated 120 semester units and also meet all the accreditation requirements for ACCE. In order to meet this goal the unit eliminated all of the redundancies and some of what was felt to be necessary repetition in course content. One area that was particularly affected by this process was the structures sequence. Four quarter courses (Mechanics of Structures, Strength of Materials, Reinforced Concrete and Applied Structures) had to be compressed into three semester courses.

Mechanics of Structures and Strength of Materials were the two courses found to have most of the repetition in this sequence. It was common practice in these courses to provide a thorough review of the physics concepts used as a basis for structures at the beginning of both courses and a review of Mechanics at the beginning of Strength of Materials. Because of this repetition the structures faculty decided to combine these two courses and leave the Reinforced Concrete and Applied Structures courses essentially intact.

Past experience, however, indicated that this approach had problems. First, to cover material from two courses consisting of 50 contact hours each in one class of 45 contact hours, not only would the repetition have to be eliminated but the pace of the course would have to be very brisk. Second, the course would be based on physics concepts, which past experience indicated had not been well learned. Third, not all of the students would have had physics the preceding semester which would make the grasp of the basic concepts even more difficult. Fourth, this lack of fully grasping the background concepts would hinder the students understanding of the structural applications. And, fifth, there was not time in the single course to provide even a rudimentary review. Needed, therefore, was some method to provide a review of these structural concepts outside of class.

Inoue reports that a tutorial format of computer-assisted instruction is particularly suitable to teaching basic concepts (Inoue, ’99). Inoue also reports that the strengths of this format are that it explains the basic concepts, tests the understanding of the concepts and guides the student through a logical sequence in learning the material. Therefore, the structures faculty decided to use interactive computer tutorials to provide the needed review.

This paper will describe the development of the tutorials, the development of an evaluation questionnaire, and the results of the questionnaire. This paper will also discuss the results of the questionnaire and make recommendations for modification and future applications.

The purpose of the project was to provide a review outside the classroom of basic structural concepts from physics to prepare the students to better understand mathematical applications presented in the beginning structures class. The topics considered important were forces, trusses, stress, strain, Poison’s Ratio, temperature stress, stress concentration, center of gravity, moment of inertia, and beams. Also, to accommodate different learning styles, it was decided to present the concepts in writing, and through visual illustrations and video clips. Further, recognizing that different students react differently to different professors, a choice of video clips from different professors was offered for each topic.

Macromedia Flash was used to create the tutorials for use through Web-CT. Web-CT is an internet based course material presentation software program. The tutorials and quizzes work better on Web-Ct because it gives the students an opportunity to review them at their convenience and Web-Ct keeps up with all the grades and it is only a one-time effort for the instructor. The interactive nature of the tutorials would also be difficult to duplicate if the same were done on paper. The tutorials were divided into three sections: an interactive tutorial, video clips of explanations, and a unit quiz. Web-CT also provided the means to record quiz performance and time spent on each tutorial. The interactive tutorial consisted of simple, interactive, context sensitive, often humorous, multiple-choice questions. Each question was accompanied by an illustration (see figure 1). Any choice would take the student to a screen with feedback on the student’s choice at the top, and an explanation of the concept and a picture illustrating the concept at the bottom (see figure 2). The oral explanations were provided through video clips of professors in an office setting. These video clips were created in real media and were placed on a streaming server to enable fast downloads. A multiple-choice quiz was given at the end of each unit with immediate feedback on the student’s performance.

Each tutorial was designed to be completed in 15 to 30 minutes including viewing of video clips. To encourage the use of the tutorials it was decided to have the unit quizzes comprise 5% of the student’s term grade. The students were given a schedule when the tutorials should be viewed and the quizzes taken for credit prior to that subject being covered in class. After the scheduled end date the tutorials could still be viewed but the quizzes could no longer be taken. A missed quiz would result in a grade of 0. The quiz grades would be recorded and saved by the Web-CT program.

Inoue reports that there is no formally validated method to evaluate the performance of a computer-assisted instruction tutorial. Therefore, the professors developed their own evaluation strategy.

Four areas were considered to be of particular importance in evaluating the tutorials: the reaction of the students, quiz performance, time spent on the tutorials and a comparison of grades between the semester course and the quarter courses.

A specialist in questionnaire development and evaluation from the University’s College of Education was employed to develop and evaluate a questionnaire to determine the reaction of the students. The specialist attended the class to learn format and methodology, and interviewed students and professors to determine concerns. Using this information the consultant developed a questionnaire consisting of 42 questions, most in a multiple-choice format. He tested the validity of the questions and eliminated 4. The first 16 questions measured the students’ opinion of the overall quality of the web site, the quality of the tutorials, the quality of the video clips, and the quality of the tutorial quizzes. Question 17 gave the students the opportunity to type in comments. The next 15 questions were used to get a response on the general concerns that the professors and students expressed during the interviews and to validate the students’ responses to the first 16 questions. And, finally, the students were asked if they would have completed the tutorials and quizzes if the quiz results had not been part of the course grade. The questions are given in Appendix A.

The survey was administered using Web CT. The program recorded that a student had completed the questionnaire but did not correlate responses with individual students so that confidentiality could be maintained while allowing extra credit to be given to encourage participation.

A comparison of the percentage of letter grades of the semester course with those of the quarter courses was used to provide a preliminary, rough evaluation of effectiveness. The total number of students taking the courses during the last year of quarters and the first year of semesters was tallied according to course grades and percentages were calculated based on the total number of students for the year.

The questionnaire was completed and ready for use in Fall Semester, 2001.The data for students’ reaction, quiz performance, and time spent on the tutorials was limited to one section of 26 students. Also, the only year of data for student grades for the semester course was the transition year. In this year it was anticipated the unsettled atmosphere and necessary changes to the courses would have an adverse effect on student performance. It was decided that an evaluation at this time would give a preliminary indication of the success of the project, but that more data would have to be collected for a definitive evaluation.

Twenty-four of the twenty-six students in the course answered the survey. The size of the data was relatively small and the data was collected from only one group. A close inspection of the data revealed that the data conformed to a gaussian population. Since the data collected was being used to describe only one group, mean and standard deviation were considered sufficient for statistical evaluation. The mean and standard deviation for all pertinent questions were calculated. A detailed statistical analysis including correlating the students’ responses was not performed because the survey had only twenty-four respondents. After conducting the survey for at least another year, the authors expect to make a more elaborate statistical analysis.

As stated above, four specific areas were identified as important for evaluation: the reaction of the students, quiz performance, time spent on the tutorials and a comparison of grades between the semester course and the quarter courses. The students’ reaction was determined through the questionnaire. Students were asked to rate the tutorials, quizzes and the videos on a scale of 5 (5 for most effective and 1 least effective). The student response tabulated in table 1 below indicates that the online videos were the least effective among the three.

The same four questions were asked to the students about four specific areas: The interactive tutorials, the online quizzes, the online videos and the overall web content. Their responses to these sixteen questions are summarized in Table 2. Each question was on a scale of one to seven, one for most favorable answer and seven for least favorable answer.

Furthermore the students were encouraged to write comments about the questions. The comments presented in Table 3 are summarized from the written responses. Similar comments were grouped together and counted.

These results from tables 1, 2 and the comments in table 3 indicate that the students often completed the tutorials, quizzes and videos. The results also indicate that the quizzes and tutorials were more helpful than the videos, and that they felt that the material was better covered in the quizzes and tutorials than in the videos. As can be seen, the students believed the basic objective of the tutorials was met; that is, the tutorials helped in the review of the basic concepts. However, the students suggest that improvement could be made to video content, to the depth of the material in the tutorials and to the quiz questions. They also indicated that it would be beneficial to cover mathematical procedures.

Fifteen questions were used to validate the students’ responses to the preceding questions and to address general concerns of the students and the professors. The results for these questions are in Table 4

The responses confirmed that the material could have been covered in more depth, that math calculations should be included, that the videos needed improving, and that most of the students found the quizzes to be fair and appropriate. The students did not feel strongly that the tutorials helped course performance. Most students felt that the amount of time spent on web-based content was reasonable and the material itself was easy to follow. The students reported that five percent of the course grade for on-line quizzes was reasonable. Only half of the students reported that they would have completed the online quizzes if they were not a part of the grade.

Web-CT allows the course instructor to measure certain web statistics such as the total number of hits per page, the total amount of time spent by all the students in a particular page and the average amount of time spent by students per hit. These statistics along with the number of students taking each quiz and the average for each quiz is presented in Table 5.

The mean and standard deviation for the number of hits are skewed due to the fact that students visited the first two tutorials more than the others. The authors believe that this may be attributed to the students getting used to the system. If the first two tutorials are ignored, the average number of hits was 60.82 (SD 10.25) and the total time spent was a mean of 3:36 hours (SD 1:30 hours). With an overall average of just under 4 minutes per hit, the tutorials took much less time than was anticipated. This data also indicates that most students reviewed the tutorials at least twice before taking the quiz and that most of the students completed the quizzes. The quiz average, 3.95 out of a possible 5, indicates that the tutorials were successful in providing a review of the material.

A preliminary evaluation of the comparison of the class grades of the semester and quarter system are presented in table 6. The grades are from the last year of quarter system and the first year of the semester courses.

As can be seen, the preliminary indication is that the tutorials are equally successful in providing the needed review as the in class review provided in the quarter courses. It is possible that with some modification of the tutorials, student performance will show that the computer tutorials provide a superior review to that done in the quarter courses.

In response to the need to provide a review of basic principles from physics for a new structures course, a series of computer-based tutorials was developed for use outside of class. A preliminary evaluation of the tutorials was conducted using a questionnaire, tutorial generated data and a comparison of grades made in the new course with those of the old courses. The preliminary results indicated that, while some improvement was needed, the tutorials were successful in providing a needed review of basic principles. Suggestions for improvement of the tutorials included greater depth of coverage of the topics, the addition of mathematical procedures, an improvement to the video clips and a revision of quiz questions to make them more easily understood. Because of the limited population involved, however, a definitive evaluation cannot yet be made. It is recommended that these revisions be made, the study be continued to provide a definitive evaluation and that use of this tool as a method of providing review from pre-requisite courses be further investigated.

Inoue, Yukiko (1999). Evaluating Intelligent Tutoring Systems, Information Analysis, 70, 1-9.