by Charles Weibel (May, 1999)

  • Other effectiveness studies
  • Average Final Exams Scores
  • Class formats
  • Placement data
  • First semester
  • Second semester

    In the last few years, the Math department at Rutgers has changed the way it teaches Calculus. This is especially true in the Calculus sequence for the Mathematical and Physical Sciences, Math 151-152.

    This is the first of two studies undertaken by the Mathematics Department, as an attempt to determine the relative effect of the new formats. We restrict it to a comparison of the different formats in which we teach Math 151-152. A second report will compare these formats to the old lecture/recitation style we had traditionally used, using data from the Calculus sequence Math 135-136.

    Since 1996, we have taught Math 151-152 in 3 comparable formats: the standard 4-credit sections (taken by most students), the 5-credit sections with attached "Practicum" (numbered Math 157-158) and the 6-credit "Excel" sections (which are actually Math 153-154).

    To measure the effectiveness of these different formats, we used the common final exam scores for each semester. This test was taken during finals week by almost all students in these courses; the small percentage who needed a make-up test were excluded from our data. In both semesters, the final exam was scored on a scale of 0-200 points.

    Here is a brief description of what we found. We will quantify these conculsions below.

    Average Final Scores

    In Fall 1997, the final exam was taken by 700 students and had a mean score (average) of 116 points; 199 students received an 'F' on this exam. There was no significant difference between average scores of the 4-credit sections and the 6-credit (EXCEL) sections (114 versus 116), but the average score for the 5-credit Practicum sections was 125, which is significantly higher.

    In Spring 1998 the final exam was taken by 540 students and had an average score of 131 points; only 95 received an 'F' on this exam. A large part of the reason for the different student profile in Calculus II is that many of the weaker students in Calculus I were filtered out; the average final exam score in Math 151 among continuing students was 128, not 116.

    We found clear differences in performance between students' scores on the Math 152 final exam in Spring 1998 (see Table 1). The differences between the means of these groups is significant at the usual 95% confidence level, because the Standard Errors (the standard deviations of the means) in this table are much smaller than the differences between the observed means.

    Class		# of   Average(mean)	    QUARTILES	    Standard
    Format		Exams and Std. Error	First Median Third  Deviation
    ------		----  -------------	----- ------ -----  -------  
    All sections	540	130.6 (1.7)	 107   136   165      40
    Standard	378	127.5 (1.9)	 102   130   156      37
    Practicum	120	134.8 (3.3)	 112   139   167      36
    Excel (154)	 42	146.5 (5.0)	 130   152   167      33
    Table 1. Final Exams from Math 152 in Spring 1998

    In both semesters, there is an obvious point to be made: this analysis of average scores is suspect because it does not take precalculus skills or aptitude into consideration. This information was available to us in the form of placement information.

    The Formats used in the courses

    Until recently, all Calculus at Rutgers was taught in a traditional 4-credit format, consisting of two 80-minute lectures and one 55-minute recitation each week. Most sections of Calculus 135-136 are still taught that way, but this is no longer the case with Math 151-152.

    In 1990, Rutgers initiated a 6-credit pilot program called EXCEL for first-year calculus students. Based on a program at Berkeley, it replaces the one recitation with three full periods per week devoted to workshops. The workshops involve collaborative learning in small groups of 3-4 students. In addition, students are now being asked to write up expository solutions, explaining in complete sentences how they arrive at their answer.

    EXCEL was an immediate success. Not only did we notice an increase in grade point averages over several years, but we also observed fewer losses in the number of technical majors. In addition, other departments have supported EXCEL because they have found that EXCEL students were better able to engage in collaborative learning projects. Since 1990, EXCEL has become a standard course offering, and given the designation Math 153-154.

    Encouraged by the success of the EXCEL program, the Math department changed the way it teaches Calculus, starting in Fall 1995. In order to keep the 4-credit course load, each section of Math 151-152 meets 3 times per week, with two 80-minute periods being devoted to traditional lectures. In the third period, increased from 55 to 80 minutes, the students meet in a recitation/workshop format. These classes are led by a teaching assistant (TA) and a carefully selected undergraduate, called a Peer Mentor. Typically, the first 20 minutes are spent going over the routine homework problems. In the remaining hour, the students break up into small workshop groups (of 3-4 people) and attempt more challenging problems, which are distributed as a handout. One of these workshop problems is assigned to be written up in the next week, and graded for both mathematics and exposition.

    Starting in Fall 1996, a 5-credit version of Math 151-152 was created. Again, two 80-minute periods are devoted to traditional lectures. In addition, the students meet twice a week in 55-minute classes. One class is a traditional recitation, in which a teaching assistant (TA) goes over the routine homework. The other 55-minute class is held in the workshop format described above, and run by the Lecturer and a Peer Mentor. We will refer to this as the "Practicum" format, since one 55-minute class each is afflicted with a separate course number (Math 157-158) and the name Practicum by the University Registrar.

    Placement Information

    We had a large database of placement scores for first-year students, which we were able to combine with the final exam scores for the first and second semesters of Calculus.

    In order to take Calculus I at Rutgers-New Brunswick, all students must either pass the New Brunswick Precalculus placement exam (the NBPRE) or else have passed a precalculus course. The NBPRE is scored from 0 to 40; a score of 19 or higher qualifies them to take calculus. We had the NBPRE scores for all first-year students, but not for all upper-class students. The average NBPRE score was 27.0 for the first-year students in our sample. About 1% of all first-year students had lower NBPRE scores, but were allowed into Math 151 if they had taken Precalculus over the Summer.

    In addition, our database contained admissions information. We had both the Math SAT (MSAT) and Verbal SAT (VSAT) scores for all first-year students in our database, scored from 200-800. We also had their High School Rank (HSR), measured as a percentage from 0-100, their gender (male/female) and school of admission.

    The average MSAT score in our sample was 650; almost all students in Math 151 (excluding EXCEL) had an MSAT of over 580. To no one's surprise, we determined that the VSAT score was not a significant predictor of the final exam scores. The average high school rank was HSR=81.4%, with the first and third quartiles being 72.8% and 93%, respectively.

    For Fall Semester, there was little statistical difference between the placement scores of the first-year students taking the standard sections and the practicum sections. This is illustrated in Table 2. However, the profiles for the students in EXCEL were different. This was no accident: selection into EXCEL is done by each College (primarily Douglass College or the Engineering College) on the basis of these numbers.

    Class Format	# FY	 NBPRE MSAT HSR
    ------		-------	 ----- ---- ----
    All sections	605	 27.04 650  81.4
    Standard	408	 27.43 653  82.1
    Practicum	152	 27.12 664  81.6
    Excel (153)	 45	 23.30 570  74.3
    Table 2. Placement statistics for first-year students in Fall 1997

    The First Semester Results

    In Fall 1997 we had final exam data for 689 students, including 605 first-year students. The mean for all students was 116.4 points, and the mean for first-year students was 120.8 points, not a great difference. Because we had available the admissions data for first-year students, and because this information was correlated with final exam scores, we decided to report our statistical analysis for first-year students. The analysis of all students was also done, but the results are qualitatively similar and less complete.

    An analysis of these scores yields the following table.

    Class		# Exams	    Mean	Std.Error Standard
    Format		 ALL(FY)   ALL(FY)	 ALL(FY)  Deviation
    ------		-------	 ------------	---------  ------- 
    All sections	689(605) 116.4(120.8)	1.7(1.8)   44(43)  
    Standard	478(408) 113.6(118.7)   2.0(2.2)   45(44)  
    Practicum	165(152) 124.8(127.8)	3.1(3.0)   39(37)  
    Excel (153)	 46 (45) 115.8(116.1)	6.0(6.1)   41(41)  
    Table 3. Final Exams from Math 151 in Fall 1997

    Comparing the Standard Error with the means, we see that only the mean for the Practicum sections is significantly higher. However, this does not mean that there are no differences -- because the mean does not take a student's prior skill level into consideration.

    Using only the NBPRE placement scores for first-year students, a regression for the final exam score F151 yielded the best linear fits:

     FORMAT		Least squares equation		NBPRE=23  NBPRE=27 NBPRE=31
    ---------	-------------------------	--------  -------- --------
    Standard:  	F151 = (4.87)NBPRE - 15.3	  96.7	   116.2    135.7
    Practicum: 	F151 = (3.87)NBPRE + 23.2	 112.2	   127.7    143.2
    Excel:     	F151 = (4.16)NBPRE + 18.3	 114.0	   130.6    147.3

    After adjusting for the placement scores, we see that weak students did much better (16-18 points better) in the Practicum and Excel sections, with the students in EXCEL doing slightly better. The same is true for students with average and high placement scores.

    Here is a scatter plot of NBPRE placement scores versus 151 final exam scores.

    PRECAL vs. 151 Final

    Figure 1: PRECAL placement scores versus 151 Final Exam scores

    This model is one more confirmation of the success we have seen in EXCEL, and which has been documented before. Because the number of students in EXCEL was so small, and had a different profile, we excluded EXCEL (Math 153) from the rest of the study. Since 45 of the 46 students in EXCEL were first-year students, the main study which excluded Math 153 contained 560 first-year students. Out of these, we had full placement information for 545 students.

    Finding the Best Predictor

    In our study, both the New Brunswick Placement Exam (NBPRE) and the Math SAT scores (MSAT) were highly correlated with the final exam scores; their correlation coefficients were .51 and .41, respectively. This suggests that we use a combination of the NBPRE and MSAT scores as a predictor of the final exam scores. This is supported by research studies which show that both the NBPRE and MSAT scores are indicators of performance in Calculus.

    One such study is "Prediction of Grades in College Mathematics Courses as a Component of the Placement Validity of SAT-Mathematics Scores," by B. Bridgeman and C. Walker of ETS (published by the College Board in 1990). In their study, they found that local placement tests like the NBPRE have a good correlation (about .5) with Calculus grades. In contrast, the MSAT, which is only a general measure of mathematical aptitude, only had a correlation of about .35 with Calculus grades.

    The NBPRE and MSAT scores are not independent variables; the correlation between NBPRE and MSAT was .49 for the first-year students in our study. In addition, the MSAT is a factor in admission to Rutgers, while the NBPRE scores are used in placement into Calculus and Precalculus classes at Rutgers-New Brunswick.

    We performed a Factor Analysis on the variables F151, NBPRE, MSAT and HSR in order to determine if there was a better indicator variable. It turned out that there were two significant factors, with eigenvalues 2.1 and 1.0. The most significant factor was:

    (F151, NBPRE, MSAT, HSR) = (.78, .28, .82, .87).

    The secondary factor was essentially the HSR. To test the effect of this factor, we defined two new predictor variables,

    NEWP = (0.87)NBPRE + (0.82)MSAT + (.28)HSR,
    REGR = (3.37)NBPRE + (1.69)MSAT + (.45)HSR.

    This had correlation coefficients of .56 and .55 with the final exam scores F151 of the first-year students, compared with the correlation of .52 for the NBPRE alone. We confirmed by visual inspection of the scatter plots of the data that there was not a real difference between working with NBPRE of the factors NEWP or REGR. Apparently this was because 90% of the scatter was absorbed by the NBPRE.

    Lecture or Practicum?

    We also ran a more sophisticated stepwise linear regression, using the t-test to decide which variables are significant. This showed that the High School Rank (HSR) was just as important as MSAT scores in predicting final exam scores. To determine the effect of the format, we introduced the indicator variable LORP, defined to be zero for Standard sections and one for Practicum sections. The best linear fit was:

    FINAL = (9.0)LORP + (3.37)NBPRE + (.45)HSR + (0.169)MSAT - 121.2

    In summary, given identical placement data, a student in the practicum sections could expect to score about 9 points higher on the final exam.

    The Second Semester Results

    In Spring 1998 we had 543 students take the Math 152-154 common final exam. In addition to placement information, it was important to know which Calculus course they had taken in the previous semester. There were 115 students did not take Calculus I in the fall, 22 who had taken Math 135, the other Calculus I class offered by Rutgers, leaving only 406 students who had taken Math 151-3 in the Fall. Their scores on the final exam (F152) are summarized in the following table.
      Two-term   #    F152	std.   std.   Two-term   #    F152   std.   std.
      Sequence Exams  mean	error  dev.   Sequence Exams  mean  error   dev.
      -------- -----  ----	-----  ---    -------- -----  ----- -----   ---
       any-152: 543	  131	 1.7   39	151-154:   21  145.4   6.8    31
      none-152: 115   125	 3.5   38       153-154:   24  146.5   6.7    33
       135-152:  22   129.6	 7.7   36       153-152:   26  141.5   7.1    36
       151-152: 406   134	 1.9   38
    Table 5. Final exams in Spring 1998 by Course Sequence

    Clearly the students who had not taken Calculus I in the previous semester ("none") did not fare as well. Students who had taken Math 135 also had lower scores, but this was not statistically significant. Since it seemed clear from the data (and experience) that the performance in the Calculus I course was significant, we excluded these students from further study.

    Next, we analyzed the data for the students in Math 154 (EXCEL). Of these, 24 students had taken Math 153 (EXCEL) in the fall, and 21 had taken Math 151 (17 took the 4-credit, and four students took the 5-credit version of Math 151). One student had taken Math 152 in the Fall, and did very poorly on the final, so we excluded him as an outlier. Faced with these small numbers, we had to content ourselves with general trends.

    For these 45 students, the average score was 145.9, significantly higher than the scores of students who had never been in EXCEL. This included 6 students who flunked the final exam (scored below 95 points); all of them had also flunked the final exam the previous semester. The 21 who came from Math 151 into EXCEL averaged 150 on the final, with no significant correlation to their performance on the 151 final exam. The 24 who took EXCEL both semesters had an average final score of 146.5, and there was a significant correlation with the Math 153 final score (F153):

    	Least squares equation		F153 score: 100;  116;  130;  150
      EXCEL 154 FINAL = (0.58)F153 + 79.2	            137.2 146.5 154.6 166.2
    By visual inspection of the scatter plot, it appeared that this equation gave a good fit for the best students, and was not skewed by the six poor students in the sample. The slope of this line was similar to that of the least squares lines for Math 151-152 below, reflecting the reality that the final exam for Calculus I is a good predictor of the final exam score for Calculus II.

    When we restricted to first-year students with full placement data, who took Math 151 in the Fall, and excluded the 45 students in EXCEL (Math 154), we had 316 students in the study. In Spring 1998 we had final exam data for 540 students. However, since we wanted to analyze the effect of the previous semester's final, and we wanted to exclude EXCEL, we had only 388 students in our study. This included 348 first-year students, for which we had both the Math 151 final score and all placement information. There were 91 other first-year students with incomplete data which we ignored.

    151/152		#     152   Std. Std.   F151
    Formats		Exams Mean  Err. Dev.   Mean  NBPRE
    ------		----- ----- ---- ----   ----  -----
    All sections	388   132.4 1.7	 37.5	130.3  28.2
    LL (control)	251   130.3 2.5	 36.8	129.0  28.3
    PL		 45   111.4 6.2	 41.9	126.7  26.2
    LP		 50   144.5 5.2	 36.8	128.4  29.0
    PP		 42   152.9 5.8	 37.6	144.6  28.5
    Table 6. Final Exams in Spring 1998 All students, by format in both semesters

    Here is a scatter plot of 151 final exam scores vesus 152 final exam scores.

    151 vs. 152 Final Exams

    Figure 2: Final Exam Scores: Calc I versus Calc II

    We considered the LL group (students in the standard 4-credit version both semesters) as our control group. The correlation between the Fall and Spring final exams scores was reasonable (.54), and there was a concomitant correlation with the MSAT and NBPRE scores (the correlations were between .4 and .44).

    Clearly the worst scores are in the PL group, the group of 45 students who went from the Practicum to the Standard sections. In searching for features of this group that might explain this, we observed that their final exam scores were almost independent of the MSAT and NBPRE scores (the correlations were .01 and .17).

    The LP group, consisting of 50 students who switched from the Standard sections to the Practicum sections, was equally interesting. Their scores were significantly higher than those in the control group. Here the curious feature was that the Math 151 final score was almost independent of the predictors MSAT and NBPRE (the correlations here were .17 and -.07, respectively).

    The best performers were in the PP group (in the Practicum both semesters). For this group there was a very strong correlation (.77) between the Math 151 and 152 final exam scores.

    Multiple Regression for Math 152 FINAL
    versus Math 151 Final (F151), NBPRE, LORP and PRELORP for first-year students.