Recently I was looking at the NSRE math scores of some schools around the state, and it occurred to me to share with you the results from a few schools I've seen who's NSRE trends exemplify what we all hope to achieve.  At the same time I thought I would share some miscellaneous thoughts about factors I consider when interpreting scores.

Jim Abrams

--------------------------------------------------------------------


1. The DOE School Improvement Website  When you are in the process of looking at your NSRE math results and considering action planning, I can't say enough good things about the DOE's School Improvement Support 'website'.  If you haven't been there you owe it a visit. http://data.ed.state.vt.us/apg/index.html .  Not only can you look at your own scores a number of ways, but you can also look at comparisons with some other schools.

2. Reading Scores: Examples of score trends you would like to see. I am going to say a few words about reading scores that you may already know, or maybe not.  In doing so I want to use a few schools as examples, schools who's scores look very promising at first inspection.  To know for sure whether they clearly represent improved teaching, I would have to talk with the schools more.  But they certainly represent the take we all would like to see. Below are listed several elementary schools, a middle school and a high school. Look at them on the DOE site  in conjunction with the narrative below ( Use the question: Are we making progress toward all students meeting or exceeding state standards?).  These schools are not the only schools with impressive trends.  They simply happen to be some good examples I noticed recently.

Lowell Village (4)
Derby Elementary (4)
Manchester Elementary (4)
Floodbrook (8)
Northfield High (10)

Measurement Error. When you look at scores there are several things you want to consider.  First is 'measurement error'.  On the DOE site its shown as a confidence interval. [a little bar above and below the actual score with a line in between].  Schools with a small N will have a large confidence interval.  The 'true score' for your school could be anywhere in that interval.  With large confidence intervals what you need to look at is the scores over time.  That will tell you in general whether you are looking at reasonably 'true score' in a given year or a one year variation due to random measurement factors.  You can't ever tell for sure in any year, but looking at a trend in scores representing small N's like Lowell's (above) you gain some confidence that their performance is not random score variations.

If you have a school with a very small N (under 20) and patterns don't appear as cleanly as one of the sample schools above, I would do a rolling average.  Take the average of the first year and the second, and compare that to an average of the second year and the third, the third and the fourth etc.  The trend line produced by this technique reduces the impact of measurement error but at the same time it also dampens the relative changes.  Still, if there was too much noise with small N's to determine much of anything, I would use this technique and then look for a positive slope in the overall trend line.

Cohort Effect Subjectively, everyone who has taught has experienced the grade group ("class") of kids that was substantially better or worse academic performers than usual.  This is what is called a cohort effect.  I heard a math specialist at a department of education in a nearby state say he believed that classes with up to a hundred students could exhibit noticeable cohort effect on a state test. I am keeping an open mind about his conjecture. In my own experience I have clearly seen a significant cohort effect in a class of 60 students.

I have two ways I look for cohort effect. First, are scores in a year unusual high or low compared to other years.  Are all three scores for that year comparably high (or low). When you see an unexpected jump you always have to ask two questions.  What do the teachers say about 'the class'.  Was it an especially good or poor performing class. [by the way, this qualitative assessment is best to get before teachers see the scores in question]  The second thing to ask is "was there anything significant that changed in the school in the year in question or the preceding year.

I have seen a high school exhibit a 23 point gain in a single year in a class with 60 students.  The teachers had predicted they would see a bump that year.  And they predicted that the gain would not be sustained the next year.  Both prediction came true.  It was clearly a 'cohort effect'.

In a different case I saw a 20 point gain in students achieving or exceeding the standards in concepts in a single year that represented a different kind of change. It was in a school with an  N above 100.  Looking closely at changes in the school the most likely explanation we found was a single outstanding eighth grade teacher.  I wouldn't have guessed it was possible but that explanation fit the data well, and much better than any alternative hypothesis that we could produce.

Socio-economic effect. Another factor that can effect scores is SES, basically poverty and affluence.  The single largest predictor of academic performance is the education level of a student's mother.  This is in itself correlated with income level.  Good scores from a school don't necessarily represent good teaching compared to another school.  It may say much more about the students.  

If you want to look at a strong indicator of teaching power, look at progress since the new standards exam was initiated.  If you can see movements from 15-30 percent of your students making or exceeding standards in concepts or problem solving up into the realm of 50-70 percent, with a trend that looks trustworthy, then you are most likely looking at some teaching power.  Simply having some of the highest scores in the state that are relatively flat over time may reflect the capability of the students much more than the teaching.   

Changes in the Test: A last factor that can effect scores is changes in the test form used by New Standards.  Its unfortunate that that happens, but with the exception of the grade 10 scores a couple of years ago, I believe the effect is relatively small, less than say 5-7 percentage points.  The assessment folks at the DOE can speak much more knowledgeably about this.

The same test form was used in 1999 and 2000. Sometimes I use those years to check whether a relatively modest change in scores in a school with a large N may be attributable to the test rather than 'real' improvements in performance.

Trends.  In the end, if you look at much of what has been said, it is the trend data that tells the most about a schools scores and obviously improvements in scores.  We are fortunate now to have a sufficient number of years to see progress even in schools with small class sizes.  And we are fortunate to have the DOE School Improvement Support Site.   Look at the schools I have listed above.  They are examples of NSRE trends you would like to see.  These schools suggest what is possible, not only in the clear progress trends they demonstrate but also the absolute levels they have reached to date.

They give us hope.  And they challenge us to further efforts.   


'Jim Abrams

*********************************************
Jim Abrams
Director of Mathematics Education
Vermont Institute for Science, Math and Technology
7 West Street
Montpelier, Vt   05602
Ph: (802) 828-0069
[log in to unmask]