Recently I was looking at the NSRE math scores of some schools around the
state, and it occurred to me to share with you the results from a few
schools I've seen who's NSRE trends exemplify what we all hope to achieve.
At the same time I thought I would share some miscellaneous thoughts about
factors I consider when interpreting scores.
Jim Abrams
--------------------------------------------------------------------
1. The DOE School Improvement Website When you are in the process of
looking at your NSRE math results and considering action planning, I can't
say enough good things about the DOE's School Improvement Support 'website'.
If you haven't been there you owe it a visit.
http://data.ed.state.vt.us/apg/index.html . Not only can you look at your
own scores a number of ways, but you can also look at comparisons with some
other schools.
2. Reading Scores: Examples of score trends you would like to see. I am
going to say a few words about reading scores that you may already know, or
maybe not. In doing so I want to use a few schools as examples, schools
who's scores look very promising at first inspection. To know for sure
whether they clearly represent improved teaching, I would have to talk with
the schools more. But they certainly represent the take we all would like
to see. Below are listed several elementary schools, a middle school and a
high school. Look at them on the DOE site in conjunction with the narrative
below ( Use the question: Are we making progress toward all students meeting
or exceeding state standards?). These schools are not the only schools with
impressive trends. They simply happen to be some good examples I noticed
recently.
Lowell Village (4)
Derby Elementary (4)
Manchester Elementary (4)
Floodbrook (8)
Northfield High (10)
Measurement Error. When you look at scores there are several things you want
to consider. First is 'measurement error'. On the DOE site its shown as a
confidence interval. [a little bar above and below the actual score with a
line in between]. Schools with a small N will have a large confidence
interval. The 'true score' for your school could be anywhere in that
interval. With large confidence intervals what you need to look at is the
scores over time. That will tell you in general whether you are looking at
reasonably 'true score' in a given year or a one year variation due to
random measurement factors. You can't ever tell for sure in any year, but
looking at a trend in scores representing small N's like Lowell's (above)
you gain some confidence that their performance is not random score
variations.
If you have a school with a very small N (under 20) and patterns don't
appear as cleanly as one of the sample schools above, I would do a rolling
average. Take the average of the first year and the second, and compare
that to an average of the second year and the third, the third and the
fourth etc. The trend line produced by this technique reduces the impact of
measurement error but at the same time it also dampens the relative changes.
Still, if there was too much noise with small N's to determine much of
anything, I would use this technique and then look for a positive slope in
the overall trend line.
Cohort Effect Subjectively, everyone who has taught has experienced the
grade group ("class") of kids that was substantially better or worse
academic performers than usual. This is what is called a cohort effect. I
heard a math specialist at a department of education in a nearby state say
he believed that classes with up to a hundred students could exhibit
noticeable cohort effect on a state test. I am keeping an open mind about
his conjecture. In my own experience I have clearly seen a significant
cohort effect in a class of 60 students.
I have two ways I look for cohort effect. First, are scores in a year
unusual high or low compared to other years. Are all three scores for that
year comparably high (or low). When you see an unexpected jump you always
have to ask two questions. What do the teachers say about 'the class'. Was
it an especially good or poor performing class. [by the way, this
qualitative assessment is best to get before teachers see the scores in
question] The second thing to ask is "was there anything significant that
changed in the school in the year in question or the preceding year.
I have seen a high school exhibit a 23 point gain in a single year in a
class with 60 students. The teachers had predicted they would see a bump
that year. And they predicted that the gain would not be sustained the next
year. Both prediction came true. It was clearly a 'cohort effect'.
In a different case I saw a 20 point gain in students achieving or exceeding
the standards in concepts in a single year that represented a different kind
of change. It was in a school with an N above 100. Looking closely at
changes in the school the most likely explanation we found was a single
outstanding eighth grade teacher. I wouldn't have guessed it was possible
but that explanation fit the data well, and much better than any alternative
hypothesis that we could produce.
Socio-economic effect. Another factor that can effect scores is SES,
basically poverty and affluence. The single largest predictor of academic
performance is the education level of a student's mother. This is in itself
correlated with income level. Good scores from a school don't necessarily
represent good teaching compared to another school. It may say much more
about the students.
If you want to look at a strong indicator of teaching power, look at
progress since the new standards exam was initiated. If you can see
movements from 15-30 percent of your students making or exceeding standards
in concepts or problem solving up into the realm of 50-70 percent, with a
trend that looks trustworthy, then you are most likely looking at some
teaching power. Simply having some of the highest scores in the state that
are relatively flat over time may reflect the capability of the students
much more than the teaching.
Changes in the Test: A last factor that can effect scores is changes in the
test form used by New Standards. Its unfortunate that that happens, but
with the exception of the grade 10 scores a couple of years ago, I believe
the effect is relatively small, less than say 5-7 percentage points. The
assessment folks at the DOE can speak much more knowledgeably about this.
The same test form was used in 1999 and 2000. Sometimes I use those years to
check whether a relatively modest change in scores in a school with a large
N may be attributable to the test rather than 'real' improvements in
performance.
Trends. In the end, if you look at much of what has been said, it is the
trend data that tells the most about a schools scores and obviously
improvements in scores. We are fortunate now to have a sufficient number of
years to see progress even in schools with small class sizes. And we are
fortunate to have the DOE School Improvement Support Site. Look at the
schools I have listed above. They are examples of NSRE trends you would
like to see. These schools suggest what is possible, not only in the clear
progress trends they demonstrate but also the absolute levels they have
reached to date.
They give us hope. And they challenge us to further efforts.
'Jim Abrams
*********************************************
Jim Abrams
Director of Mathematics Education
Vermont Institute for Science, Math and Technology
7 West Street
Montpelier, Vt 05602
Ph: (802) 828-0069
[log in to unmask]