There are significant changes coming to New York State’s school accountability system now that the US Department of Education has approved the state’s Every Student Succeeds Act (ESSA) plan.
Below are some highlights of the changes coming to school districts. A few caveats: (a) I completed my analysis without any technical support from NYSED, (b) my analysis is not comprehensive of all changes and (c) there will likely be additional changes as the approved application is transformed into a set of more granular implementation rules.
Bureaucratic obfuscation
But first, a few general observations.
It always astonishes me how — whether consciously or unconsciously — bureaucracies prefer opaqueness to transparency. The 217 page, single-spaced, dense application has no table of contents. As one superintendent commented matter-of-factly: “That’s because they don’t want you to read it.”
As a first principle, accountability systems should be transparent and understandable by all stakeholders. By this measure, New York’s accountability system fails miserably. If folks are discouraged to read about the accountability system and when they do it’s so confusing they can’t understand it, they are prevented from evaluating it critically and challenging its weaknesses. Much of New York’s accountability system remains cloaked within obscure black box calculations that just about no one, including the state’s superintendents, understands. Sadly this will likely always be a huge problem.
The Big Picture
Grade 3-8 ELA & Math assessment results:
- The Performance Index (PI) will now reward advanced performance (i.e., students achieving level 4)
- The PI calculation will penalize you for each opt out that puts you below 95% participation
- Goal setting will be based on a gap closing methodology and will be updated annually based on each school’s most recent baseline performance
- Middle school students who take Math regents will have their performance included in the middle school’s PI calculation
Gr. 9 – 12 results:
- Graduation goal setting will be based on the same gap closing methodology as above
- Science and Social Studies PIs will now be used as measures of school quality and student success.
New metrics:
- There will be a chronic absence reduction goal and a College, Career and Civic readiness goal, which will assign a weight (between 0 and 2.0) to each member of a graduating cohort. The weight will be based on the level of rigor of a student’s high school program and performance.
- Coming next are an out of school suspension reduction goal, as well as a “high school readiness measure for middle school students.” What that metric will include is not stated.
The revised performance index calculation
As in the past, students taking New York’s assessments are rated on the following four point rubric:
- Level 1 = Basic
- Level 2 = Basic Proficient
- Level 3 = Proficient
- Level 4 = Advanced
However, whereas in the past the PI was a number between 0 – 200, the PI will now be a number between 0 – 250. The revised PI calculation, which now credits advanced performance, is below:
PI = (% scoring Level 2) + (% scoring Level 3 * 2) + (% scoring Level 4 * 2.5)
Note: the denominator for any calculation of percentage will be the greater of the number of continuously enrolled tested students or 95% of continuously enrolled students. That is, the denominator will never be less than 95% of continuously enrolled students. This change creates a structural penalty for schools and districts with students opting out of the state assessments. Further
“The NYSED will require districts and schools with a consistent pattern of testing fewer than 95% of students in their general population and/or 95% of their students in one or more specific subgroups to create a plan that will address low testing rates resulting directly or indirectly from actions taken by the school or district, which we are calling institutional exclusion, while recognizing the rights of parents and students.” (p. 76)
This is a pretty astonishing statement. The NYSED appears to be indicating that it will be able to differentiate between an opt out who has been “institutionally excluded” “directly or indirectly from actions taken by the school or district” and an opt out whose parents have exercised their right to refuse their child’s participation in a state assessment. Not surprisingly, the application does not articulate the criteria for that differentiation or the process for assigning the charge of “institutional exclusion.”
Gr. 3 – 8 ELA & Math goal setting methodology
Set an end goal: The state has set the following Gr. 3 – 8 end goal for all students and all subgroups:
Achieve a PI of 200, “which would mean that all students, on average, were proficient.” Crucially, this end goal does not have an associated target date.
Calculate the long-term (5-yr) improvement goal: Close the gap between the baseline PI and 200 by 20% .
5-yr goal formula: ((200 – baseline PI) * .20) + baseline PI
Example assuming a baseline PI of 120
200 – 120 = 80
80 * .2 = 16
120 + 16 = 136
5-yr goal: 136
The application does not explicitly say what the annual goal will be but I believe it’s safe to assume it is 1/5 of the 5-yr goal. In the example above, therefore, the annual goal would be to improve the PI from 120 to 123. Finally,
- The state will calculate a new 5-yr goal for every school and district every year.
- Newly arrived English language learners who are exempt from taking the language arts assessment are not included in the computation of the PI.
- Gr. 7-8 students who take Regents Examinations in Mathematics will have their scores included in the Elementary/Middle PI using the following conversion:
85 or above = Level 4
75 – 84 = Level 3
65 – 74 = Level 2
Below 65 = Level 1
4 yr graduation rate goal setting methodology
Set an end goal: The state has set the following graduation rate goal:
Achieve a 95% adjusted cohort graduation rate (4 yrs by 6/30). Crucially, this end goal does not have an associated target date.
Calculate the long-term (5-yr) improvement goal: Close gap between baseline graduation rate and 95% by 20%.
5-yr goal formula: ((95% – baseline graduation rate) * .20) + baseline graduation rate
Example assuming a baseline graduation rate of 80%
95% – 80% = 15%
15 * .2 = 3
80% + 3% = 83%
5-yr grad rate goal: 83%
The application does not explicitly say what the annual goal will be but I believe it’s safe to assume it is 1/5 of the 5-yr goal. In the example above, the annual goal would be to improve the graduation rate from 80% to 80.6%.
I am shocked that once again there is no dropout reduction goal. Dropout is the greatest failure of a K-12 system. The most recent published graduation data indicates that statewide nearly 1 in 10 of the 2011 cohort dropped out of school. That’s more than 19,000 students. Why the state and feds would not want to see that number decline is beyond me. There is a distinct difference, in my opinion, between strategies to increase graduation rate and strategies to reduce or eliminate dropout.
Chronic absence
Chronic absence is defined as absence that is > 10% of possible attendance days (as calculated based on each student’s enrollment date and possible number of attendance days.)
For the first time that I know of, the NYSED has published detailed statistics on statewide chronic absence (p. 83). The numbers are breathtaking. For example, here are a few statewide 2016-17 chronic absence numbers for grades 9 – 12:
- All students: 24.2%
- Black students: 33.3%
- Hispanic students: 34.0%
- American Indian/Alaskan Natives: 37.4%
- Students with disabilities: 35.2%
Chronic absence goal setting methodology
Set an end goal: The state has set the following end goal for all students and all subgroups:
Achieve a < 5% chronic absence rate. Crucially, this end goal does not have an associated target date.
Calculate the long-term (5-yr) improvement goal: Close gap between baseline chronic absence rate and 5% by 20%.
5-yr goal formula: baseline chronic absence rate – ((baseline chronic absence rate – 5%) * .20)
Example assuming a baseline chronic absence rate of 15%:
15% – 5% = 10%
10 * .2 = 2
15% – 2% = 13%
5-yr grad chronic absence goal: 13%
In the case of chronic absence, the application is clear. The annual goal is 1/5 of the 5-yr goal. In the example above, the annual goal would be to reduce the chronic absence rate from 15% to 14.6%.
Note the application first says that chronic absence includes all excused and unexcused absences. It later clarifies that suspended students (who must receive alternate instruction) and students absent an extended period for medical reasons (and would receive instruction at home) will not be considered absences.
College, Career, and Civic Readiness Index
The College, Career, and Civic Readiness Index is a number that will range from 0 to 200 and will be computed by multiplying the number of students in an accountability cohort demonstrating college and career readiness by the weighting for the method by which the student demonstrated college and career readiness, divided by the number of students in the accountability cohort. (p.64)
In simpler terms: Each member of a graduating cohort will receive a weight (between 0 and 2.0) based on the level of rigor of their high school program and performance as described below. A minimum score would be 0 (all students achieved a weight of 0) and 200 (all students achieved a weight of 2.0).
In my opinion this index is a huge step forward from the state’s prior measure of college and career readiness, especially the John King legacy measure of > 75 on an ELA Regents and > 80 on a Math Regents exams. Finally, The state also plans to create an indicator of civic engagement at some time in the future.
There will be the same gap closing methodology utilized here as with the other metrics. However, while the state has set an end goal of 175, I suspect the implementation of this metric will be delayed for the simple reason that as far as I know many school districts do not record data such as AP exam scores or participation in duel enrollment course in their student management systems — or if they do record such data it may not be flagged in a way that allows the query by the graduating cohort necessary for the above calculation.
How do you fail under the new accountability system?
According to the application (p.70), the state will differentiate all public schools, including charter schools, into the following categories:
- Comprehensive Support and Improvement Schools
- Targeted Support and Improvement Schools
- Schools in Good Standing
- Recognition Schools
Further, “To determine the category into which a subgroup will be differentiated, New York State assigns a Performance Level from 1-4 for each measure for which a subgroup in a school is held accountable.” And how does the state calculate a school’s performance level?
Now we’re entering the black box of NYSED’s “decision tables.” After reading the following two inherently contradictory — and therefore, incoherent — statements, I gave up:
“New York State does not explicitly weight indicators, but rather uses a series of decision rules to differentiate between schools. These decision rules give the greatest weight to academic achievement and growth (in elementary and middle schools) and academic achievement and graduation rate (in high schools).”
So in New York State, we do not weight indicators but we do weight indicators. Really?!?
Over the last few years, if a school district I was working with was placed on what was then called “focus” status or removed from that status, I always make a point of asking the superintendent: “Do you know explicitly why you were labeled a ‘focus’ school or district?” In every case, the superintendent’s answer was the same: “I have no idea.”
If you listen really closely, you can hear me crying “Uncle!”