Limitations

SchoolDecoder publishes the Decoded Rank, a weighted composite of five school-level signals. School Alpha (context-adjusted performance) is the largest component at 35%; the other 65% blends cohort learning rate, equity-within, multi-year trajectory, and participation. This page describes what the composite and its components can claim, what they cannot claim, and where the model is most cautious. The intent is for parents, journalists, educators, and researchers to know up front where the rank is reliable and where it is not.

The Decoded Rank is one lens, not a verdict. Each component carries its own limitations; the composite inherits all of them. The Decoded Rank composite methodology documents the per-component construction; this page documents the caveats.

What School Alpha CAN claim

School Alpha is the residual between a school's actual standardized test performance and its expected performance given public socioeconomic and enrollment context. From that, the score can fairly support several kinds of statements.

• A school performed above expected for its context in a given year, or below.
• The school's raw scores may understate its performance for its context (a positive Alpha at a lower-resource school).
• The school's raw scores may overstate school effect when most of the score is consistent with the advantages students bring with them (a high raw rank with a modest decoded rank).
• The school's pattern of performance is consistent with stronger-than-expected or weaker-than-expected performance for the school's public context.
• Two schools with similar context can be compared on the residual, with confidence bands attached.

These are descriptive statements about a residual, not causal statements about teaching, curriculum, or leadership.

What School Alpha CANNOT claim

• No causality. School Alpha cannot prove that a school caused better or worse outcomes for students. It is a residual, not an experiment. SchoolDecoder does not write "this school causes better outcomes," "this school adds value," or "this school is better at teaching."
• No teaching-quality verdict. A high Alpha is consistent with strong school-level effects, but other explanations are possible: omitted advantages, selection of students into the school, measurement noise, or year-specific factors. The score is not a teacher rating, a principal rating, or a curriculum rating.
• No value-add interpretation. SchoolDecoder is cross-sectional. It compares schools to expectation given their context in a given year. Value-added modeling tracks the same students over time; that requires longitudinal student-level data we do not have. Where state-published growth metrics exist, the school page shows them separately and labels them as growth, not Alpha.
• No national raw achievement comparison without NAEP linking. The Achievement Rank is a within-state rank or, where the metro shares an assessment scale, a within-metro rank. There is no national raw achievement percentile in this product. Cross-state Alpha rankings can be aggregated more carefully because residuals are computed within each state's testing context, but they still carry a methodology disclosure.

The dependent variable is a composite, not a scale score

The performance number that goes into the Alpha model is a derived index — the Performance Level Composite, or, on parent-facing pages, the achievement composite — and not a true scale score from the test publisher.

This is forced by what the launch states publish. Neither the Oregon Department of Education's Assessment Group Reports nor the Washington State Report Card posts school-level mean scale scores. The public surface for both states is the four-level breakdown — percent at Level 1, Level 2, Level 3, and Level 4, plus tested counts. SchoolDecoder synthesizes a single continuous number per school, grade, subject, and year by weighting the four levels (1×%L1 + 2×%L2 + 3×%L3 + 4×%L4, normalized by the total). The data sources page walks through the formula and the rank-correlation evidence in full.

The honest limitation: the composite is lossier than a true scale score. Two schools that are close on the composite may be further apart on the underlying continuous score, or closer together; the composite cannot tell us which. Within each (state, grade, subject, year) cell, the composite preserves school rank ordering against the unpublished scale score with high fidelity — published validation studies typically report rank correlations of 0.95 or higher — but it does not preserve metric distances at the same fidelity. SchoolDecoder treats composite differences as ordinal once they enter the Alpha model, and the Alpha step z-scores the composite within the cell before regressing, which means most of the residual ordering survives the compression.

Practically, the consequence is that SchoolDecoder ranks are close approximations of where ranks would land if the underlying scale scores were public, but they are not identical to them. If researcher-access scale scores become available — through a state DOE data request, a NAEP-linked calibration, or a future change in what either state publishes — SchoolDecoder will switch the modeling input from the composite to the true scale score. Based on the rank-correlation evidence, we expect most schools' Decoded Ranks to move by at most a few positions; the methodology page will be updated to record the switch and any rank shifts will be footnoted on affected school pages.

Race, omitted variables, and the bias audit

Race is not a predictor in the SchoolDecoder model. The model uses income, neighborhood education, free/reduced-price lunch share (with adjustments for the Community Eligibility Provision), English-language-learner share, and special-education share. This is an editorial choice as well as a modeling one: a school's Decoded Rank should not move because of the racial composition of its students.

The honest limitation is that the SES index does not capture every structural factor that shapes student outcomes. Wealth, family education, housing stability, access to outside-of-school resources, and historical patterns of opportunity are correlated with — but not identical to — the variables in the index. Because of that, the residual can still reflect omitted variables, and a school's Decoded Rank is not a clean estimate of school effect alone.

SchoolDecoder publishes a bias audit subpage that reports the distribution of School Alpha by racial composition quartile of served students. The intent is to surface, not hide, any residual relationship between Alpha and student demographic composition. If the audit shows a systematic relationship, it is reported in plain language; the model is not laundered through silence. The audit is updated alongside each metro release.

The broader point: omitting race from the predictors does not by itself remove omitted-variable risk. It means the rank does not move because of race directly. The SES index is the best public proxy we have for school context; it is not a complete one.

Free/reduced-price lunch and the Community Eligibility Provision

Free/reduced-price lunch (FRL) share is one of the most commonly cited proxies for school poverty. It is also one of the noisiest, particularly for schools that participate in the Community Eligibility Provision (CEP).

CEP allows eligible schools to serve free meals to all students without collecting individual household applications. CEP schools may report FRL participation at or near 100% regardless of the underlying household economic status of enrolled students. Treating that 100% as ordinary FRL would overstate poverty for some schools and would distort the SES index.

Per the SchoolDecoder model rule, FRL is excised from the SES index for CEP schools entirely. There is no downweighting; FRL is removed for that school, and substitutes are used in priority order:

1. Direct-certification rate where the state publishes it
2. State "economically disadvantaged" field where the state publishes it
3. ACS tract data only

Each school record stores a CEP flag and the substitution path used. CEP-affected schools are not flagged on the public scorecard surface today, but the methodology is consistent and traceable.

School geocode is a proxy, not ground truth

A school's census tract is assigned by spatially joining the school's geocoded location to a TIGER/Line tract polygon. The tract describes where the school building sits. It does not describe the residential distribution of the school's enrolled students.

For neighborhood schools serving a defined catchment, the tract is a reasonable proxy for the families served. For magnets, charters, choice schools, and boundary-exception schools, the tract proxy is weaker — students may be enrolled from across the district or across the metro, and the school's surrounding tract may look richer or poorer than the families actually served.

This caveat matters most for choice schools, magnets, and charters. SchoolDecoder carries a visible enrollment-model badge on school pages and metro rows. The badge is a disclosure, not a model adjustment: neighborhood schools are typically assigned by address, while opt-in, charter, magnet, alternative, and virtual schools may reflect family self-selection as well as school effects.

Neighborhood SES is estimated from school location and public enrollment demographics. It may not fully represent the residential distribution of enrolled students, especially for choice, magnet, charter, or boundary-exception schools.

Closing this gap requires school attendance boundary data and, ideally, student-level residential information. Boundary data is on the SchoolDecoder roadmap; student-level residential information is not — and would carry significant privacy concerns even if it were available.

Small-school caution

Small schools are noisier than large schools by construction. A class of 18 tested students will produce a more variable achievement composite than a grade of 180. Ranking a small school next to a large school without adjustment would let a small school land near the top or bottom of a metro list on chance variation.

SchoolDecoder applies shrinkage: the published School Alpha for a small school is pulled toward zero based on tested count, grade/subject coverage, and year-to-year volatility when prior Alpha history is available. Confidence bands also widen. The visible effect on the school page is a Medium or Low confidence label and a wider rank interval in the rank tooltip.

The implication for parents: a small school's Decoded Rank is real, but it should be read as a tier with uncertainty, not as a precise position. Two small schools at adjacent ranks may not be meaningfully different. A small school below the state's suppression threshold for tested count is not publicly ranked at all.

Cross-state assessment incompatibility

For the launch metro, Portland-Vancouver, both Oregon and Washington use Smarter Balanced for grades 3–8 ELA and math on a common scale, and the consortium-level cut scores match. Raw Achievement Rank can be shown across the full metro for those grades and subjects without scale violation.

For metros that cross state lines into incompatible assessments — Kansas City, Memphis, the Washington DC region, and others — raw Achievement Rank is shown within each state separately. The states' assessments are on different scales; mixing them into one raw ranking would be a methodology violation.

School Alpha is more robust to this because it is computed within each state, grade, subject, and year before aggregation. Even so, residual distributions can vary across testing systems for reasons unrelated to school effect (different test noise, different difficulty calibration, different scoring conventions). Cross-state Alpha rankings carry a methodology disclosure on the metro page, and where the assessments differ substantially, SchoolDecoder may publish split state-side metro views instead of a unified metro ranking.

Year-over-year stability and assessment transitions

Alpha scores should be reasonably stable across adjacent years for the same school. Wild year-over-year swings in the absence of a known cause usually reflect noise, not signal, and SchoolDecoder runs an annual stability check to catch them.

State assessment systems do change. Colorado moved from PARCC to CMAS. State cut scores are occasionally revised. New tests cannot be compared cleanly to old tests. When SchoolDecoder detects an assessment transition, the year of the transition is marked as a break and SchoolDecoder does not compute a trend across the boundary. The school page footnotes the transition; the metro and state pages disclose it.

The COVID-era testing gap (spring 2020 missing in most states; reduced participation in 2021) is treated as a break. Adjacent-year comparisons across that boundary are footnoted.

Suppression and small cohorts

State suppression rules limit the school-level results that can be released publicly. Tested counts below the state's threshold (typically n less than 10) are suppressed by the state and are not used in SchoolDecoder's model. Schools whose only available data falls below the suppression threshold are not publicly ranked.

Schools may also be excluded from the public ranking if test participation is materially low, if the school is a special program, hospital school, juvenile justice school, or other non-comparable setting, or if grade or subject coverage is too thin to support a stable Alpha. Charter and magnet schools are flagged but included; alternative schools are reported separately rather than mixed into the main rankings.

When a school appears as a stub page rather than a ranked page, the page says so honestly and does not display fake numbers.

One lens, not a verdict

The Decoded Rank is one summary number derived from public data. It is useful for separating raw scores from the advantages students bring, and it is useful for surfacing schools that perform above or below expectation given their context. It is not the whole picture of a school.

A school is a building with a culture, a staff, a community, a budget, a history, and a thousand details no public dataset captures. The Decoded Rank is one lens that the public data supports cleanly. It deserves to sit next to a parent's visit, a conversation with current families, a meeting with school staff, and the parts of a school that no model will ever measure.

For how the score is computed, see the methodology page. For the data behind the inputs, see the data sources page.