Limitations

SchoolDecoder publishes the Decoded Rank, a weighted composite of five school-level signals. School Alpha (context-adjusted performance) is the largest component at 35%; the other 65% blends cohort learning rate, equity-within, multi-year trajectory, and participation. This page describes what the composite and its components can claim, what they cannot claim, and where the model is most cautious. The intent is for parents, journalists, educators, and researchers to know up front where the rank is reliable and where it is not.

The Decoded Rank is one lens, not a verdict. Each component carries its own limitations; the composite inherits all of them. The Decoded Rank composite methodology documents the per-component construction; this page documents the caveats.

What School Alpha CAN claim

School Alpha is the residual between a school's actual standardized test performance and its expected performance given public socioeconomic and enrollment context. From that, the score can fairly support several kinds of statements.

• A school performed above expected for its context in a given year, or below.
• The school's raw scores may understate its performance for its context (a positive Alpha at a lower-resource school).
• The school's raw scores may overstate school effect when most of the score is consistent with the advantages students bring with them (a high raw rank with a modest decoded rank).
• The school's pattern of performance is consistent with stronger-than-expected or weaker-than-expected performance for the school's public context.
• Two schools with similar context can be compared on the residual, with confidence bands attached.

These are descriptive statements about a residual, not causal statements about teaching, curriculum, or leadership.

What School Alpha CANNOT claim

• No causality. School Alpha cannot prove that a school caused better or worse outcomes for students. It is a residual, not an experiment. SchoolDecoder does not write "this school causes better outcomes," "this school adds value," or "this school is better at teaching."
• No teaching-quality verdict. A high Alpha is consistent with strong school-level effects, but other explanations are possible: omitted advantages, selection of students into the school, measurement noise, or year-specific factors. The score is not a teacher rating, a principal rating, or a curriculum rating.
• No value-add interpretation. SchoolDecoder is cross-sectional. It compares schools to expectation given their context in a given year. Value-added modeling tracks the same students over time; that requires longitudinal student-level data we do not have. Where state-published growth metrics exist, the school page shows them separately and labels them as growth, not Alpha.
• No national raw achievement comparison without NAEP linking. The Achievement Rank is a within-state rank or, where the metro shares an assessment scale, a within-metro rank. There is no national raw achievement percentile in this product. Cross-state Alpha rankings can be aggregated more carefully because residuals are computed within each state's testing context, but they still carry a methodology disclosure.

Oregon and Washington use a composite, not a scale score

The performance number that goes into the Alpha model is a derived index — the Performance Level Composite, or, on parent-facing pages, the achievement composite — and not a true scale score from the test publisher.

This is forced by what the launch states publish. Neither the Oregon Department of Education's Assessment Group Reports nor the Washington State Report Card posts school-level mean scale scores. The public surface for both states is the four-level breakdown — percent at Level 1, Level 2, Level 3, and Level 4, plus tested counts. SchoolDecoder synthesizes a single continuous number per school, grade, subject, and year by weighting the four levels (1×%L1 + 2×%L2 + 3×%L3 + 4×%L4, normalized by the total). The data sources page walks through the formula and its limitations.

The honest limitation: the composite is lossier than a true scale score. Two schools that are close on the composite may be further apart on the underlying continuous score, closer together, or reversed; the public level shares cannot tell us which. SchoolDecoder treats composite differences as ordinal once they enter the Alpha model, and the Alpha step standardizes the composite within each state, grade, subject, and year cell before regressing.

If researcher-access scale scores become available, SchoolDecoder will treat them as a distinct input and publish a full-corpus comparison before changing the production methodology. We do not promise that affected schools will move only a few positions. Any switch would require a methodology-version change and a documented before-and-after audit.

California release limitations

California publishes a verified school-level mean scale score, so its 2024 model uses that value directly and standardizes it within each grade and subject before school aggregation. This is a different declared achievement measure from the Oregon/Washington Performance Level Composite; raw points are never pooled across those measures.

Only 2024 has passed the full California ingestion and identity gate. Every California school therefore carries low confidence from the one-year-history rule. Cohort progression and multi-year trajectory are unavailable rather than inferred.

The California model also omits two context predictors that Oregon and Washington include, for two different reasons. No school-level English-learner share is wired for California at all, so that predictor is omitted rather than filled with an estimate. A federal special-education figure does exist for California, but it covers about 93% of schools and is several years old; including it would remove roughly 127 California schools from the published rankings — the ones the federal collection happens to miss — and would shift every remaining school's result. We judged ranking those schools under a slightly simpler model to be the better trade than deleting them, so California's School Alpha is adjusted for income, neighborhood education, free/reduced-price lunch status and locale only. This choice is recorded with a dated expiry and will be revisited when the state publishes an annual, near-complete special-education count. California Alphas are therefore comparable within California but not strictly spec-comparable to Oregon and Washington.

Selective enrollment is an important residual risk. Public directory flags do not identify every exam, gifted, magnet, application, or other selective program consistently. SchoolDecoder excludes California alternative and dedicated special-education schools from the main pool, but a school listed as regular can still enroll a selected population. SES adjustment does not remove that selection effect, and a high Decoded Rank must not be read as a causal estimate of teaching quality.

Race, omitted variables, and the bias audit

Race is not a predictor in the SchoolDecoder model. The model uses income, neighborhood education, free/reduced-price lunch share (with adjustments for the Community Eligibility Provision), and — where the state publishes usable school-level figures — English-language-learner share and special-education share. (See the California section above: neither is used there.) This is an editorial choice as well as a modeling one: a school's Decoded Rank should not move because of the racial composition of its students.

The honest limitation is that the SES index does not capture every structural factor that shapes student outcomes. Wealth, family education, housing stability, access to outside-of-school resources, and historical patterns of opportunity are correlated with — but not identical to — the variables in the index. Because of that, the residual can still reflect omitted variables, and a school's Decoded Rank is not a clean estimate of school effect alone.

SchoolDecoder publishes a bias audit subpage that reports the distribution of School Alpha by racial composition quartile of served students. The intent is to surface, not hide, any residual relationship between Alpha and student demographic composition. If the audit shows a systematic relationship, it is reported in plain language; the model is not laundered through silence. The audit is updated alongside each metro release.

The broader point: omitting race from the predictors does not by itself remove omitted-variable risk. It means the rank does not move because of race directly. The SES index is the best public proxy we have for school context; it is not a complete one.

Free/reduced-price lunch and the Community Eligibility Provision

Free/reduced-price lunch (FRL) share is one of the most commonly cited proxies for school poverty. It is also one of the noisiest, particularly for schools that participate in the Community Eligibility Provision (CEP).

CEP allows eligible schools to serve free meals to all students without collecting individual household applications. CEP schools may report FRL participation at or near 100% regardless of the underlying household economic status of enrolled students. Treating that 100% as ordinary FRL would overstate poverty for some schools and would distort the SES index.

Per the SchoolDecoder model rule, FRL is excised from the SES index for CEP schools entirely. There is no downweighting; FRL is removed for that school, and substitutes are used in priority order:

1. Direct-certification rate where the state publishes it
2. State "economically disadvantaged" field where the state publishes it
3. ACS tract data only

Each school record stores a CEP flag and the substitution path used. CEP-affected schools are not flagged on the public scorecard surface today, but the methodology is consistent and traceable.

School geocode is a proxy, not ground truth

A school's census tract is assigned by spatially joining the school's geocoded location to a TIGER/Line tract polygon. The tract describes where the school building sits. It does not describe the residential distribution of the school's enrolled students.

For neighborhood schools serving a defined catchment, the tract is a reasonable proxy for the families served. For magnets, charters, choice schools, and boundary-exception schools, the tract proxy is weaker — students may be enrolled from across the district or across the metro, and the school's surrounding tract may look richer or poorer than the families actually served.

This caveat matters most for choice schools, magnets, and charters. SchoolDecoder carries a visible enrollment-model badge on school pages and metro rows. The badge is a disclosure, not a model adjustment: neighborhood schools are typically assigned by address, while opt-in, charter, magnet, alternative, and virtual schools may reflect family self-selection as well as school effects.

Neighborhood SES is estimated from school location and public enrollment demographics. It may not fully represent the residential distribution of enrolled students, especially for choice, magnet, charter, or boundary-exception schools.

Closing this gap requires school attendance boundary data and, ideally, student-level residential information. Boundary data is on the SchoolDecoder roadmap; student-level residential information is not — and would carry significant privacy concerns even if it were available.

Small-school caution

Small schools are noisier than large schools by construction. A class of 18 tested students will produce a more variable achievement composite than a grade of 180. Ranking a small school next to a large school without adjustment would let a small school land near the top or bottom of a metro list on chance variation.

SchoolDecoder applies shrinkage: the published School Alpha for a small school is pulled toward zero based on tested count, grade/subject coverage, and year-to-year volatility when prior Alpha history is available. Confidence bands also widen. The visible effect on the school page is a Medium or Low confidence label and a wider rank interval in the rank tooltip.

The implication for parents: a small school's Decoded Rank is real, but it should be read as a tier with uncertainty, not as a precise position. Two small schools at adjacent ranks may not be meaningfully different. A small school below the state's suppression threshold for tested count is not publicly ranked at all.

Cross-state assessment incompatibility

For the launch metro, Portland-Vancouver, both Oregon and Washington use Smarter Balanced achievement levels for grades 3–8 ELA and math. Their public school files do not include publisher scale scores, so SchoolDecoder standardizes the achievement measure within each state before forming the shared metro pool. The cross-state Achievement Rank is a pooled standardized placement, not a raw-score comparison.

For metros that cross state lines into incompatible assessments — Kansas City, Memphis, the Washington DC region, and others — raw Achievement Rank is shown within each state separately. The states' assessments are on different scales; mixing them into one raw ranking would be a methodology violation.

School Alpha is more robust to this because it is computed within each state, grade, subject, and year before aggregation. Even so, residual distributions can vary across testing systems for reasons unrelated to school effect (different test noise, different difficulty calibration, different scoring conventions). Cross-state Alpha rankings carry a methodology disclosure on the metro page, and where the assessments differ substantially, SchoolDecoder may publish split state-side metro views instead of a unified metro ranking.

Year-over-year stability and assessment transitions

Alpha scores should be reasonably stable across adjacent years for the same school. Wild year-over-year swings in the absence of a known cause usually reflect noise, not signal, and SchoolDecoder runs an annual stability check to catch them.

State assessment systems do change. Colorado moved from PARCC to CMAS. State cut scores are occasionally revised. New tests cannot be compared cleanly to old tests. When SchoolDecoder detects an assessment transition, the year of the transition is marked as a break and SchoolDecoder does not compute a trend across the boundary. The school page footnotes the transition; the metro and state pages disclose it.

The COVID-era testing gap (spring 2020 missing in most states; reduced participation in 2021) is treated as a break. Adjacent-year comparisons across that boundary are footnoted.

Suppression and small cohorts

State suppression rules limit the school-level results that can be released publicly. Tested counts below the state's threshold (typically n less than 10) are suppressed by the state and are not used in SchoolDecoder's model. Schools whose only available data falls below the suppression threshold are not publicly ranked.

Schools may also be excluded from the public ranking if test participation is materially low, if the school is a special program, hospital school, juvenile justice school, or other non-comparable setting, or if grade or subject coverage is too thin to support a stable Alpha. Charter and magnet schools are flagged but included; alternative schools are reported separately rather than mixed into the main rankings.

When a school appears as a stub page rather than a ranked page, the page says so honestly and does not display fake numbers.

One lens, not a verdict

The Decoded Rank is one summary number derived from public data. It is useful for separating raw scores from the advantages students bring, and it is useful for surfacing schools that perform above or below expectation given their context. It is not the whole picture of a school.

A school is a building with a culture, a staff, a community, a budget, a history, and a thousand details no public dataset captures. The Decoded Rank is one lens that the public data supports cleanly. It deserves to sit next to a parent's visit, a conversation with current families, a meeting with school staff, and the parts of a school that no model will ever measure.

For how the score is computed, see the methodology page. For the data behind the inputs, see the data sources page.