SNAP Applicability Scale

A four-axis signature for labeling how much decision weight a study deserves in your context.

Open the interactive scorecard

Purpose

This framework answers a different question than study validity.

Validity asks: did the study measure what it claims?

Applicability asks: how much decision weight should I assign this evidence for real-world spend and inference in my context?

The SNAP Applicability Scale provides a compact, four-axis signature for labeling evidence in decision conversations. It is designed for practical governance, product decisions, training policy, organizational rollouts, and cross-context comparisons, from small business to regulated enterprise.

It does not replace existing evidence quality frameworks. It synthesizes concerns that those frameworks address separately (see the crosswalk below) into a single label optimized for practitioner use.

Notation

The four axes form the acronym SNAP. Each axis uses a unique letter prefix followed by a number, so any component of a signature is instantly identifiable by its prefix.

A note on Provenance direction. Setting, N-Scale, and Attribution all ascend toward “stronger for decision-making.” Provenance does not. P1 (observed, engaged) is not inherently better than P4 (experimental). A well-designed experiment with constructed data is often more useful than sloppy naturalistic observation. Provenance measures a property of the data (distance from unmediated reality), not its quality. Higher P means more constructed, not worse. The appropriate Provenance level depends on the research question, not a universal preference.

A signature reads: S3-N1-A3-P4. Spoken: “S-three, N-one, A-three, P-four.”

Two optional modifiers, Evidence Currency and Hybrid Data, attach to the signature without adding axes.

The composite signature is a label, not a formula. It is read as a profile, not summed into a single score.

Setting (S)

How closely the study environment matches the conditions where the phenomenon actually occurs. Higher numbers indicate stronger real-world alignment.

• S1, Toy Task: puzzle-like. No meaningful stakes. Low transfer.
• S2, Controlled Lab: structured tasks. Short duration. Artificial constraints. Moderate transfer.
• S3, High-Fidelity Simulation: realistic scenario. Representative tools. Partial stakes. Good transfer potential.
• S4, Field Workflow: real tools. Near-real incentives. Operational constraints. Some scaffolding still present.
• S5, In-Situ Natural Environment: real stakes, incentives, and constraints. Observed where the phenomenon naturally occurs.

N-Scale (N)

Counts observed humans. It is not a validity score and not a proxy for truth. It is a practical signal of how broadly an effect has been observed in real people.

If the claim is based on generated, simulated, or otherwise non-human data, do not force it into this axis. Use Provenance (P).

• N0, Anecdote (N < 10): signal only. Hypothesis generation. Not decision-grade.
• N1, Pilot Signal (10 to 99): early directional evidence. Mechanism discovery. Not generalizable without replication.
• N2, Small Field Study (100 to 999): patterns begin to stabilize. Useful for local adoption guidance. Still population-fragile.
• N3, Organizational Evidence (1,000 to 9,999): company-scale repeatability. Defensible for internal policy in similar contexts.
• N4, Industry Evidence (10,000 to 99,999): cross-context regularities. Strong support for standard practice.
• N5, Societal Evidence (100,000+): macro behavior at population scale. Often observational. Hard to randomize. Sufficient ceiling for most organizational and industry-level decisions.

An extended scale (N6 civilizational, N7 global, N8 species-level) is reserved for platform-scale and population analysis. It is defined for forward compatibility but not required for practitioner use.

Attribution (A)

How confidently you can attribute an observed outcome to a specific cause rather than correlation. Separate from human exposure (N) and setting (S). Each level represents a decision-relevant break: the point at which a different class of action becomes defensible.

• A0, Descriptive: anecdotes, case reports, uncontrolled observations. Good for hypothesis discovery. No causal claim warranted.
• A1, Correlational: cross-sectional surveys, longitudinal observation, pre/post without a credible counterfactual. Signals association. Confounding remains.
• A2, Quasi-Experimental: natural experiments, difference-in-differences, regression discontinuity, matched comparisons. A credible counterfactual without randomization.
• A3, Experimental: a single randomized controlled trial, A/B test, or randomized encouragement design. Strong internal validity when well-run. Still one study in one context.
• A4, Replicated: causal finding independently confirmed across teams, contexts, or populations. The effect is not an artifact of one lab, sample, or implementation.
• A5, Converged: multiple independent causal methods and/or high-quality systematic synthesis converge on the same conclusion. Triangulation eliminates method-specific artifacts. Strongest form of causal evidence available.

Provenance (P)

What kind of data are we actually looking at? This axis measures distance from unmediated reality, not evidence quality. As you move up the scale, data becomes more constructed and easier to misinterpret or game without careful documentation.

• P1, Observed, Engaged: voluntary, explicit interaction. Form entries, explicit feedback, deliberate user actions intended as inputs.
• P2, Observed, Byproduct: captured as a side effect of doing real work. Clickstreams, time-in-tool, edit history, workflow logs. High scale. Higher surveillance and interpretation risk.
• P3, Observed, Passive: ambient capture without a deliberate action. Cameras, microphones, sensors, location traces. Highest surveillance risk. Often high noise and strong behavioral distortion.
• P4, Experimental / Trial: human data generated under an intervention protocol. RCTs, A/B tests, randomized encouragement designs. High attribution potential. Lower naturalism than observed data.
• P5, Curated / Benchmark: hand-assembled or standardized datasets designed for measurement convenience. Benchmark suites, test sets, leaderboards. Great for comparability. Often brittle for transfer.
• P6, Synthetic / In Silico: generated via simulation, statistical modeling, or generative methods. Useful for stress-testing, rare-event exploration, and safe experimentation. Not evidence of reality without grounding.

Modifiers

Hybrid Data (H)

If a study mixes data from multiple Provenance types, it does not receive a single P score. Label each component separately and attach the Hybrid flag.

Example: a study combining workflow telemetry (P2) with post-task surveys (P1) and synthetic augmentation for rare cases (P6) is labeled P2/P1/P6 (H). The flag signals that provenance requires scrutiny and that the proportions and transformations between sources should be documented.

Evidence Currency

Evidence ages at different rates depending on domain velocity. A 2019 study on construction safety transfers well to 2026. A 2023 study on AI-assisted coding may already describe tools that no longer exist. Currency is assessed relative to the domain’s rate of change, not absolute time:

• Current: conducted within the current technology or practice generation for its domain. Tools, workflows, and conditions are still operative.
• Recent: one generation behind. Core mechanisms may hold, but specific tools, interfaces, or market conditions have shifted. Apply with caution and check for superseding evidence.
• Historical: two or more generations behind. Useful for foundational mechanisms. Specific findings should not be assumed to transfer.

In fast-moving domains like AI tooling, a generation may be 12 to 18 months. In stable domains like occupational safety, a generation may be a decade. The practitioner determines the generation length for their domain and documents it.

Reading the Composite Signature

Format: S-N-A-P (modifiers). Examples:

• S3-N1-A3-P4 Current
• S4-N3-A3-P4
• S5-N5-A1-P2
• S4-N4-A4-P2/P1 (H)

This is a profile, not a formula. Different decision types weight the axes differently.

For mechanism alerts (should we be cautious about this?): weight Attribution and Setting most heavily. A small study with strong causal design in a realistic setting (S4-N1-A3) can justify preventive action even without scale. N-Scale matters less because the goal is early detection, not proof of generalizability.

For organizational policy (should we change how we operate?): weight N-Scale and Setting most heavily. You need evidence that the effect occurs at your scale and in conditions resembling yours. Strong attribution with no contextual match (S1-N1-A3) is a lab curiosity, not a policy input.

For industry standards (should the field adopt this?): weight N-Scale and Attribution together, and require replication (A4+). A single large observational study (S5-N5-A1) has behavioral gravity but weak causal grounding. A replicated experimental finding at industry scale (S4-N4-A4) is stronger for standard-setting.

For technology adoption (should we buy or build this?): weight Setting and Evidence Currency most heavily. Outdated evidence in artificial conditions is the most common failure mode in vendor-cited research.

Decision Thresholds

Mechanism alerts: N1 or N2 evidence is sufficient to trigger caution if the effect size is large, the mechanism is coherent, and the downside risk is high. Alerts can be justified at A2 or A3 even if Setting is weak, because the goal is prevention, not proof. Currency should be Current or Recent.

Organizational policy: require at least N3 (or replicated N2), and S4 or higher, and A3 or higher. If Attribution is A1 or A2, treat the claim as directional and require additional verification before committing to policy. If Currency is Historical, require corroboration from more recent evidence.

Industry standards: require N4 or higher, evidence from multiple settings, replication across context classes (A4+), and Current or Recent evidence. A1 or A2 evidence at any scale supports trend identification, not standard-setting.

Framework Crosswalk

The SNAP Applicability Scale synthesizes concerns that existing frameworks address in isolation.

• GRADE primarily maps to Attribution (A). Its evidence levels track causal confidence, and its directness criterion partially overlaps with Setting (S). GRADE does not address data provenance (P) or human exposure scale (N) as separate concerns.
• PRECIS-2 maps almost entirely to Setting (S). Its nine domains decompose what SNAP captures as a single S1 to S5 score. Practitioners wanting finer resolution on Setting can use PRECIS-2 scoring within that axis.
• RE-AIM addresses implementation concerns downstream of evidence assessment. Reach partially overlaps with N-Scale, and Effectiveness touches Attribution. RE-AIM is best used after SNAP scoring, to plan rollout of interventions that have passed decision thresholds.
• TRL measures technology maturity, not evidence maturity. The two are orthogonal: a technology at TRL 9 (operational) can still have weak applicability evidence (S1-N1-A1-P5). They complement each other when evaluating AI product claims.

Worked Example

Shen and Tamkin (2026), the Anthropic study on how AI assistance impacts skill formation:

• Setting: S3 (job-adjacent onboarding simulation)
• N-Scale: N1 (N=52)
• Attribution: A3 (single randomized experiment)
• Provenance: P4 (experimental/trial)
• Evidence Currency: Current

Signature: S3-N1-A3-P4 Current

Interpretation: this is not a universal law. It is a strong mechanism alert. The causal design is solid (A3), the setting has reasonable fidelity (S3), and the evidence is current. The limitation is scale (N1): the effect has been observed in 52 people. That is sufficient to trigger organizational caution (delegation-first AI use may erode novice skill formation unless cognitive engagement is enforced) but not sufficient for policy or standard-setting without replication (A4+) and larger samples (N3+).

Practical Use

When evaluating evidence for an AI intervention or any claim about human behavior, label it: Setting, N-Scale, Attribution, Provenance, Currency, mechanism clarity, downside risk. Then determine whether the evidence is:

• Exploratory: hypothesis-generating, not decision-grade
• Adoption-guiding: directional for teams willing to monitor and adjust
• Policy-grade: defensible for internal organizational commitment
• Industry-defining: sufficient for cross-organization standard-setting

The interactive scorecard walks through all of this and computes the grade for you.

Core Rule

Small-N evidence cannot define the world. But it can reveal failure modes worth preventing early.

Skill debt is real. AI can hide it.