The Three Levels of Expert Knowledge

In the 1990s, cognitive scientists Robert Hoffman and Gary Lintern proposed a framework for understanding expert knowledge that's been validated across domains from surgery to firefighting to chess. They described three levels:

Level 1: Domain Constraints — the functional relationships and physical laws that govern a domain. In software engineering, this is "React re-renders when state changes" or "HTTP is stateless." It's the ground truth that doesn't change with perspective.

Level 2: Concepts — categories, terminology, and mental models. "What server components are." "The difference between REST and GraphQL." "What a DAG is." This is textbook knowledge — necessary but not sufficient.

Level 3: Reasoning Strategies — decision heuristics, perceptual cues, and problem-solving logic. "When you see a hydration mismatch, check whether you're reading window in a server component." "If the query asks for a count, use object detection, not CLIP." This is the knowledge that makes experts fast and accurate. It's also the knowledge that's hardest to capture.

Most AI skills are overwhelmingly L2. They describe what things are. They rarely encode how to decide what to do.

This isn't a new problem. Cognitive scientist John Anderson identified it in the 1980s with ACT-R theory: declarative knowledge ("knowing that") is accessible to self-report and explicit instruction. Procedural knowledge ("knowing how") is compiled through practice into fast, unconscious routines that experts can't easily articulate. The more expert someone becomes, the less access they have to the knowledge that makes them expert.

Anderson called this the automation gap. We call it the procedural knowledge problem.

The Report Card: What We Grade On

Each element traces to a specific research tradition — decades of cognitive science about how experts actually think. Here's where they come from.

Where These Ideas Come From

Gary Klein
RPD Model
1989 Crandall, Klein
CTA Methods
2006 Imre Lakatos
Proofs & Refutations
1976 George Polya
How to Solve It
1945 Atul Gawande
Checklist Manifesto
2009

Recognition-Primed Decision Making (Gary Klein, 1989)

Klein spent decades studying how experts actually make decisions — not in labs, but on firegrounds, in ICUs, on aircraft carriers. His finding: experts under pressure don't compare options. They recognize the situation as a familiar type, which immediately activates a standard response. The cognitive work isn't "which option is best?" — it's "what is happening here?"

This is why decision points are the highest-weighted element in our rubric. A skill that says "here are 5 API design patterns" is declarative. A skill that says "if you need partial updates, use PATCH; if you need idempotency, use PUT" is procedural — it encodes the recognition step that Klein showed is where expertise actually lives.

Cognitive Task Analysis (Crandall, Klein, Hoffman, 2006)

CTA is the field dedicated to extracting the knowledge that experts can't self-report. The key technique is the Critical Decision Method: you walk an expert through a specific past incident and probe for the decision points they navigated. "What did you notice? What were you expecting? What would a novice have missed?"

The method works because it provides cognitive scaffolding — concrete cases that help experts access compiled procedural knowledge. You can't ask a chess grandmaster "how do you evaluate positions?" and get a useful answer. But you can show them a specific board state and ask "what do you see?" and they'll tell you things a novice literally cannot perceive.

This is the theoretical basis for our upgrade pipeline — a meta-DAG that upgrades skills by running cognitive task analysis on the skills themselves.

Imre Lakatos: Proofs and Refutations (1976)

Lakatos, a philosopher of mathematics, showed that knowledge grows through a dialectic of conjecture and refutation — not through accumulating certified truths. His key concept: monster-barring, where someone encountering a counterexample redefines their terms to exclude it rather than learning from it.

"That's not a real polyhedron" is monster-barring. "That edge case doesn't count" is monster-barring. "Let me add another NOT-FOR clause" is monster-barring.

Our curator system explicitly tracks monster-barring: if a skill responds to failure by narrowing its scope rather than improving its reasoning, that's a degenerating research programme in Lakatos's terms, and the system flags it.

George Polya: How to Solve It (1945)

Polya's four-phase framework — Understand, Plan, Execute, Review — seems obvious until you realize how often agents skip phases. The most common failure: Understanding to Execution, skipping Planning entirely. This produces technically correct work that answers the wrong question.

The "Looking Back" phase is where Polya's framework connects to our quality system: after every DAG execution, do the four Polya questions get asked? Did the solution satisfy the contract? Were there unstated assumptions? Is this generalizable? Without Looking Back, each solved problem is an isolated event. With it, each solution compounds into transferable expertise.

Atul Gawande: The Checklist Manifesto (2009)

Gawande showed that even world-class experts — surgeons, pilots — benefit from simple checklists at critical pause points. The checklist doesn't replace expertise; it catches the things experts reliably forget under stress.

His key distinction: DO-CONFIRM checklists (for experts — perform from memory, then verify) vs. READ-DO checklists (for novices — execute each step as read). The power comes from the pause point itself — the moment where execution stops and verification happens.

This is why quality gates are in our rubric: a skill that says "ensure quality" is an aspiration. A skill that says "you're done when all 7 of these conditions are true" is a checklist that catches what agents reliably forget.

The Data: 469 Skills, Graded

Average procedural knowledge score: 57/100. Average procedural content: 20%.

Element-by-element breakdown:

Almost every skill says what it's NOT for (easy to write). Only 57% have quality gates (hard to write — requires knowing what "done" looks like). This is the procedural gap made visible.

What Each Grade Looks Like

Zero out of 469 skills scored above 90. Not one. The best skill in the library — design-justice — scored 84 (a B). So what would an A actually require?

An A-grade skill would need all five elements at high quality and they'd need to reinforce each other. The decision tree would route you to specific failure modes. The failure modes would reference specific worked examples. The worked examples would end by verifying quality gates. The quality gates would include conditions that catch the failure modes. It's a closed loop — each element points at the others.

Concretely, an A-grade api-architect would look something like:

None of our skills do this yet. They have decision trees and failure modes and quality gates, but these exist as parallel sections — not as an integrated reasoning structure. Getting from B to A isn't about adding more content. It's about making the content cross-reference itself so an agent navigating one element is naturally led to the others.

That's the next frontier. For now, B is the ceiling and the gap between B (84) and F (54% of the library) is where the leverage is.

Decision tree — not principles, not guidelines, a literal diagnostic flowchart:

This is pure procedural knowledge. It asks a diagnostic question and prescribes the pattern. Most skills skip the diagnosis and just describe the patterns.

Quality gates — testable, binary conditions for "you're done":

These are Gawande pause points. You're done when all boxes check. Many skills have aspirational principles but no gates for "task complete."

Why B and not A? The working examples could be deeper — more case studies showing trade-offs in real implementations. The failure modes exist but don't include detection rules (the "if you see X, you've violated pattern Y" format that clip-aware-embeddings nails).

This skill has the best failure modes in the library. Each anti-pattern ends with a detection rule:

This is procedural logic: a testable condition (does the query ask for a count?) that triggers an action (switch to DETR). It's the difference between "here are 5 things CLIP can't do" (declarative) and "here's how to detect in real time that you're about to misuse CLIP" (procedural).

Strong failure modes, strong worked examples, but weaker procedural logic. The skill says "always set up dead letter queues" — that's prescriptive (always do this), not conditional (do this when X). Compare to design-justice's "is the user on a shared device? Yes -> use privacy mode."

C-grade skills give you patterns and anti-patterns, but require you to diagnose which applies. B-grade skills include the diagnosis.

Conceptually rich, procedurally thin. The anti-patterns are mental mistakes (event-level thinking, parameter obsession), not execution failures (timeout, schema mismatch). No worked examples show how to apply stock-flow analysis to a real problem.

This is the epistemic limit for domains that are primarily conceptual. Some skills should be declarative-heavy — systems-thinking is genuinely about mental models, not procedures. But the audit correctly identifies that an agent can't execute mental models without procedural scaffolding.

This skill is a job description disguised as expertise:

It says "here's what a research analyst can do" — not "here's how to do research in 7 steps." No decision points, no failure modes, no quality gates. An agent given this skill knows it should do research. It still doesn't know how.

A deprecation notice. "Use dag-orchestrator instead." Score of 7 is generous.

See For Yourself: Before & After

Toggle between Before and After to see what the upgrade pipeline does to each skill. Notice the pattern: the upgraded versions are often shorter — declarative concept dumps are verbose, procedural knowledge is information-dense.