fix(compilers/openapi): refuse a $ref resolving through itself - #229
Merged
Conversation
The pre-parse cycle scan let two shapes reach speakeasy's resolver that hang
the process rather than producing a diagnostic. Both are reachable from under
a hundred bytes, and both are validation-clean: $ref is a legal Path Item
field in 3.1 and a pointer that fails to resolve is a resolution problem, not
a schema one, so a caller that validates first is not protected.
A pointer that passes through the reference being resolved deadlocks.
Reference.resolve holds that reference's own write lock across the pointer
walk, and the walk read-locks every reference it traverses, so re-entering one
waits on a lock its own goroutine holds. sync.RWMutex is not reentrant, and no
concurrency is involved. The scan resolved the pointer and asked only where it
landed, so it read
paths:
/a: {$ref: '#/paths/~1a/t'}
as a dangling reference and let it through -- but the resolver never evaluates
the last segment; it deadlocks walking the '#/paths/~1a' prefix.
followRefChain now reports a three-valued verdict instead of (cyclic,
leftComponents), which also retires the "only meaningful alongside
cyclic=true" footgun. The split between them is what the resolver can see, not
where the pointer points: its guard extends its reference chain only once a
hop completes, so a closed cycle is caught there -- and the all-components
spelling of one carries the better message, so those stay its to report. A hop
that re-enters a reference never completes, so nothing upstream can report it,
and the components spelling deadlocks exactly like the document-position one.
Re-entrancy is therefore refused whatever the pointer names.
Re-entrancy is a property of a pointer and the chain reading it, not of a node
alone, so the safe-memo is narrowed: a chain is recorded terminating only when
no hop passed through a reference. Without that the refusal depends on
declaration order -- declaring the dangling reference first records it as safe
and the later chain short-circuits past the re-entrant hop. Every hop in a
real document passes through mappings like components and schemas and no $ref
node at all, so the memo stays in force where it earns its keep.
A pointer the scan read differently from the resolver stack-overflows. The
resolver trims and percent-decodes the pointer half of a $ref
(references/reference.go GetURI and GetJSONPointer), so '#/paths/~1a ' names
/a there while a scan reading the raw value called it dangling. That
self-reference left the resolution cache pointing at its own reference, and
GetObject's delegation recursed until the stack was exhausted. Reading the
pointer the way the resolver reads it is not a nicety here: any divergence is
a reference the scan cannot see, and one trailing space was enough to be one.
nodeview.InternalPointer now mirrors that normalization and PointerPath walks
the normalized form, keeping every node it passes through -- a target alone
cannot express a danger that is not at the destination. ResolvePointer had no
callers left and is gone.
Six reproducers are added and seeded into the fuzz corpus, with four negative
controls pinning the shapes that must stay accepted -- each measured against
the resolver first, since refusing one would be refusing a document that
compiles. cycle_path_item_prefix_chain.yaml declares its paths in the order
that hides the bug, which is the order the memo narrowing exists for.
Both defects were found by FuzzCycleDetector, where they presented only as
workers dying with "EOF": go test -fuzz wires worker stdio to /dev/null, so
the worker's 10s watchdog panic is discarded and the input recorded as the
crasher is whichever one the worker happened to hold, not the one at fault.
Reported upstream as speakeasy-api/openapi#231, with a fix in
speakeasy-api/openapi#230; the scan stays regardless, since a replace
directive would not reach anyone consuming this module as a library.
Review follow-ups on the re-entrant-reference refusal.
legal-prefix-names-offchain-ref claimed to cover a pointer whose prefix names
a reference off the chain reading it, but its /b carried a `get` and so was
not a reference at all. The fixture exercised a weaker shape than its name,
and a mutation that refused on any reference prefix -- the natural way to get
this rule too wide -- passed it. /b now references a component path item, which
is what the name always said, and the mutation reddens it. Renamed to
legal-prefix-through-offchain-ref, since what matters is the traversal, not
the naming.
legal-schema-prefix-into-path-item read backwards: the chain it covers starts
at the schema and re-enters it through a path item, not the other way round.
Renamed to legal-schema-chain-reentry, and the block comment now says why each
control is safe -- two pass through a reference that is not on the chain, two
re-enter from a schema position, which resolves through jsonschema rather than
the openapi Reference wrapper and never takes the lock.
The reproducer list described cycle_pointer_whitespace_self as a deadlock
under the shared re-entrancy comment. It is a stack overflow, and its cause is
the pointer normalization rather than a re-entrant hop, so it carries its own
note.
The package comment, Cycles, refCycles and diag.CyclicRef all described the
refusals as they were before this change. Each now covers the re-entrant one,
and the counts ("two refusals", "three classes") are gone rather than
incremented -- a maintained count is one more thing to get wrong, and neither
comment needed the number.
The refusal in compilers/openapi/internal/scan models one version of speakeasy's behavior. Until now the evidence that the model was right was four hand-written negative controls and a set of reproducers -- an argument, not a measurement, and the same kind of argument that produced the components carve-out this branch had to correct. So ask the resolver. For each of thirty-five generated shapes (five reference positions x seven pointer forms) the oracle compares what the compiler does with what the resolver actually does, and holds the compiler to both directions a model can be wrong in: a shape the resolver cannot survive must be refused, and a shape it resolves cleanly must not be. A shape it survives but reports an error on binds neither way, and getting that distinction wrong is not hypothetical -- the oracle's first run "failed" on four shapes because it read "the resolver returned" as "the document is fine". An exact self-reference at a document position returns, reporting a circular reference. Refusing it early is a choice about which message is better, not an over-refusal, so the helper reports what the resolver said rather than only that it returned. Both halves run in a subprocess. A deadlock and a stack overflow cannot be observed from inside the process they happen to, and the compiler is subject to both if its refusal regresses -- measured in-process, the oracle would hang on exactly the regression it exists to catch rather than failing. Planting that regression now reddens six shapes in ten seconds, each naming the spec. It lives entirely in a _test.go file, so neither the coverage gate nor the architecture test is affected by a harness that talks to the dependency directly. Alongside it, check-coverage.sh sets an explicit -timeout. A regression in this refusal is a test that never returns rather than one that fails, and go test's ten-minute default is a long time to wait to be told which goroutine is stuck.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The pre-parse cycle scan let two shapes reach speakeasy's resolver that hang the process instead of producing a diagnostic. Both are reachable from under a hundred bytes.
Both are also validation-clean — zero errors from the parser's validation and from
Validate().$refis a legal Path Item Object field in 3.1, and a pointer that fails to resolve is a resolution failure rather than a schema violation, so a caller that validates before compiling is not protected.A pointer that passes through the reference being resolved deadlocks
Reference.resolveholds that reference's own write lock across the pointer walk, and the walk read-locks every reference it traverses, so re-entering one waits on a lock its own goroutine holds.sync.RWMutexis not reentrant; no concurrency is involved.The scan resolved the pointer and asked only where it landed. There is no
tkey, so it read this as a dangling reference and let it through — but the resolver never evaluates the last segment. It deadlocks walking the#/paths/~1aprefix.The same shape reaches the in-flight reference through
GetObject's cache delegation, so it is not only self-reference:A pointer the scan read differently from the resolver stack-overflows
The resolver trims and percent-decodes the pointer half of a
$ref, so'#/paths/~1a '— one trailing space — names/athere, while a scan reading the raw value called it dangling. That self-reference left the resolution cache pointing at its own reference, andGetObject's delegation recursed until the stack was exhausted.Any divergence between how this package reads a pointer and how the resolver reads one is a reference the scan cannot see. One space was enough to be one.
Changes
followRefChainreports a three-valued verdict (terminates/cycles/reenters) instead of(cyclic, leftComponents), which also retires the "only meaningful alongsidecyclic=true" footgun the old signature documented against itself.The components carve-out is re-cut along what the resolver can see, not where the pointer points. Its guard extends its reference chain only once a hop completes, so a closed cycle is caught there — and the all-components spelling carries the better message, naming the chain, so those stay its to report. A hop that re-enters a reference never completes, so nothing upstream can report it, and the components spelling deadlocks exactly like the document-position one. Re-entrancy is refused whatever the pointer names.
The safe-memo is narrowed. Re-entrancy is a property of a pointer and the chain reading it, not of a node alone, so a chain is recorded terminating only when no hop passed through a reference. Without that the refusal depends on declaration order. Every hop in a real document passes through mappings like
componentsandschemasand no$refnode at all, so the memo stays fully in force where it earns its keep.nodeview.InternalPointermirrors the resolver's normalization, andPointerPathwalks the normalized form while keeping every node it passes through — a target alone cannot express a danger that is not at the destination.ResolvePointerhad no callers left and is removed.Test plan
Six reproducers under
testdata/openapi/, wired intocycleReproducersso they are also fuzz seeds, and intoknownInvalidfor the corpus sweep.Four negative controls pin the shapes that must stay accepted — a prefix naming an off-chain reference, both hops dangling, and the two schema-position shapes. Each was measured against the real resolver before being written down: refusing any of them would be refusing a document that compiles today.
cycle_path_item_prefix_chain.yamldeclares its paths in the order that hides the bug, which is the order the memo narrowing exists for.Planted-defect checks rather than assertions that merely look right, run against the branch as it will merge — in both directions, since a rule like this fails by being too wide as readily as too narrow:
path-item-prefix-chain, the fixture whose declaration order exists for itThe over-refusal pair is what caught a bad control:
legal-prefix-names-offchain-refclaimed to cover a prefix naming an off-chain reference, but its/bcarried agetand so was not a reference at all. It passed the too-wide mutation. Fixed to match its name, renamedlegal-prefix-through-offchain-ref, and it now reddens.TestCompile_ReentrantPrefixRefusedInBothDeclarationOrderscovers both orders at the compile boundary.Gate:
gofmtclean,go vetclean,golangci-lint0 issues,go build ./..., coverage gate passed at 4221/4221 statements.FuzzCycleDetectorran 10 minutes / 22.8M executions clean. Before the fix it failed at 29s, 12.6s, 87s and 116s across four runs.Notes
Reported upstream as speakeasy-api/openapi#231, with a fix in speakeasy-api/openapi#230. The scan stays regardless of whether that lands: a
replacedirective is ignored when this module is consumed as a dependency, so it would protectcmd/morphicand CI while doing nothing for anyone importing morphic as a library. This refusal is what covers them.The refusal is now measured, not argued
internal/harness/resolver_oracle_test.gois a differential oracle over 35 generated shapes (5 reference positions x 7 pointer forms). For each it compares what the compiler does against what the resolver actually does, holding the compiler to both directions a model can be wrong in: a shape the resolver cannot survive must be refused, and a shape it resolves cleanly must not be.Two things it taught us that hand-written controls could not:
It lives entirely in a
_test.gofile, so neither the coverage gate nor the architecture test is affected by a harness that talks to the dependency directly.check-coverage.shnow sets an explicit-timeout, since a regression in this refusal is a test that never returns rather than one that fails.