feat(crypto)!: track leanVM main (internalized XMSS + reworked aggregation) - #539
feat(crypto)!: track leanVM main (internalized XMSS + reworked aggregation)#539MegaRedHand wants to merge 18 commits into
Conversation
…ation)
Point the leanVM dependency at `main`. leanVM's `main` ("Xmss api rework")
removed the `lean-multisig` and `leansig_wrapper` crates ethlambda imported,
internalized XMSS in its own `xmss` crate (dropping the external leanSig
dependency), and reworked the recursive-aggregation API. Tracking main is
therefore a migration of the whole signature stack, not a version bump.
Changes:
- Cargo: replace `leansig`/`lean-multisig`/`leansig_wrapper` with `xmss` +
`rec_aggregation` (git, branch=main), plus `ssz` (ethereum_ssz, used by the
xmss SSZ impls) and `postcard` (secret-key format).
- types::signature: rebuild ValidatorPublicKey/Signature/SecretKey on leanVM's
`xmss` crate. Keys/signatures (de)serialize via SSZ (`PUB_KEY_SSZ_LEN` /
`SIGNATURE_SSZ_LEN`); secret keys via postcard. `SIGNATURE_SIZE` and the new
`PUBLIC_KEY_SIZE` are sourced from the xmss scheme constants. The sliding
preparation-window API becomes a fixed activation range (`advance_preparation`
is now a no-op; keys warm their signing cache on demand).
- crypto: rewrite aggregation/verification against `rec_aggregation`. Proofs now
serialize with `to_bytes()`/`from_bytes()` and embed participant pubkeys, so
the verify paths cross-check the embedded set against the expected validators
instead of attaching them at decode time. `setup_prover`/`setup_verifier` are
replaced by the idempotent `init_aggregation_bytecode`.
- Validator public keys are now 32 bytes (was 52); genesis/state layouts and
fixtures updated accordingly.
BREAKING: this changes the on-wire signature/proof formats and the genesis key
format. It is interop-incompatible with clients still on the previous scheme and
requires regenerated genesis keys; the fixture-driven spec tests need
regenerated fixtures. Landing this needs ecosystem coordination.
🤖 Kimi Code ReviewSecurity & Correctness
Performance & Memory
Code Quality & Maintainability
Testing
Summary The migration from the external Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
Greptile SummaryMigrates the validator signature stack to leanVM’s internal XMSS and reworked recursive aggregation APIs.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure eligible for this follow-up review remains. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| crates/common/crypto/src/lib.rs | Migrates aggregation and verification to the new leanVM proof APIs, adds explicit initialization, and serializes proofs without participant public keys. |
| crates/common/crypto/src/signature.rs | Rebuilds validator key and signature wrappers around leanVM XMSS with SSZ wire encoding and postcard secret-key persistence. |
| crates/common/types/src/state.rs | Changes validator public keys from 52 to 32 bytes and updates the consensus-state wire layout accordingly. |
| crates/common/types/src/attestation.rs | Updates the fixed XMSS signature wire size and blank-signature representation for the internal XMSS scheme. |
| bin/ethlambda/src/main.rs | Initializes leanVM before cryptographic paths and wires the configurable prover allocator into startup. |
| crates/common/types/src/genesis.rs | Updates genesis public-key parsing, test vectors, and pinned state and block roots for the new key format. |
| crates/storage/src/state_diff.rs | Updates storage reconstruction fixtures to use the new validator public-key width. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Startup[Node startup] --> Init[Initialize leanVM bytecode and allocator]
Keys[Postcard XMSS secret keys] --> Sign[XMSS signing]
State[Validator registry: 32-byte public keys] --> Resolve[Resolve participant public keys]
Sign --> Type1[Create Type-1 aggregate]
Resolve --> Type1
Type1 --> Wire1[Public-key-free proof bytes]
Wire1 --> Type2[Merge Type-1 proofs into Type-2]
Type2 --> Block[Signed block]
Block --> Decode[Attach resolved signer sets]
State --> Decode
Decode --> Verify[Verify bindings and recursive proof]
Reviews (2): Last reviewed commit: "Merge branch 'main' into build/leanvm-tr..." | Re-trigger Greptile
🤖 Codex Code Review
Aside from that, the signature/aggregation migration looks internally consistent: the proof-verification paths now bind embedded pubkeys back to the expected validator sets, and the reaggregation path still sits behind full block-proof verification. I couldn’t run Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
…e branch Tracking `branch = main` re-resolves on every fetch, so the build could change under us. Pin the current main HEAD for reproducibility; the comment notes it is main's tip and the rev can be bumped to move forward.
Moves the pin two commits forward from a73ab11 (`Xmss api rework`, #262): - `105c5c69` rec_aggregation: optional pubkey-less aggregate serialization - `c83b40f0` fully single-threaded verifier (`Vec` instead of `ArenaVec`), plus a `forbid-parallelism` feature that asserts it `105c5c69` split `SingleMessageInfo` into a `SingleMessageCore` (message, slot, bytecode claim) plus the pubkey set, so the binding checks in the Type-1 and Type-2 verify paths reach through `.core`. The `xmss` crate is byte-identical between the two revs, so key and pubkey formats are unchanged: the genesis keys generated for a73ab11 verify as-is. `c83b40f0` makes proof verification sequential rather than pool-dispatched. The `forbid_parallelism()` guards it adds are inert unless leanVM's `forbid-parallelism` feature is enabled, which we do not enable. Validated on a 3-node all-ethlambda devnet with fresh new-format genesis: real leanVM proving on every block, Type-2 aggregates verified from gossip in ~50ms, finality advancing one slot per slot, no errors.
leanVM `105c5c69` restored optional pubkey-less aggregate serialization, so the wire format no longer has to change relative to `main`. Proofs serialize via `to_bytes_without_pubkeys()` and the caller attaches the resolved signer set at decode with `from_bytes_without_pubkeys()`, exactly as `compress_without_pubkeys()`/`decompress_without_pubkeys()` did before the migration. Every public signature in the crypto crate now matches `main` again, including `split_type_2_by_message`, which regains its `pubkeys_per_component` argument (`reaggregate.rs` already resolves that layout for its merge step, so the call site just passes it through). Attaching the caller's set is what binds a proof to its participants: leanVM sorts and de-duplicates it, then the SNARK checks it against the hash the proof commits to. So the explicit `PublicKeySetMismatch` comparison the embedded-pubkey form needed is gone, and a wrong set now fails inside the verifier — covered by a new `test_verify_wrong_pubkey_set_fails`. Validated on a 3-node all-ethlambda devnet: 114 block Type-2 aggregates decoded and verified through the pubkey-less path (111 carrying attestation components), p50 43.7ms, finality tracking head at 3 slots, no errors. Reaggregation never triggered there, so `split_type_2_by_message` is covered by the Type-2 merge/verify/split round-trip test instead.
`ensure_prover_ready` only called `init_aggregation_bytecode`, so the arena was never engaged and every prover buffer used the system allocator. `setup_prover` lives in the `lean-multisig` facade, not in `rec_aggregation`, so importing crate by crate hid it. Engaging the arena arms a panic: only one proof may run at a time. Take a permit around each proving entry point; without it 5 of the 7 crypto tests panic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves the overlap with #541, which moved the signature primitives out of `ethlambda-types` and into `ethlambda-crypto` while this branch was rewriting those same primitives for leanVM's internalized XMSS. The two wire-size constants cannot follow `signature.rs` into `ethlambda-crypto`: `types::attestation::XmssSignature` and `types::state::ValidatorPubkeyBytes` are defined in terms of them, and `ethlambda-crypto` already depends on `ethlambda-types`. #541 hit the same constraint and hardcoded `SIGNATURE_SIZE` in `types`. Keep that placement, but source both from leanVM's xmss crate so they track the scheme parameters: main's literal 2536 is the old leanSig size and is wrong for leanVM's wire format. `ethlambda-types` therefore keeps a narrow `xmss` dependency for the two constants only, and `ssz`/`postcard` move to `ethlambda-crypto` along with the signing code that needed them.
`ethlambda-types` pulled in leanVM's `xmss` crate solely to source two integers, dragging the whole signing backend into the dependency graph of a crate that only describes wire formats. Hardcode `SIGNATURE_SIZE` and `PUBLIC_KEY_SIZE` there instead, and pin them to the scheme in `ethlambda-crypto` -- the one crate that sees both sides -- with const assertions. A leanVM bump that changes the XMSS parameters now fails to compile with a named error rather than silently resizing every signature and pubkey on the wire.
The const assertions named the mismatched constants but not their values: a
const panic message must be a string literal, so it cannot interpolate the two
numbers a reader needs in order to fix the constant.
Express the check as an array-length mismatch instead. rustc then evaluates both
sides and prints them ("expected an array with a size of 2536, found one with a
size of 1208"), so the corrected value is right there in the error.
A named constant would read better than `const _` but trips `dead_code`, which
is denied in CI, so the intent lives in the comment -- including a warning not
to take rustc's suggestion to edit the array length, which would silence the
guard rather than fix the constant.
leanVM's `setup_prover` engages a bump arena whose slabs never go back to the OS: `free` is a no-op and `begin_phase` only rewinds the per-thread bump pointers, so a node's RSS ratchets up to its allocation high-water mark and stays there whether or not it keeps proving. The new rev adds `setup_prover_without_arena`, which initializes the same prover on the system allocator. Proving is slower; memory is bounded and proofs are identical, which is the trade a long-lived node wants.
Every prover entry point paired `ensure_prover_ready()` with `prover_permit()`, and neither is useful without the other: setup is a no-op after the first call, and proving without the permit is exactly the concurrent-proof case leanVM panics on. Fold them into `acquire_prover()`, which returns the guard, so a caller cannot reach the prover half-armed. Setup now runs under the permit, which serializes the one-time bytecode compile instead of racing every waiting caller into the same `OnceLock`. The idempotency test drops each guard before taking the next: the permit is not reentrant, so holding two at once in one thread would deadlock.
The permit was taken at the top of each entry point, so a caller sat on the process-wide prover lock while it converted pubkeys, zipped signatures and postcard-decoded child proofs: work that touches no prover state and that grows with the number of aggregated signatures. Take it immediately before the aggregate/merge/split call instead. Decoding does need one thing setup provided: a Type-2 decode rebuilds the bytecode claim, and without the aggregation bytecode it returns `None`, which surfaces as a bogus `DeserializationFailed`. The decoding paths now call `ensure_verifier_ready()`, its documented prerequisite, which is a `OnceLock` init rather than a lock.
The previous commit had the decoding entry points call `ensure_verifier_ready()` and then `acquire_prover()`, which is the two-call pattern the merge was meant to remove: it splits one setup across two names and invites a caller to assume decoding is verifier-only work. Since those paths need setup before they can decode, they take the permit from the start and let it cover the decode. `aggregate_signatures` keeps the narrow scope: it only reshapes its inputs and never decodes, so nothing there needs setup.
The system allocator is the right default (bounded RSS), but it costs proving throughput, and a host with memory to spare should be able to buy that back. A `OnceLock<bool>` carries the choice: `enable_prover_arena()` sets it, and the first `acquire_prover()` latches it via `get_or_init(|| false)`, so the allocator cannot change under a prover that already ran. The setter reports whether it won the latch instead of failing silently, since being called too late is the likely mistake and it is invisible otherwise. Covered by an integration test rather than a lib test: the latch is process-wide, so enabling the arena in the lib binary would change the allocator under every other test and make the result ordering-dependent.
Gives operators the memory/throughput trade at boot: off keeps RSS bounded on the system allocator, on buys proving speed with memory that is never returned to the OS. Applied immediately after parsing, ahead of every other init, since the allocator choice latches on the first proof. The too-late branch warns rather than passing silently; it should be unreachable from here, but a future init that proves earlier would otherwise flip the node to the wrong allocator with no trace.
Setup was reached through `acquire_prover`, which coupled two unrelated things: a caller wanting the prove lock also triggered backend init, so the decoding paths had to hold the lock across work that needs no lock purely to get the bytecode compiled first. `init_leanvm(use_arena)` now does both halves in one place and binaries call it right after parsing arguments. `acquire_prover` is just the permit again, so every entry point takes it immediately before proving, and `ensure_verifier_ready` is gone: nothing needs a lazy hook anymore. The arena `OnceLock` goes too, since the choice is a parameter rather than latched global state. Ahead of the Hive test-driver branch in main, which verifies signatures. Tests that touch a proof call `init_leanvm` themselves, standing in for the startup call. This moves ~2.3s of bytecode compilation from the first proof or verification into startup. Nodes paid it either way; it is now off the first duty's critical path, at the cost of a slower boot for nodes that never prove.
🤖 Kimi Code ReviewThis PR migrates the XMSS signature scheme from the standalone Critical: Initialization Order & Concurrency
Consensus & SSZ Correctness
API Compatibility & Code Quality
Testing
Minor
Summary: The PR correctly handles the cryptographic migration with compile-time size checks and proper concurrency controls. Address Item 1 (debug assertions for initialization) and Item 7 (structured error preservation) before merging. The consensus-layer changes (key/signature sizes) are breaking but internally consistent. Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code Review
Open question: I did not find a consensus-state/fork-choice correctness bug in the touched STF or attestation-processing paths themselves. I could not run Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview SummaryThis is a large, well-documented dependency migration (leanSig+lean-multisig devnet4 → leanVM's internalized Strengths:
Findings (see structured list above):
No issues found in the actual fork-choice, attestation, or state-transition logic — this PR is scoped tightly to the signature/aggregation backend swap and pubkey size change, and the cryptographic call sites ( Automated review by Claude (Anthropic) · sonnet · custom prompt |
2 participants
What
Moves the leanVM dependency off the
devnet4-eralean-multisig+leansig_wrappercrates and onto currentmain, pinned ata5909d1.leanVM's
mainremoved the crates ethlambda imported, internalized XMSS in its ownxmsscrate (dropping the external leanSig dependency), and reworked the recursive-aggregation API. So this is a migration of the whole signature stack, not a version bump:cargo updatefails outright (no matching package named leansig_wrapper).It also takes the prover off leanVM's bump arena by default, which is what stops a node's RSS from ratcheting upward for its whole lifetime.
Changes
rev e2592df)rev a5909d1)lean-multisig,leansig_wrapper,leansig(devnet4)xmss+lean-multisig(leanVM's facade),ssz(ethereum_ssz),postcardxmsscratecompress_without_pubkeys()to_bytes_without_pubkeys()— unchanged shape: pubkeys stay off the wirePUB_KEY_SSZ_LEN)SIGNATURE_SSZ_LEN)--prover-arenainit_leanvm(use_arena)at startupadvance_preparationis now a no-op)types::signature: rebuiltValidatorPublicKey/ValidatorSignature/ValidatorSecretKeyonxmss. Pubkeys/sigs (de)serialize via SSZ; secret keys via postcard.SIGNATURE_SIZE+PUBLIC_KEY_SIZEare static-asserted against the scheme constants, so a leanVM bump that changes them breaks the build instead of silently producing wrong-length keys.crypto: rewired aggregation/verification against leanVM's facade. The wire format and every public signature are kept as onmain: proofs serialize without their participant pubkeys, and the caller attaches the resolved signer set at decode (from_bytes_without_pubkeys). A set other than the one aggregated attaches fine but fails verification, since the proof binds a hash of the set — covered by a newtest_verify_wrong_pubkey_set_fails.VerificationError::ProofError(#[from] ProofError)becomesVerificationFailed(String): leanVM'sProofErrorlives in itsbackendcrate, which the facade imports but does not re-export, so it is not nameable here.Memory: off the arena by default
leanVM's
setup_proverengages a bump arena that never returns pages to the OS —freeis a no-op, and a phase reset only rewinds the per-thread bump pointers (zk-alloc: noMADV_DONTNEED, nomunmap, and on Linuxenable_arenaalso disables heap trimming and large-allocation mmap process-wide). RSS therefore climbs to the process's allocation high-water mark and stays there, which is the unbounded growth seen on long-lived devnet nodes.a5909d1addssetup_prover_without_arena, which runs the same prover on the system allocator: slower, bounded, identical proofs. That is now the default, with--prover-arenato opt back in on hosts with memory to spare where proving latency is the constraint.Prover setup and the proving permit
Setup used to be lazy and reached through the permit helper, which coupled two unrelated things: taking the prove lock also triggered backend init, so the decoding paths had to hold the lock across work that needs no lock, purely to get the aggregation bytecode compiled first.
init_leanvm(use_arena)does both halves in one place; the binary calls it straight after parsing arguments, ahead of the Hive test-driver branch (which verifies signatures).acquire_prover()is just the permit again, so every entry point takes it immediately before the prove call rather than across argument conversion and proof decoding.ensure_verifier_readyis gone — nothing needs a lazy hook anymore.This moves ~2.3s of bytecode compilation (measured, 3 runs, both allocators) from the first proof or verification into startup. Nodes paid it either way; it is now off the first duty's critical path, at the cost of a slower boot for nodes that never prove.
The wire format of the proof contents and the genesis key format change, even though the pubkey-less framing matches
main:forkchoice_spectestsValidatorPubkey length != 32stf_spectestsssz_spectestsexpected 1208, got 2536signature_spectestsEverything else in the workspace passes, including all 11
ethlambda-cryptotests (the slow ignored ones plus a new integration test covering the arena path).Landing this needs ecosystem coordination; it is not a drop-in for the live devnet.