feat: LEP-5 availability commitment support (Merkle proof challenge)#274
feat: LEP-5 availability commitment support (Merkle proof challenge)#274mateeullahmalik wants to merge 1 commit intomasterfrom
Conversation
mateeullahmalik
force-pushed
the
feat/lep5-availability-commitment
branch
from
613a4e6 to
a79249d
Compare
Re-reviewed after efeba07. Two previous issues (HashLeaf domain separation and unconditional BuildCommitmentFromFile) are fixed. Two remain, and one new issue found:
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![roomote-v0[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
pkg/cascadekit/commitment.go
Outdated
| buf[len(root)+2] = byte(counter >> 8) | ||
| buf[len(root)+3] = byte(counter) | ||
|
|
||
| h := merkle.HashLeaf(counter, buf) // reuse BLAKE3 — domain doesn't matter here |
There was a problem hiding this comment.
merkle.HashLeaf(counter, buf) is a leaf-hashing function that includes domain separation (typically a 0x00 prefix byte and the leaf index). The comment says "domain doesn't matter here" but it does: if the chain-side derives challenge indices using raw BLAKE3 (without the leaf domain prefix), the SDK and chain will compute different indices, causing proof verification to fail. The counter is also effectively included twice -- once as the HashLeaf index parameter and once in the big-endian bytes already appended to buf. Consider using a plain BLAKE3 hash of buf instead, or confirm the chain uses the identical HashLeaf call for index derivation.
Fix it with Roo Code or mention @roomote and request a fix.
| for uint32(len(indices)) < m { | ||
| // BLAKE3(root || uint32be(counter)) | ||
| buf := make([]byte, len(root)+4) | ||
| copy(buf, root) | ||
| buf[len(root)] = byte(counter >> 24) | ||
| buf[len(root)+1] = byte(counter >> 16) | ||
| buf[len(root)+2] = byte(counter >> 8) | ||
| buf[len(root)+3] = byte(counter) | ||
|
|
||
| h := merkle.HashLeaf(counter, buf) // reuse BLAKE3 — domain doesn't matter here | ||
| // Use first 8 bytes as uint64 mod numChunks | ||
| val := uint64(h[0])<<56 | uint64(h[1])<<48 | uint64(h[2])<<40 | uint64(h[3])<<32 | | ||
| uint64(h[4])<<24 | uint64(h[5])<<16 | uint64(h[6])<<8 | uint64(h[7]) | ||
| idx := uint32(val % uint64(numChunks)) | ||
|
|
||
| if _, exists := used[idx]; !exists { | ||
| used[idx] = struct{}{} | ||
| indices = append(indices, idx) | ||
| } | ||
| counter++ |
There was a problem hiding this comment.
When m == numChunks (which happens when challengeCount >= numChunks, e.g. 8 challenges on a file that produces 4-8 chunks), this loop must discover every index in [0, numChunks) through random sampling. With small numChunks, the last few indices become increasingly unlikely to hit, and counter has no upper bound. While it will terminate probabilistically, adding a safety cap (e.g. counter > m * 100) with an error return would prevent a potential hang in degenerate cases.
Fix it with Roo Code or mention @roomote and request a fix.
| for uint32(len(indices)) < m { | ||
| // BLAKE3(root || uint32be(counter)) | ||
| buf := make([]byte, len(root)+4) | ||
| copy(buf, root) | ||
| buf[len(root)] = byte(counter >> 24) | ||
| buf[len(root)+1] = byte(counter >> 16) | ||
| buf[len(root)+2] = byte(counter >> 8) | ||
| buf[len(root)+3] = byte(counter) |
There was a problem hiding this comment.
Minor: buf is re-allocated every iteration but its length (len(root)+4) is constant. Moving the allocation before the loop and just overwriting the counter bytes each iteration avoids unnecessary GC pressure.
| for uint32(len(indices)) < m { | |
| // BLAKE3(root || uint32be(counter)) | |
| buf := make([]byte, len(root)+4) | |
| copy(buf, root) | |
| buf[len(root)] = byte(counter >> 24) | |
| buf[len(root)+1] = byte(counter >> 16) | |
| buf[len(root)+2] = byte(counter >> 8) | |
| buf[len(root)+3] = byte(counter) | |
| buf := make([]byte, len(root)+4) | |
| copy(buf, root) | |
| for uint32(len(indices)) < m { | |
| // BLAKE3(root || uint32be(counter)) | |
| buf[len(root)] = byte(counter >> 24) | |
| buf[len(root)+1] = byte(counter >> 16) | |
| buf[len(root)+2] = byte(counter >> 8) | |
| buf[len(root)+3] = byte(counter) |
Fix it with Roo Code or mention @roomote and request a fix.
mateeullahmalik
force-pushed
the
feat/lep5-availability-commitment
branch
2 times, most recently
from
9ff4779 to
efeba07
Compare
SDK Client (BuildCascadeMetadataFromFile): - Build Merkle tree from file chunks using chain SVC params - Generate challenge indices and AvailabilityCommitment - Attach commitment to CascadeMetadata at registration Supernode Server (Register): - After data hash verification, verify Merkle root against on-chain commitment - Generate chunk proofs for challenged indices - Pass proofs through SimulateFinalizeAction and FinalizeAction Lumera Client Layer: - Thread ChunkProofs through FinalizeCascadeAction and SimulateFinalizeCascadeAction - Include chunk_proofs in finalize metadata JSON - Update interface, impl, helpers, mocks, test fakes New: pkg/cascadekit/commitment.go - BuildCommitmentFromFile: chunk file, build tree, derive indices - VerifyCommitmentRoot: rebuild tree and verify against on-chain root - GenerateChunkProofs: produce Merkle proofs for challenge indices - SelectChunkSize: adaptive chunk sizing per LEP-5 spec go.mod: enable local lumera replace for PR-103 compatibility
| return nil, fmt.Errorf("build merkle tree for verification: %w", err) | ||
| } | ||
|
|
||
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { |
There was a problem hiding this comment.
The slice-to-array conversion [merkle.HashSize]byte(commitment.Root) panics if commitment.Root is not exactly 32 bytes. While the chain validates root length at registration, the supernode deserializes this from a protobuf message received over the network -- a corrupted or truncated payload could produce a root of unexpected length. Adding a length guard before the conversion would prevent a runtime panic.
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { | |
| if len(commitment.Root) != merkle.HashSize { | |
| return nil, fmt.Errorf("invalid root length: got %d, expected %d", len(commitment.Root), merkle.HashSize) | |
| } | |
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { |
Fix it with Roo Code or mention @roomote and request a fix.
| return nil, fmt.Errorf("build merkle tree for verification: %w", err) | ||
| } | ||
|
|
||
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { |
There was a problem hiding this comment.
[merkle.HashSize]byte(commitment.Root) will panic at runtime if commitment.Root is not exactly 32 bytes. Since commitment.Root comes from on-chain protobuf deserialization, a malformed or truncated value would crash the supernode. The chain-side equivalent (bytesToMerkleHash in svc.go) validates the length before converting. Consider adding a length check here, or extracting a helper similar to the chain's approach.
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { | |
| var expectedRoot [merkle.HashSize]byte | |
| if len(commitment.Root) != merkle.HashSize { | |
| return nil, fmt.Errorf("invalid commitment root length: got %d, expected %d", len(commitment.Root), merkle.HashSize) | |
| } | |
| copy(expectedRoot[:], commitment.Root) | |
| if tree.Root != expectedRoot { |
Fix it with Roo Code or mention @roomote and request a fix.
There was a problem hiding this comment.
Pull request overview
Adds LEP-5 Storage Verification Challenge (SVC) support by introducing availability commitments (Merkle roots + deterministic challenge indices) during cascade registration and submitting chunk inclusion proofs during finalize/simulate finalize.
Changes:
- Add commitment building (SDK) and commitment verification + chunk proof generation (supernode) gated on
AvailabilityCommitment != nil. - Extend finalize/simulate finalize plumbing across adaptors, lumera action_msg module, mocks, and tests to accept optional
ChunkProofs. - Introduce
pkg/cascadekit/commitment.goutilities for chunking, Merkle tree construction, root verification, and proof generation.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| supernode/cascade/stream_send_error_test.go | Updates stub client signatures to include optional chunk proofs. |
| supernode/cascade/register.go | Verifies on-chain commitment root and generates chunk proofs before simulating/finalizing. |
| supernode/cascade/ica_verify_test.go | Updates fake client finalize/simulate signatures to include chunk proofs. |
| supernode/cascade/events.go | Adds new LEP-5 progress events (root verified, proofs generated). |
| supernode/adaptors/lumera.go | Extends LumeraClient finalize/simulate finalize interface + implementation to pass chunk proofs through. |
| sdk/action/client.go | Builds and attaches an availability commitment when registering (skips very small files). |
| pkg/testutil/lumera.go | Updates test mocks for action/action_msg modules to accept chunk proofs and fixes action types import aliasing. |
| pkg/lumera/modules/action_msg/interface.go | Extends module interface to accept optional chunk proofs for finalize/simulate. |
| pkg/lumera/modules/action_msg/impl.go | Plumbs chunk proofs into finalize/simulate message creation. |
| pkg/lumera/modules/action_msg/helpers.go | Encodes ChunkProofs into cascade metadata for finalize messages. |
| pkg/lumera/modules/action_msg/action_msg_mock.go | Regenerates/updates gomock signatures for finalize/simulate with chunk proofs. |
| pkg/cascadekit/request_builder.go | Updates metadata construction callsite for new commitment-aware signature. |
| pkg/cascadekit/metadata.go | Extends NewCascadeMetadata to optionally set AvailabilityCommitment. |
| pkg/cascadekit/commitment.go | New LEP-5 commitment + proof helpers (chunking, Merkle tree, deterministic indices). |
| go.mod | Pins lumera dependency to pseudo-version including LEP-5/SVC support. |
| go.sum | Updates checksums for the updated lumera dependency version. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // BuildCommitmentFromFile constructs an AvailabilityCommitment for a file. | ||
| // It chunks the file, builds a Merkle tree, and generates challenge indices. | ||
| // challengeCount and minChunks are the SVC parameters from the chain. | ||
| func BuildCommitmentFromFile(filePath string, challengeCount, minChunks uint32) (*actiontypes.AvailabilityCommitment, *merkle.Tree, error) { | ||
| fi, err := os.Stat(filePath) | ||
| if err != nil { | ||
| return nil, nil, fmt.Errorf("stat file: %w", err) | ||
| } | ||
| totalSize := fi.Size() | ||
| if totalSize < MinTotalSize { | ||
| return nil, nil, fmt.Errorf("file too small: %d bytes (minimum %d)", totalSize, MinTotalSize) | ||
| } | ||
|
|
||
| chunkSize := SelectChunkSize(totalSize, minChunks) | ||
| nc := numChunks(totalSize, chunkSize) | ||
| if nc < minChunks { | ||
| return nil, nil, fmt.Errorf("file produces %d chunks, need at least %d", nc, minChunks) | ||
| } | ||
|
|
||
| chunks, err := ChunkFile(filePath, chunkSize) | ||
| if err != nil { | ||
| return nil, nil, err | ||
| } | ||
|
|
||
| tree, err := merkle.BuildTree(chunks) | ||
| if err != nil { | ||
| return nil, nil, fmt.Errorf("build merkle tree: %w", err) | ||
| } | ||
|
|
||
| // Generate challenge indices — simple deterministic selection using tree root as entropy. | ||
| m := challengeCount | ||
| if m > nc { | ||
| m = nc | ||
| } | ||
| indices := deriveSimpleIndices(tree.Root[:], nc, m) | ||
|
|
||
| commitment := &actiontypes.AvailabilityCommitment{ | ||
| CommitmentType: CommitmentType, | ||
| HashAlgo: actiontypes.HashAlgo_HASH_ALGO_BLAKE3, | ||
| ChunkSize: chunkSize, | ||
| TotalSize: uint64(totalSize), | ||
| NumChunks: nc, | ||
| Root: tree.Root[:], | ||
| ChallengeIndices: indices, | ||
| } | ||
|
|
||
| return commitment, tree, nil | ||
| } |
| // Compute data hash (blake3) as base64 using a streaming file hash to avoid loading entire file | ||
| h, err := utils.Blake3HashFile(filePath) | ||
| if err != nil { | ||
| return actiontypes.CascadeMetadata{}, "", "", fmt.Errorf("hash data: %w", err) | ||
| } | ||
| dataHashB64 := base64.StdEncoding.EncodeToString(h) | ||
|
|
||
| // Derive file name from path | ||
| fileName := filepath.Base(filePath) | ||
|
|
||
| // LEP-5: Build availability commitment (Merkle root + challenge indices) | ||
| challengeCount := uint32(paramsResp.Params.SvcChallengeCount) | ||
| if challengeCount == 0 { | ||
| challengeCount = 8 // default | ||
| } | ||
| minChunks := uint32(paramsResp.Params.SvcMinChunksForChallenge) | ||
| if minChunks == 0 { | ||
| minChunks = 4 // default | ||
| } | ||
| // LEP-5: Build availability commitment. Files below MinTotalSize (4 bytes) | ||
| // are too small for meaningful storage verification — skip commitment for them. | ||
| var commitment *actiontypes.AvailabilityCommitment | ||
| if fi.Size() >= cascadekit.MinTotalSize { | ||
| var err2 error | ||
| commitment, _, err2 = cascadekit.BuildCommitmentFromFile(filePath, challengeCount, minChunks) | ||
| if err2 != nil { | ||
| return actiontypes.CascadeMetadata{}, "", "", fmt.Errorf("build availability commitment: %w", err2) | ||
| } | ||
| } |
| chunks := make([][]byte, 0, n) | ||
|
|
||
| buf := make([]byte, chunkSize) |
| return nil, fmt.Errorf("build merkle tree for verification: %w", err) | ||
| } | ||
|
|
||
| if tree.Root != [merkle.HashSize]byte(commitment.Root) { |
| indices := make([]uint32, 0, m) | ||
| used := make(map[uint32]struct{}, m) |
| func VerifyCommitmentRoot(filePath string, commitment *actiontypes.AvailabilityCommitment) (*merkle.Tree, error) { | ||
| if commitment == nil { | ||
| return nil, nil // pre-LEP-5 action, nothing to verify | ||
| } | ||
|
|
||
| chunks, err := ChunkFile(filePath, commitment.ChunkSize) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("chunk file for verification: %w", err) | ||
| } | ||
|
|
||
| if uint32(len(chunks)) != commitment.NumChunks { | ||
| return nil, fmt.Errorf("chunk count mismatch: got %d, expected %d", len(chunks), commitment.NumChunks) | ||
| } |
What
Adds LEP-5 availability commitment support. When a client registers a cascade action, the SDK now builds a Merkle tree over the file's chunks and submits the root + challenge indices as part of the on-chain metadata. When the supernode receives the file for processing, it independently verifies the Merkle root, generates inclusion proofs for the challenged chunks, and includes those proofs in the finalize transaction.
Why
LEP-5 introduces Storage Verification Challenges (SVC). The chain needs to verify that supernodes actually store the data they claim to store. The availability commitment (Merkle root) is submitted at registration time, and chunk proofs are submitted at finalization — the chain then verifies the proofs against the stored root. This is the foundation for on-chain storage accountability.
How it works
SDK side (client registration):
svc_challenge_countandsvc_min_chunks_for_challengefrom chain paramsAvailabilityCommitmentto cascade metadataSupernode side (file processing):
Backward compatibility:
AvailabilityCommitment != nilnilproofs gracefullyChain dependency
Depends on lumera#103 which adds the SVC params, commitment validation, and proof verification to the chain. Currently pinned to a pseudo-version (
v1.11.1-0.20260308102614-4d4f1ce3f65e) — will update to a tagged release once lumera #103 is merged.Testing