feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layer

[sieniven]

Summary

This PR revamps the entire flashblocks rpc layer design, similar to the previous Realtime-feature (RTA) cache capabilities on the since discontinued xlayer-erigon (refer to: https://github.com/okx/xlayer-erigon/tree/dev/zk/realtime).

Essentially, this revamp implements the full solution 2 mentioned in the upstream reth's issue discussion thread: paradigmxyz/reth#20491, which is a more elegant design to the complex op-reth's flashblock rpc layer that is tightly coupled to the underlying anchoring canonical chainstate.

Motivations

The best design for allowing minimal latencies is to allow a full state accumulation from incoming flashblocks that are ahead of the canonical chainstate, while allowing for a hybrid sync approach that:

1. State accumulated from incoming flashblocks that are ahead of the canonical chainstate are stored entirely in memory, while maintaining the underlying canonical chainstate as an anchor.
2. Optimize default CL sync from the state accumulation through a pre-warming mechanism of already executed flashblocks sequence (skip payload re-validation entirely)
3. Optimize flashblocks sequence validation with current Reth's payload validator optimizations. This includes
   • State root strategies (sparse trie caching, parallel merklization, and default synchronous SR resolution)
   • Prewarming states mechanism
   • Asynchronous resolution of transaction root, receipt root and log bloom on block assembling
4. Support processing flashblocks sequence block access list for faster state executions
5. Leverage on current reth engine's changeset cache, for asynchronous deferred computing of trie data
6. Automatic flushing of flashblocks state layer and in-memory trie changesets (up till CHANGESET_CACHE_RETENTION_BLOCKS), once canonical chainstate catches up
7. State queries (pending and confirmed states, also supporting latest tags) on flashblocks rpc tries to read from both the flashblocks state cache and underlying canon provider
8. State queries handle edge cases automatically such as chain re-orgs, flashblocks re-orgs, and flashblocks state layer lagging behind canonical chainstate

New flashblocks rpc design

1. Flashblock cache layer

• Adds the new FlashblockStateCache layer which includes both the pending state and confirm state caches
• Confirm state cache indexes and stores full block data (state and stateless) in moemory
• Pending state cache stores the current flashblocks sequence state received from the sequencer
• On cache misses on the flashblocks state cache, all state queries are proxied to the underlying canon provider
• To handle edge case where canon provider is ahead of flashblocks state cache and cache reorg has not yet been triggered, if state retrieval hits on both underlying canon provider and flashblocks cache, we prioritize canon provider as the source of truth
• Holds the flashblocks state layer's trie data changeset cache, used for sparse trie task computation for accelerated flashblocks sequence pending state computation

2. Revamp execution environment for flashblocks sequence validation

The flashblocks RPC layer revamp contains significant optimizations and alignment to the upstream reth's payload validator logic. The old logic was vastly over-complicated, and the revamped version now encapsulates the flashblocks sequence validation inside the new FlashblockSequenceValidator. It aligns the sequence validation logic to the highly optimized upstream reth's engine payload validator.
Ref: https://github.com/paradigmxyz/reth/blob/main/crates/engine/tree/src/tree/payload_validator.rs

Below are some of the key optimizations:

1. Change logic to by default always calculate state root. Merklization on payload validation has been optimized greatly on reth's v1.11 release onwards with the new sparse trie implementation (feat: sparse trie as cache paradigmxyz/reth#21583) and is no longer a bottleneck here
2. Sequence validator uses the upstream reth's payload processor which encapsulates the merklization optimization and payload validation logic. This includes:
   • State root strategies (sparse trie caching, parallel merklization, and default synchronous SR resolution)
   • Prewarming states mechanism
   • Support processing flashblock access list
   • Asynchronous resolution of transaction root, receipt root and log bloom on block assembling
3. Sequence validator supports both fresh builds (on new pending heights) and incremental builds for flashblock sequence deltas (building on top of the current pending state). This includes:
   • Support re-using cache reads across incremental builds
   • Merging suffix execution results with cached prefix execution results
   • Pre-validation of incoming flashblocks sequence
   • Executing transactions on top of in-memory state overlay providers, with pending bundle pre-state updated

3. Flashblocks eth rpc extension

• Adds the new flashblocks-eth rpc extension (and maintaining compatibility with upstream OpEthApi)
• Override all eth json-rpc APIs to support using the new flashblocks state cache layer and the underlying canon chainstate provider
• Adds support for previously missing flashblocks eth APIs:
  • eth_sendRawTransactionSync
  • eth_getRawTransactionByBlockHashAndIndex
  • eth_getRawTransactionByBlockNumberAndIndex

4. Remove sequence manager, revamp to RawCache

• Removes the sequence manager entirely and its overly complicated updating logic from the execution task completion thread (updating calculated SR near target flashblocks count and re-broadcasting the same sequence)
• Simplifying to use a raw cache, which accumulates the raw flashblocks received by the sequencer
• Raw cache maintains similar logic to get the next best build-args, while tracking the last fully executed flashblocks sequence index

5. Add handlers for flashblocks state update logic

• Add handle_flashblocks handler - handle receiving a raw flashblock, and updating the raw cache
• Add handle_flashblocks_state handler - handle spawning task execution for flashblocks sequence validation and updating the flashblocks state cache
• Add handle_canonical_block - logic to handle canonical chainstate updates to flush flashblocks state cache layer, and also handle extreme cases like chain reorgs, etc

Supported flashblocks eth APIs

Block apis
[✅] eth_blockNumber
[✅] eth_getBlockByNumber
[✅] eth_getBlockByHash
[✅] eth_getBlockReceipts
[✅] eth_getBlockTransactionCountByNumber
[✅] eth_getBlockTransactionCountByHash

Transaction apis
[✅] eth_getTransactionByHash
[✅] eth_getRawTransactionByHash
[✅] eth_getTransactionReceipt
[✅] eth_getTransactionByBlockHashAndIndex
[✅] eth_getTransactionByBlockNumberAndIndex
[✅] eth_getRawTransactionByBlockHashAndIndex
[✅] eth_getRawTransactionByBlockNumberAndIndex
[✅] eth_sendRawTransactionSync

State apis
[✅] eth_call
[✅] eth_estimateGas
[✅] eth_getBalance
[✅] eth_getTransactionCount
[✅] eth_getCode
[✅] eth_getStorageAt

TODO:
Support eth_getLogs in the future

Design flaws on op-reth flashblocks rpc

1. The flashblocks rpc layer is tightly coupled to the canonical chainstate height. Thus, although they have since added optimistic builds to advance the pending state, however confirmed chainstates (using "latest" tags) are still dependent on the canonical chainstate
2. Flashblocks consensus sync mode does not make sense since flashblocks finality is less probably than default blocks. Flashblocks consensus sync also cannot currently account for rollbacks
3. Lack of support or design consideration for hybrid sync approach - tightly coupled with reason 2. Using a hybrid sync approach (CL sync + flashblocks state updates) is ideal for flashblocks design. However, full payloads from CL driven sync are currently re-executed on the default payload validator on engine_newPayload (even though the flashblocks sequence had already executed and validated the exact same payload)
4. SR calculations are triggered based on configurable values instead of a more deterministic approach, which can be a point of failure on rpc consensus. Also with reth's sparse trie async task feature implemented on reth v1.11 onwards, incremental SR calculations on payload validation becomes much cheaper
5. No flashblocks BAL implemented

On subsequent reth version upgrades (todo):

1. Add perf(tasks): add LazyHandle<T>, use for hash post state paradigmxyz/reth#22347 fixes once reth version is upgraded
2. Add perf: share executed tx counter with prewarming paradigmxyz/reth#22647 optimizations to share tx index
3. refactor(engine): PayloadProcessor::spawn_state_root paradigmxyz/reth#22604

Todos:

1. Add support for flashblocks block access list for faster flashblocks sequence execution and validation

[JimmyShi22]

@claude please review this PR and comment inline

[sieniven]

Note that the wip tasks left are:

1. flashblocks handling pipeline integrating all components (flashblocks state cache, sequence validator, and the flashblocks eth api handle)
2. unit test coverages on flashblocks state cache and flashblocks payload validator

They will be added in soon asap.

[sieniven]

@claude please review this PR and comment inline

[github-actions]

Claude finished @sieniven's task in 2m 34s —— View job

---

PR Review: feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layer

Tasks:

• Read PR context and understand the scope
• Review new cache module (cache/mod.rs, cache/confirm.rs, cache/pending.rs, cache/raw.rs)
• Review new execution module (execution/validator.rs, execution/mod.rs, execution/assemble.rs)
• Review new RPC layer (rpc/src/flashblocks.rs, rpc/src/default.rs, rpc/src/helper.rs)
• Review service layer changes (flashblocks/src/service.rs)
• Review handle/handler changes (flashblocks/src/handle.rs)
• Review test utilities and subscription changes
• Compile and summarize findings

---

Overall Assessment

This is a significant architectural improvement over the previous implementation. The separation into cache, execution, and RPC layers is clean and well-reasoned. The design aligns well with the motivation stated in the PR. The codebase is well-documented with clear comments explaining invariants. However, there are several bugs and structural concerns to address.

Note: As stated in the review, handle_flashblocks_state and pipeline wiring are WIP and not yet integrated, so some issues below are pre-emptive on incomplete code.

---

Critical Issues

1. Known bug: Stale hash index on duplicate block height in ConfirmCache
📄 crates/flashblocks/src/cache/confirm.rs:570-588

The test test_confirm_cache_insert_duplicate_height_leaks_stale_hash_index documents this explicitly as a "known limitation." When insert() is called twice for the same height (which can happen during flashblock re-orgs at the same block height), the old block hash remains in hash_to_number. This means get_block_by_hash(old_hash) would still return Some(height) and could return the new block for the old hash — silent data corruption.

The fix is to look up and remove the old entry from hash_to_number in insert() before overwriting:

// In insert(), before the HashMap inserts:
if let Some((old_hash, _)) = self.blocks.get(&height) {
    self.hash_to_number.remove(old_hash);
}

Fix this →

2. Potential panic in commit_pending_sequence - receipt index out of bounds
📄 crates/flashblocks/src/execution/validator.rs:418

receipt: executed_block.execution_output.result.receipts[idx].clone(),

This indexes directly without bounds checking. If the execution output has fewer receipts than transactions (e.g., due to a system transaction that doesn't produce receipts, or an error), this will panic. Use .get(idx) and handle the None case.
Fix this →

3. process_flashblock is incomplete — raw cache not integrated
📄 crates/flashblocks/src/service.rs:150

fn process_flashblock(&mut self, flashblock: OpFlashblockPayload) {
    self.notify_received_flashblock(&flashblock);
    // TODO: Insert into the raw flashblocks cache
}

The primary pipeline connecting received flashblocks to the RawFlashblocksCache is a TODO. The FlashblocksRpcService does not hold the raw cache, sequence validator, or state cache — meaning the entire execution pipeline is unconnected. This is the main "WIP" item the author noted. The review notes this as a blocker for merge.

---

Significant Issues

4. DEFAULT_TX_CACHE_SIZE is 10M entries — potential unbounded memory
📄 crates/flashblocks/src/cache/confirm.rs:16

const DEFAULT_TX_CACHE_SIZE: usize = DEFAULT_CONFIRM_BLOCK_CACHE_SIZE * 10_000;
// = 1_000 * 10_000 = 10_000,000 entries

Each CachedTxInfo holds a full signed transaction and receipt. 10M entries pre-allocated via HashMap::with_capacity could reserve significant memory at startup. OP Stack blocks can have up to ~30k transactions, so 1000 blocks × 30k txs = 30M would still exceed this. Consider basing the capacity on a realistic block tx count, or not pre-allocating with with_capacity at all.
Fix this →

5. blocking_recv() in potentially async context
📄 crates/flashblocks/src/execution/validator.rs:265-278

let (receipts_root, logs_bloom) = {
    receipt_root_rx.blocking_recv()...
};
let transactions_root = result_rx.blocking_recv()...

execute_sequence uses blocking_recv(). If this is ever called from an async context without spawn_blocking, it will block the tokio runtime thread. The PR does not show where execute_sequence is called from (the handle_flashblocks_state handler is WIP). This must be called within spawn_blocking or a dedicated thread to be safe.

6. get_state_provider_by_id flushes cache on read path
📄 crates/flashblocks/src/cache/mod.rs:172-176

Err(e) => {
    warn!(..., "Failed to get flashblocks state provider: {e}. Flushing cache");
    self.inner.write().flush();
    None
}

A read operation (state provider lookup) triggers a write-side cache flush on error. This is aggressive and has a lock upgrade pattern (drops read lock, acquires write lock). Between the drop and the write, other threads could observe intermediate state. More importantly, a transient error on one read will flush data that other concurrent readers may have gotten correct results from. Consider returning None without flushing, or using a separate error-tracking mechanism before deciding to flush.

7. Magic + Duration::from_secs(1) in commit_pending_sequence
📄 crates/flashblocks/src/execution/validator.rs:423

pending: PendingBlock::with_executed_block(
    Instant::now() + Duration::from_secs(1),
    ...
),

This 1-second offset looks like a workaround to keep the pending block "alive" for a short time. It's not documented. This should have a comment explaining why +1 second is used and what happens when the Instant expires.

---

Minor Issues

8. eth_blockNumber override may confuse clients
📄 crates/rpc/src/flashblocks.rs:238-245

async fn block_number(&self) -> RpcResult<U256> {
    let fb_height = self.flashblocks_state.get_confirm_height();
    let canon_height: U256 = EthApiServer::block_number(&self.eth_api)?;
    Ok(std::cmp::max(fb_height, canon_height))
}

Returning the flashblocks confirm height as the "block number" could confuse clients that call eth_blockNumber and then immediately call eth_getBlockByNumber on that height — if the block hasn't been confirmed canonically yet, the canonical provider will return None. The cache fallback in block_by_number handles this for flashblock heights, but other APIs that aren't overridden (e.g., eth_getLogs) would not see the block. Document this behavior explicitly.

9. prevalidate_incoming_sequence accepts stale flashblocks below canon height
📄 crates/flashblocks/src/execution/validator.rs:467-473

let canon_height = self.flashblocks_state.get_canon_height();
if incoming_block_number > canon_height + 1 {
    return Err(...);
}
Ok(None)

When incoming_block_number <= canon_height (i.e. the flashblock is stale — for a block already canonical), the validation passes and returns Ok(None). The sequence would then be executed unnecessarily. Consider explicitly rejecting incoming_block_number < canon_height here.

10. get_best_revision result is not connected to BuildArgs construction
📄 crates/flashblocks/src/cache/raw.rs:151-167

get_best_revision() returns the index of the last consecutively received flashblock, but there's no public method on RawFlashblocksCache that uses this to construct BuildArgs for the validator. This seems like a key missing piece — without it, the raw cache has no way to drive the execution pipeline.

11. RawFlashblocksCache has &mut self on methods but wraps Arc<RwLock<>>
📄 crates/flashblocks/src/cache/raw.rs:35-42

The outer RawFlashblocksCache requires &mut self for handle_canonical_height and handle_flashblock, but internally it just locks the Arc<RwLock<...>>. The &mut self constraint is more restrictive than necessary — &self would suffice. This prevents sharing the outer handle across tasks unless wrapped in another Arc<Mutex<>>.

12. assemble_flashblock diverges from upstream OpBlockAssembler for Ecotone blob gas
📄 crates/flashblocks/src/execution/assemble.rs:78-84

} else if chain_spec.is_ecotone_active_at_timestamp(timestamp) {
    (Some(0), Some(0))

The upstream sets blob_gas_used = Some(0) for Ecotone but the actual executed blob_gas_used from the block execution is discarded. For Jovian this uses the real value but Ecotone hardcodes to 0. Verify this matches the upstream OpBlockAssembler behavior exactly.

13. utils.rs - block_from_bar is dead code?
📄 crates/flashblocks/src/cache/utils.rs

block_from_bar clones header and body but doesn't appear to be called from any non-test code in this PR. If it's only used internally for potential future use, confirm it's needed or remove to keep the codebase clean.

---

Positive Observations

• Excellent test coverage on cache modules (confirm.rs alone has 20+ unit tests covering edge cases including the known limitation)
• Good use of BTreeMap for O(log n) range eviction in ConfirmCache
• The DeferredTrieData pattern in spawn_deferred_trie_task elegantly avoids blocking the hot path
• await_state_root_with_timeout races approach is correct and avoids deadlocks
• Documentation on get_executed_blocks_up_to_height safety invariants is thorough
• The merge_suffix_results logic correctly handles the cumulative gas offset for incremental builds
• The tx_index in PendingSequence / ConfirmCache enables O(1) lookups avoiding linear scans