rotation of outbound peers by pompon0 · Pull Request #3037 · sei-protocol/sei-chain

pompon0 · 2026-03-06T16:52:06Z

To make the p2p network uniformly random, nodes need to be able to change their peers periodically (otherwise network would be centralized at bootstrap peers). This PR implements an algorithm based on stable hashing, which makes nodes assign random priority to every node ID, and connect to peers with higher priority.

This pr also separates inbound and outbound connection pools to simplify logic. In particular it is possible that there will be 2 concurrent connections between 2 peers (inbound + outbound). Avoiding duplicate connections is best effort - nodes try not to dial peers that they are connected to, but in case 2 peers dial each other at the same time they let those 2 connections be.

Basic requirements:

peermanager maintains a pex table: addresses of peers of our peers. This bounds the network view size (even though it is an unauthenticated network) that our node needs to maintain and provides enough exposure to select new peers. Peers are periodically reporting their connections, therefore peermanager keeps only fresh addresses
on startup a spike of dials is expected - node will try to connect to peers that it was connected to before restart
during stable operation node will dial peers at a low rate like 1/s or 0.1/s, and the node should be selected from a fresh set of addresses - i.e. we cannot snapshot a list of currently available addresses and try to dial them all (it will take hours)
despite low dial rate, node should attempt to round robin over the ever changing peer candidates set - i.e. it should not get stuck dialing the same bad address over and over
in the stable-hash-based approach, each peer ID obtains a priority for dialing - it should be taken into account
implementation should support replacing low priority peers with higher priority peers (to support convergence to a random graph). The churn of the connections should be low though, so that connection efficiency is not affected. My initial guesstimate would be that we should allow replacing a connection every ~1min.
We need to support a pex table with ~100 * 100 = 10k addresses total (100 connections per peer is a safe estimate with the current implementation). Whether we can affort just rank all the addresses on every dial attempt is a borderline IMO.
the addresses inserted to pex table should be made available for dialing ASAP, without any active polling, if possible.

github-actions · 2026-03-06T16:53:03Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Mar 18, 2026, 5:43 PM

sei-tendermint/internal/p2p/peermanager_pool.go

sei-tendermint/internal/p2p/peermanager_updates.go

+		for id := range s.last {
+			if _,ok := GetAny(conns,id); !ok {
+				delete(s.last, id)
+				update = PeerUpdate{
+					NodeID: id,
+					Status: PeerStatusDown,
+				}
+				return true
+			}
+		}


sei-tendermint/internal/p2p/peermanager.go

wen-coding · 2026-03-11T18:22:41Z

sei-tendermint/internal/p2p/peermanager.go

+		regular: newPoolManager(&poolConfig{
+			MaxIn:      options.maxInbound(),
+			MaxOut:     options.maxOutbound(),
+			FixedAddrs: bootstrapAddrs,


Is it possible someone set all bootstrap addresses in persistent so FixedAddrs is now empty?

Yeah, peers are either persistent or not. I can add logic to move bootstrap addresses to persistent addresses in case of overlap to make it slightly more foolproof, wdyt?

As long as we can use the PEX from persistent peers in regular pool, this is probably fine. Moving bootstrap addresses to persistent pool is probably bad, because if everyone tries to connect to bootstrap nodes and never disconnect, then bootstrap will have all its incoming connections occupied and start to reject new-comers, which is worse.
In fact, we should probably intentionally disconnect old connections on bootstrap nodes. But we can do that in the future.

To be clear, I didn't mean to make all bootstrap nodes persistent - I just meant that IF the same peer has an address in both in bootstrap list AND persistent list THEN both addresses could be used to establish the persistent connection to this peer (while currently the address from the bootstrap list is just discarded).

ah I see, that's fine then

sei-tendermint/internal/p2p/peermanager_pool.go

sei-tendermint/internal/p2p/router.go

sei-tendermint/internal/p2p/peermanager_pool.go

sei-tendermint/internal/p2p/pex/reactor_test.go

sei-tendermint/internal/p2p/peermanager.go

sei-tendermint/internal/p2p/router.go

wen-coding

This looks good to me, I think my main concern is we are trusting PEX from any connected peer blindly, but we don't have to fix that in this PR.

wen-coding · 2026-03-17T23:17:21Z

sei-tendermint/internal/p2p/peermanager_pool.go

+				}
+				// We have found a new best candidate.
+				best = utils.Some(addr.pNodeID())
+				clear(bestAddrs)


So an attacker doesn't know your seed, but it can inject fake addresses and try its luck on the fake addresses being high priority for someone, and then we will periodically try the fake addresses?

yes, also there is nothing much to do here - the attacker can do sybil attack and sybil peers are not distinguishable from other peers.

wen-coding · 2026-03-17T23:33:55Z

sei-tendermint/internal/p2p/router.go

+	}))
+	for endpoint := range endpointSet {
+		c, err := utils.WithOptTimeout1(ctx, r.options.DialTimeout, func(ctx context.Context) (tcp.Conn, error) {
+			return tcp.Dial(ctx, endpoint.AddrPort)


So if an attacker tries to use you to launch DoS attack, it can tell you IP of validator B belongs to node A. Of course you will find out later after dialing succeeds and you find out this is node B not node A, but we don't remove the address right? So the attacker can kind of delays your connection to A, and make you launch a mini dos attack to B?
We might need some feedback if we see a node mismatch, also we might want to continue dialing other IP of A? (You don't have to do that in this PR though)

we dial all the IPs of A that we know of (see the outer loop). Attacker can slightly delay us connecting to A, but it doesn't matter. There is not much we can do about the fact that peer can send us mismatching (node key, ip) pairs.

I think we can punish sender of the PEX if it gives us random garbage. Maybe the sender should only give us outbound connection so they are sure the IP:port is correct. But we don't have to do that until later.

I think we can punish sender of the PEX if it gives us random garbage

what if peer of our peer is the attacker though and presents us with a different identity than when it did to our peer?

Right, I guess there are two layers:

I care about staked validators a lot, and I trust staked validators from staked peers

For other non-staked validators I don't care that much, this is best effort. The important ones I care are in my persist list always.

pompon0 added 28 commits February 25, 2026 19:45

added self-declared addresses to handshake

a7787f6

Merge remote-tracking branch 'origin/main' into gprusak-pex

69ff8e5

self addr

811f86d

pex addrs in handshake

9954688

just 1 addr

5cd87bc

error logging in newPeerManager

d0f363b

caching isPublic

ba4fdba

removed selfID from peerManagerInner

32aee27

asdf

24820ff

tests

4e896ef

flag for enabling pex on handshake

3580e31

compilation

337ecd8

added tests

3b5cb73

removed dead branch handling opaque url

a0d36ab

move PexOnHandshake after authentication

17e9433

poolConn

350afac

Merge branch 'gprusak-less-addrs' into gprusak-stablehash

d853866

thoughts

1a0c532

Merge remote-tracking branch 'origin/main' into gprusak-pex

731df5e

Merge remote-tracking branch 'origin/main' into gprusak-stablehash

0995acf

WIP

3bfaac6

before combining dialing and out

80b414b

need to figure out how to limit churn on outbound connections

939d986

pool revamp almost ready, rotation supported

1b8921b

before moving pex to peerManager

9ce0937

WIP, peermanager doesn't look very well

ed0d6a2

seems complete

7c620ce

some updates to router

cfdd0f7

github-advanced-security bot found potential problems Mar 6, 2026

View reviewed changes

pompon0 added 6 commits March 11, 2026 13:51

tests WIP

d17db5f

tests refactored

800ed3c

more tests

0f26cab

last test

80a1c11

pm tests are back

8209c02

pm tests are back

143df90

pompon0 requested a review from wen-coding March 11, 2026 17:47

pompon0 added the non-app-hash-breaking label Mar 11, 2026

fixed defaults

b1c0b8b

wen-coding reviewed Mar 12, 2026

View reviewed changes

fixed multi-address dialing

c79bd81

pompon0 requested a review from wen-coding March 16, 2026 15:12

pompon0 marked this pull request as ready for review March 16, 2026 15:14

pompon0 added 3 commits March 16, 2026 16:21

test fixes

4792b80

Merge remote-tracking branch 'origin/main' into gprusak-pex

498175d

Merge branch 'gprusak-pex' into gprusak-stablehash

3ca76c9

pompon0 requested a review from sei-will March 16, 2026 15:37

pompon0 added 9 commits March 16, 2026 16:43

spurious log

ef3a829

fixed lint

1df96a7

deflaking statesync test

43986ef

simplified the dial test

7f522a2

applied comments

7df80d9

Merge branch 'main' into gprusak-stablehash

466643c

Merge branch 'main' into gprusak-stablehash

32fdea7

fixed peermanager priority function

6acbaeb

Merge branch 'main' into gprusak-stablehash

48d0d13

sei-will approved these changes Mar 17, 2026

View reviewed changes

wen-coding approved these changes Mar 17, 2026

View reviewed changes

Merge branch 'main' into gprusak-stablehash

4d907fe

pompon0 enabled auto-merge March 18, 2026 19:08

Conversation

pompon0 commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pompon0 Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wen-coding left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pompon0 Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pompon0 commented Mar 6, 2026 •

edited

Loading

github-actions bot commented Mar 6, 2026 •

edited

Loading

pompon0 Mar 16, 2026 •

edited

Loading

pompon0 Mar 18, 2026 •

edited

Loading