We, Raul K (@raulvk, Protocol Labs, libp2p) and I (@protolambda, EF Eth2 research), look forward to the gossipsub testing and appreciate the work so far, but have some concerns about the setup of the tests. Primarily about the testing network topology. Here is our feedback:
Whiteblock gossipsub tests feedback
Since gossipsub is not just being tested for organized and small networks, but for a random, large scale and decentralized network like Eth2, topology of network tests are a big concern.
We feel that topology properties where gossips make a difference, such as graph components with loops and routes with different limitations, are largely ignored.
Some of the libp2p performance and gossipsub properties may be derived from basic topologies still, but we simply cannot rely on just these for the full extent of the Eth2 gossipsub use cases.
Linear topology: The value of this test setup is not clear. It is unrealistic that nodes in a real p2p network will be daisy chaining with one another, so this test does not mimic a setup we are likely to encounter in production. Unfortunately all test series except for 10 use this topology, so they will produce mostly meaningless results.
It’s also unclear how measurements would be done, as the test executor selects message producers randomly. So the observations will be entirely different with each message, as messages propagate to the edges.
Tree structure: Similar to the older whiteblock tree topology referenced here, it limits messages to a single propagation path, completely diminishing the effects of gossip and putting nodes in stress-points unrealistically.
Fully connected: Again, the results of gossip are diminished because of every node only being one hop away from any information. Although the difference between mesh and fanout in gossipsub should affect this somewhat, it is mostly unrealistic.
We like to start a discussion on how to improve the topology and testing, before resources are wasted on running these existing topologies.
A simple start would be a network with each peer randomly connected to N other peers (both directions, i.e. not over-connect the initial nodes during generation). We are aware there are concerns with this too (randomness being the primary concern), and like to work to a solution for this, instead of resorting to oversimplification of the topologies. Gossipsub is a large effort, and there are definitely ways to formulate a good testing topology with joint network expertise.