EndOfStream DUPLICATE_BLOCK repeated stream restarts

[derektriley]

When a Block Node has already persisted and verified a block, if the consensus node sends the BlockHeader for that block, it will receive a
PublishStreamResponse , EndOfStream with Code of DUPLICATE_BLOCK. The CN will end the stream and restart the stream at the specified block_number + 1 in the EndOfStream.

This can lead to a situation where if multiple consensus nodes are streaming to the same block node, one consensus node can be in a state where it is constantly behind, being told it's duplicating an existing block and the stream will be continually restarted. The consensus node currently has a rate limiting to EndOfStream messages, where after a certain number of them occur within a timeframe, a longer backoff is applied.

Potential Solutions:

1. Send a SkipBlock instead of an EndOfStream if the block falls within a buffer/window (last x blocks).
2. Make the EndOfStream DUPLICATE_BLOCK it's own message type. The CN would treat this as an acknowledgement + skipping to streaming that block.
3. Remove EndOfStream DUPLICATE_BLOCK and if the CN sends a BlockHeader for an already persisted and verified block, then immediately send the lastest BlockAcknowledgement.

[jsync-swirlds]

We'll need to discuss this carefully.
The block node ends the stream when a duplicate block is encountered for good reason, and we probably should continue to do so. Also, any given publisher may encounter a consistent stream of DUPLICATE_BLOCK responses under normal operation and this is expected. So long as the publisher takes note of the last known verified and persisted block (block_number field) and sends blocks after that block, it should remain stable.

It might be reasonable to offer a SKIP_BLOCK in limited circumstances, but we need to be very careful about that to ensure the block node remains reliable against a buggy/misbehaving publisher or unreliable network.

[ata-nas]

We'll need to discuss this carefully. The block node ends the stream when a duplicate block is encountered for good reason, and we probably should continue to do so. Also, any given publisher may encounter a consistent stream of DUPLICATE_BLOCK responses under normal operation and this is expected. So long as the publisher takes note of the last known verified and persisted block (block_number field) and sends blocks after that block, it should remain stable.

It might be reasonable to offer a SKIP_BLOCK in limited circumstances, but we need to be very careful about that to ensure the block node remains reliable against a buggy/misbehaving publisher or unreliable network.

Indeed this is something we should very thoroughly discuss. I do not think we should be mixing SKIP and DUPLICATE. If we decide to go with not closing the stream as described above, this, at least to me, means an API/Spec change (making DUPLICATE it's own message and not an EndOfStream status code) and should be very deliberate and well though out.

[jsync-swirlds]

Indeed this is something we should very thoroughly discuss. I do not think we should be mixing SKIP and DUPLICATE. If we decide to go with not closing the stream as described above, this, at least to me, means an API/Spec change (making DUPLICATE it's own message and not an EndOfStream status code) and should be very deliberate and well though out.

I don't think we can have DUPLICATE_BLOCK not close the stream; basically options (2) and (3) are too high risk to allow (at least that's my current opinion).

[Nana-EC]

Consideration to convert to discussion. From that pull out a potential ticket or not.