Deadlock when both sides send ResendRequest simultaneously

[davidsteiner]

When both sides detect a sequence number gap at the same time, they each send a ResendRequest. However, in AwaitingResend state, all messages with a sequence number above end_seq_number are unconditionally queued (session.rs, process_message). This means each side queues the other's ResendRequest instead of processing it, and neither side ever starts resending — creating a deadlock.

The fix is to exempt ResendRequest (type "2") and SequenceReset (type "4") from being queued, since they are part of the gap recovery mechanism itself and must always be processed immediately.

[davidsteiner]

Reference: QuickFIX/J addressed this same deadlock in QFJ-673:

    private void nextResendRequest(Message resendRequest) throws IOException, RejectLogon,
            FieldNotFound, IncorrectDataFormat, IncorrectTagValue, UnsupportedMessageType,
            InvalidMessage {
        // QFJ-653: Check if seqnums are too low.
        // QFJ-673: Do not check if seqnums are too high in verify() since in case of a seqnum mismatch this will
        // enqueue the ResendRequest for later processing. This might lead to a deadlock situation in
        // which the counterparty waits for our messages to be resent and we are also waiting for our
        // ResendRequest to be satisfied in order to process the queued ResendRequest of the counterparty.
        // Instead, send out the requested messages and afterwards enqueue the ResendRequest in order to
        // later increase the target seqnum in method nextQueued(int).

Their fix in nextResendRequest skips the too-high check in verify so the ResendRequest is processed immediately, then explicitly queues it so the sequence number is accounted for when the gap fills catch up.