JetStream Consumer num_ack_pending never becomes zero after application terminates

[davidmcote]

Observed behavior

If an application terminates before acking its messages from a durable Jetstream Consumer and no further batches of messages are requested from the Consumer, num_ack_pending in ConsumerInfo never becomes zero.

Expected behavior

num_ack_pending should become zero after the JetStream Consumer's AckWait period expires on all outstanding messages.

Server and client version

nats-server 2.10.18

Host environment

Any. Apple M3 Pro.

Steps to reproduce

1. Create a stream and durable consumer w/ explicit ack & ackwait.

% nats stream add mystream
? Subjects mysubject
? Storage memory
? Replication 1
? Retention Policy Interest
? Discard Policy New
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
Stream mystream was created

Information for Stream mystream created 2024-11-08 08:17:33

              Subjects: mysubject
              Replicas: 1
               Storage: Memory

Options:

             Retention: Interest
       Acknowledgments: true
        Discard Policy: New
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0


% nats consumer add
? Consumer name myconsumer
? Delivery target (empty for Pull Consumers)
? Start policy (all, new, last, subject, 1h, msg sequence) all
? Acknowledgment policy explicit
? Replay policy instant
? Filter Stream by subjects (blank for all)
? Maximum Allowed Deliveries -1
? Maximum Acknowledgments Pending 0
? Deliver headers only without bodies No
? Add a Retry Backoff Policy No
? Select a Stream mystream
Information for Consumer mystream > myconsumer created 2024-11-08T08:18:41-05:00

Configuration:

                    Name: myconsumer
               Pull Mode: true
          Deliver Policy: All
              Ack Policy: Explicit
                Ack Wait: 30.00s
           Replay Policy: Instant
         Max Ack Pending: 1,000
       Max Waiting Pulls: 512

State:

  Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
    Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
        Outstanding Acks: 0 out of maximum 1,000
    Redelivered Messages: 0
    Unprocessed Messages: 0
           Waiting Pulls: 0 of maximum 512

2 Receive a message without acknowledging it.

% nats req mysubject body
08:20:00 Sending request on "mysubject"
08:20:00 Received with rtt 371.167µs
{"stream":"mystream", "seq":1}


% nats consumer next --no-ack mystream myconsumer
[08:20:14] subj: mysubject / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0

body

3 Poll ConsumerInfo

Notice that last activity is greater than the ack wait period yet num_ack_pending still shows there's an outstanding message.

% nats consumer info mystream myconsumer
Information for Consumer mystream > myconsumer created 2024-11-08T08:18:41-05:00

Configuration:

                    Name: myconsumer
               Pull Mode: true
          Deliver Policy: All
              Ack Policy: Explicit
                Ack Wait: 30.00s
           Replay Policy: Instant
         Max Ack Pending: 1,000
       Max Waiting Pulls: 512

State:

  Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 Last delivery: 39.50s ago
    Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
        Outstanding Acks: 1 out of maximum 1,000
    Redelivered Messages: 0
    Unprocessed Messages: 0
           Waiting Pulls: 0 of maximum 512

The message is immediately ready for redelivery if another request comes along to pull messages.

 % nats consumer next --no-ack mystream myconsumer
[08:22:40] subj: mysubject / tries: 2 / cons seq: 2 / str seq: 1 / pending: 0

body

It seems the ConsumerInfo counters are only advanced by requests which mutate the underlying stream.

As a workaround, I tried to request a zero-sized batch.

natscli appears to no-op on the client side for such a silly request.

% nats consumer next --no-ack mystream myconsumer --count 0

I crafted the batch request manually, but nats-server clamps my batchsize back up to 1 and this counts against the delivery count of the sacrificial message.

% nats req '$JS.API.CONSUMER.MSG.NEXT.mystream.myconsumer' '{"expires":5000000000,"batch":0}'
08:25:23 Sending request on "$JS.API.CONSUMER.MSG.NEXT.mystream.myconsumer"
08:25:23 Received with rtt 401.791µs
body