Kafka's promise to a consumer group is that if one consumer goes away, the others pick up its partitions. I wanted to see what that looks like from the consumers' side: how long the dead consumer's partitions sit unread, what happens to the messages it was in the middle of, and whether the consumers that stayed alive notice anything.
The setup
I made a managed Kafka 4.3 cluster on DigitalOcean, the smallest size: three brokers with 2 vCPUs and 4 GB each, in Frankfurt. It was online about five and a half minutes after I asked for it. A small Droplet in the same private network ran the clients, using the Python confluent-kafka package, which is librdkafka 2.16 underneath.
Each run got its own topic with 6 partitions, replicated 3 times. A producer wrote numbered messages at 300 a second for 150 seconds, with acks=all and idempotence on, and logged every acknowledgement. Three consumers in one group read them. Each message took 5 ms of "work", and each consumer logged every message it finished, with the partition and a timestamp.
Forty seconds in, I took one consumer away in one of three ways:
-
SIGTERM. The consumer catches it, finishes the message it's on and calls
close(), which tells the group it's leaving. - SIGKILL. The process is gone, no goodbye.
- SIGSTOP. The process freezes and stops heartbeating but keeps its connections open. Sixty seconds later I sent SIGCONT to wake it up. This is the closest I could get to a long garbage collection pause or a VM that stalled.
I ran each of those with two commit setups, and with both group protocols Kafka 4 offers:
-
Commits. "Auto" is the default: the offset counts as done as soon as
poll()hands the message over, and it's committed every 5 seconds. "Manual" stores the offset only after the work is finished, still committed every 5 seconds. -
Protocols. The classic rebalance protocol, and the new consumer protocol from KIP-848, where the broker works out the assignment. DigitalOcean's cluster had both turned on, so switching was one client setting,
group.protocol.
That's 12 runs, plus one trial run before them. Afterwards I compared what the producer had acknowledged with what the consumers had finished.
Nothing was lost
In all 21 runs, counting the trial and eight I added later, every acknowledged message was processed at least once. That held for the auto commits too, which in theory can lose a message if the consumer dies after the offset is stored and before the work is done. With 5 ms of work per message that window is small, and I never hit it.
A clean shutdown costs very little
With SIGTERM there were no duplicates in any run. The dead consumer's partitions were picked up by the others in 2.5 to 3 seconds with the classic protocol, and in 0.5 to 0.8 seconds with the new one.
The new protocol also didn't disturb anyone else. With the classic protocol, every consumer gave up all of its partitions and got a new set, even the two that weren't going anywhere. That pause was short here, under 200 ms. With the new protocol, the two survivors kept what they had and each just received one more partition.
A crash costs 45 seconds
With SIGKILL or SIGSTOP, the dead consumer's partitions sat unread for 40 to 49 seconds in every run, with either protocol. That's the session timeout. The group can't tell a crashed consumer from a slow one until 45 seconds pass without a heartbeat. In the new protocol the broker sets that timeout, with heartbeats every 5 seconds, which is part of why its runs spread from 40 to 49 seconds. The classic runs landed between 42 and 47.
So the new protocol is much faster at a planned exit and no faster at a crash. If your consumers mostly die by deployment, it helps. If they die by OOM kill, it doesn't.
Every crash replayed work
Each SIGKILL and SIGSTOP run in the main set produced between 459 and 493 duplicates. They were all messages the dead consumer had already finished, but whose offsets hadn't been committed yet. When its partitions moved, the new owner started from the last committed offset and did them again.
Auto and manual commits made no difference to this. Both commit every 5 seconds, and both lose up to 5 seconds of progress in a crash. Manual storing only changes whether a message can be skipped, and nothing was skipped either way.
The survivors stalled too
I expected the two remaining consumers to carry on as before. They didn't. Their own partitions, the ones they'd had all along, went quiet for 12 to 15 seconds after the takeover.
The logs showed why. When a survivor got the dead consumer's partition, there were about 45 seconds of backlog waiting on it, around 2,400 messages. The consumer worked through that backlog as one block, which took about 13 seconds, before it returned to its own partitions. Its own messages kept arriving and just waited.
With the classic protocol, the default assignor often gave both of the dead consumer's partitions to the same survivor: in 7 of 11 classic runs. That survivor then worked through two blocks, one after the other, about 26 seconds in all. In some of those runs it slipped a few of its own messages in between the two blocks, so its own partitions waited about 13 seconds. In three runs it didn't, and they waited 18 to 27 seconds. I don't know what decided that. The new protocol split the two partitions between the survivors in all 10 of its runs.
The frozen consumer knew it was dead
I was most curious about the SIGSTOP case, because that's where the textbook warns about zombies. A consumer wakes up, still thinks it owns its partitions, and processes messages that someone else now owns.
It barely happened. When the frozen process woke up, librdkafka noticed within 22 ms that its session had expired and reported the partitions as lost. Between waking and rejoining, it processed 0 to 4 messages per run, all of which the new owner had already done. Then it rejoined the group, 2.5 seconds later with the classic protocol and 5 seconds with the new one. That triggered another rebalance, so the comeback cost a second, smaller interruption.
That's this client's behavior. librdkafka keeps its own track of when it last heard from the group, so it knew without asking. I didn't test other clients.
The duplicates depended on the clock
The main runs all landed close to the maximum replay. With about 100 messages a second per consumer and a commit every 5 seconds, a crash can replay up to roughly 500. That looked suspicious, since the kill always came at the same second of the run.
So I ran eight more SIGKILL runs with auto commits, four per protocol, killing at 41, 42, 43 and 44 seconds instead of 40. The duplicates came out as 79, 181, 291 and 378 with the classic protocol, and 84, 197, 286 and 369 with the new one. About 100 more for every second since the last commit. In the main runs, the kill had landed just before a commit every time.
So the number to plan for isn't 480. It's anything from zero up to five seconds of a consumer's work, depending on when it dies.
What I got wrong
I thought the commit mode would matter most. I set up the whole matrix around auto versus manual, and it was the one setting that changed nothing measurable. The session timeout and the assignor decided almost everything.
My first summary table was also wrong. For each run, it took the moment the first survivor in my list got one of the dead consumer's partitions, not the earliest moment either survivor did. One frozen-consumer run came out as a 62-second takeover when it was really 44 seconds. The 62 was the second rebalance, after the frozen consumer came back.
I also explained the survivors' longest stalls wrong at first. After the main runs, I put them down to one survivor getting both of the dead consumer's partitions. The extra runs had that too, three times, and all three stalled for only about 13 seconds. Getting both partitions makes the longer stall possible, but it doesn't guarantee it.
And my gap measurement started out naive. It measured the time between the last message before the kill and the first after, which counted the message the SIGTERM'd consumer finished on its way out as "after". I changed it to the longest silence on each partition in the two minutes after the kill.
Caveats
This is one client library, one message rate, and one size of work per message. The duplicates scale with how much a consumer gets through in 5 seconds, so a faster consumer replays more. Each combination ran once, apart from the extra runs above. The cluster was otherwise idle, and the clients sat in the same data center as the brokers.
What I'd take from it
- Make consumers idempotent. A crash replayed up to 5 seconds of work, whichever commit mode I used. A shorter commit interval shrinks that, at the cost of more commits.
- Handle SIGTERM and call
close(). That's the difference between a 0.5-second handover with no duplicates and 45 seconds of silence plus a replay. - If you're on Kafka 4 and librdkafka, try
group.protocol=consumer. It made planned exits about four times faster, didn't stop the other consumers, and spread the dead consumer's partitions more evenly. - A lower session timeout shortens the crash case, and brings its own risk of kicking out consumers that are only slow.
- Watch the survivors, not just the dead one. Their own partitions stalled for 12 to 27 seconds while they caught up on someone else's backlog.
Top comments (0)