Streams Are Not Queues

02:47 UTC

Riya’s phone buzzes on the nightstand. An on-call alarm.

PagerDuty: payment-retry-consumer-group — lag 340,000 and climbing.

She opens her laptop half asleep. 340,000 payments sitting unprocessed. Somewhere in that pile is a retry for a real customer’s rent payment, stuck for eleven minutes.

She pulls up the consumer group. payment.retry.requests. Six partitions. One of them, partition 3, is wedged. A single payment record with a malformed currency code sits at the front, and every loop iteration throws the same exception. Every message behind it in that partition? Blocked. Waiting. Doing nothing.

The five other partitions are keeping up fine. But partition 3 owns every retry for a chunk of customers (by hash), and it is not moving. Nothing skips the bad record. Nothing routes around it. Nothing dead-letters it.

Six months earlier, the engineer who built this had made a reasonable decision: the team already ran Kafka, already trusted it, already knew how to monitor it. Why stand up a second messaging system for something as small as payment retries?

Riya restarts the consumer, skips the offset by hand, writes an incident report, and goes back to bed at 4:15 AM. In the morning stand-up she says the sentence that starts this whole post:

“We used a stream where we needed a queue.”

Two different questions

Riya had used Kafka for three years. She could explain partitioning to a new hire. She could sketch the topology on a whiteboard from memory. And she still built this, because the trap isn’t about knowing Kafka. It’s about not noticing that two tools that look similar (“systems that move messages between services”) are answering completely different questions.

A stream answers: what happened, in what order, and who else needs to know about it?

A queue answers: here’s a task. Do it once. Tell me if it didn’t work. The order doesn’t matter much.

That second part is easy to miss. A stream guarantees ordering within a partition. A queue deliberately trades ordering away so that any available worker can grab any available record. That trade-off is why queues can isolate a single poison message while streams can’t, and it’s also why you’d never want a queue processing your ledger postings, where “debit before credit” matters.

Mix the two up and you don’t get a slower system. You get what Riya got at 02:47: a task-shaped problem stuffed into a stream-shaped system, with none of the guardrails a real queue would have provided out of the box.

What a Kafka partition actually looks like

A Kafka partition behaves like a ledger. Append-only, ordered, retained. When a consumer reads offset 4, that record doesn’t disappear. It’s still sitting there, and it will keep sitting there until retention expires, regardless of how many consumer groups have already moved past it. Within a single partition, ordering is guaranteed: offset 3 is always processed before offset 4. That’s not a nice-to-have for a ledger service reconciling debits and credits. It’s a hard requirement.

That’s a powerful property when you need it. ledger-svc, fraud-svc, and a nightly analytics pipeline can all read the exact same payment events independently, at their own pace, without interfering with each other. You can replay six months of KYC events into a new risk model without touching production traffic. Records are consumed in the sense of observed, not destroyed.

But the same property is what made Riya’s night miserable. A classic consumer group can’t pull one record out, hold it aside, and hand it to a different worker if the first one crashes. Instead, each partition is owned by exactly one consumer in the group. Parallelism is capped by partition count, not worker count.

Kafka’s own project maintainers described this gap when they wrote KIP-932, the proposal that became “Queues for Kafka.” Teams were creating topics with far more partitions than the data actually justified, just to get enough parallel consumers. The consumer group protocol itself had no concept of acknowledging or retrying a single record: everything operated at the partition level. That’s exactly the wall Riya hit. Six partitions sized for “should be enough,” and one bad record that nobody could skip without SSH-ing into a box.

What a queue does instead

A queue makes a much narrower promise, and it makes a deliberate sacrifice to get there: it gives up strict ordering. A worker pulls a record. While it’s being processed, nobody else can see it (it’s checked out, not gone). The worker finishes and deletes it. If the worker crashes before the delete happens, the checkout expires after a timeout and the record becomes visible again for the next available worker — which might not be the same worker, and might process it out of the original sequence. The system counts how many times this has happened, and after a threshold, parks the record in a dead-letter queue.

That lack of ordering is the reason queues can isolate a poison message. Because records aren’t chained to each other in sequence, one record failing doesn’t hold up the rest. In a stream, ordering and head-of-line blocking are two sides of the same coin. In a queue, any worker can grab any visible record, so a stuck one just sits in timeout while everything else moves on.

SQS works this way. RabbitMQ works this way. Even a Postgres table can do it, using SELECT ... FOR UPDATE SKIP LOCKED, a locking clause built precisely so that several workers can each claim a different row from a shared job table without blocking each other or grabbing the same row twice.

A classic Kafka consumer group has none of this. No per-record acknowledgment. No automatic redelivery of just the one record that failed while everything else moves on. No delivery-count field to check. You either commit the offset and risk swallowing failures, or you don’t commit and risk reprocessing the whole batch behind the stuck one, which is the trap that jammed partition 3.

Same payment, different jobs

Here’s what actually clicked for Riya’s team once they sat down and re-drew the architecture: the same payment event legitimately needs to feed both patterns at the same time.

payment.events on Kafka is the stream. The ledger service, fraud scoring, and the data warehouse all read the same sequence of events independently, sometimes replaying from the beginning. Multiple readers, ordering matters (debits must land before credits), history matters.

Payment retries and webhook delivery are queue work. Each retry is a single task. It should be attempted, acknowledged if it succeeds, retried if it doesn’t, and dead-lettered if it keeps failing. One owner per task, no replay needed, and no reason retry #4821 has to finish before retry #4822.

Riya’s team had collapsed both needs into one Kafka topic because the broker was already there. The stream half of that decision was sound. The queue half is what paged her.

What retries actually cost to build in each system

This is where the incident post-mortem got interesting, because “we’ll just add retry handling” means very different things depending on what’s underneath.

With SQS, a dead-letter queue is two fields on the queue’s own config:

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:ap-south-1:111122223333:payment-retry-dlq",
    "maxReceiveCount": 5
  }
}

Every record tracks its own receive count automatically. Fail five times, SQS moves that single record into the DLQ. No retry logic, no routing code, no scheduler. Every other record in the queue keeps flowing the whole time.

With a classic Kafka consumer group, no such mechanism exists. A partition offset is committed or it isn’t; there’s no per-record attempt counter in the protocol. So teams end up building the same workaround: a chain of retry topics. Fail on payment.events? Produce the record onto payment.events.retry-5s. Fail again? retry-1m. Then retry-10m. Then a topic designated as the DLQ, with your own application code deciding when to give up.

Each arrow in that Kafka chain is a producer, a consumer, a schema, and a schedule that your team needs to maintain.

Now here’s the side-by-side of what Riya’s night would have looked like if the retry job had been on SQS instead:

On SQS, the same malformed record would have bounced through the visibility timeout five times over about six minutes, landed in a DLQ, and been reviewed on Monday. No page, no human, no other record affected. On Kafka, it took a 3 AM wake-up, an SSH session, a manual offset skip, and 99 minutes of blocked payments.

The honest counterargument: “we already run Kafka”

It’s not a simple “Kafka bad, SQS good” takeaway. Riya’s team wasn’t foolish to reach for Kafka. Sticking with infrastructure you already operate, monitor, and trust is a real argument. A second messaging system means more secrets to rotate, more dashboards to build, more failure modes to train new hires on.

And the gap is closing. Kafka 4.2 (released February 2026) shipped share groups as a GA feature via KIP-932. Share groups let multiple consumers pull from the same partition cooperatively, and each consumer explicitly acknowledges individual records as accept, release (put it back for retry), or reject (mark it as unprocessable). Delivery attempts are tracked per-record, not per-partition. That’s a genuine structural change, and it solves the core isolation problem that burned Riya.

Two caveats, though. First, DLQ support for share groups is still in development (KIP-1191, targeting Kafka 4.4 later this year), so the “reject and automatically route to a dead-letter topic” step still requires some application-level wiring as of today. Second, most existing Kafka deployments are still on classic consumer groups, where the retry-topic chain described above remains the reality. If you’re evaluating share groups now, budget time to test their failure modes in your own environment rather than assuming immediate parity with a system that’s been running these semantics for close to two decades.

The decision, simplified

Will more than one system need its own independent read through the same history? Does ordering within a partition matter (ledger reconciliation, fraud sequencing, audit reconstruction)? Then it’s a stream: retained, replayable, ordered, multi-reader. Kafka is the right tool.

Is the job really just “do this thing, try again if it fails, give up after N attempts”? Could the tasks run in any order without breaking correctness? Then it’s a queue: single-owner, delete-on-success, dead-letter on failure. SQS, RabbitMQ, or a Postgres SKIP LOCKED table will all do it, and they’ve been doing it for years.

If the answer is “both” (and in a fintech payments system, it usually is), that’s not a contradiction. It’s a sign you need the same event feeding a stream for event streaming usecase and a queue for task execution. Two systems, each doing the job it was shaped for.

Few weeks later

The page itself didn’t need a migration to fix. The actual bug was narrower: the consumer retried the same offset forever and never committed past it, win or lose. The fix that went out the next day was a small one — catch the exception, check the attempt count already carried by the retry-topic name, and either forward to the DLQ topic and commit the offset. That alone stopped partition 3 from blocking anything behind it. No new infrastructure, just a consumer that gives up correctly instead of looping forever.

That fix was enough to stop the paging. What it didn’t fix was the maintenance cost: a hand-rolled chain of retry-5s / retry-1m / retry-10m / DLQ topics, a header-based attempt counter every consumer had to check, and a producer for each hop. Riya’s team spent the following weeks deciding whether that cost was worth carrying long-term, and concluded it wasn’t — for payment retries specifically, where ordering never mattered in the first place.

So payment.events stayed exactly where it was: Kafka, six partitions, read independently by the ledger, fraud, and analytics services, replayed once when the fraud model needed six months of history to retrain on. Payment retries moved to SQS a few weeks later, as a deliberate simplification rather than an emergency fix — the retry worker was rewritten around SQS’s poll-and-delete model, made idempotent against SQS’s occasional duplicate delivery, and shadow-run against the old path for a week before the cutover. The DLQ now collects whatever fails, reviewed every Monday, with no custom retry-topic code left to maintain.

The week after cutover, a new malformed currency code came through. The SQS queue bounced it five times, parked it in the DLQ, and nobody’s phone buzzed. Riya saw it Monday morning in the review, fixed the upstream validation, and closed the ticket before lunch.

The system that runs your background jobs right now: did someone choose it for the job, or is it just whatever was already running?

Opined and edited by me, drafted and visualised with AI. © Shahbaz Ahmed, 2026. Shared under CC BY 4.0 — use it, adapt it, just credit it.

• ← Previous Post