A familiar architecture: Kafka carries business events between services and into the lakehouse, while RabbitMQ, SQS or a database-backed job table handles background work such as sending emails, generating documents or calling a model. Two systems, two sets of operational skills, two places where messages can get stuck. The reason was structural. Kafka's consumer groups assign each partition to exactly one consumer, so you cannot scale workers beyond the partition count, and one slow message holds up everything behind it.
What changed
KIP-932, "Queues for Kafka", introduced share groups. Kafka 4.0, the first release to run only in KRaft mode without ZooKeeper, offered them as early access45. The 4.2.0 release announcement describes Kafka Queues as production-ready, adding a RENEW acknowledgement type, adaptive batching in the share coordinator, limits on fetched record counts, and lag metrics13.
Consumer groups versus share groups
The Kafka documentation describes share groups as an alternative to consumer groups: consumers share a topic's records without each partition being tied to one consumer, records are acknowledged individually, and delivery attempts are counted. It suggests them for workloads that process records one at a time rather than as an ordered stream2.
| Consumer group | Share group | |
|---|---|---|
| Assignment | Each partition to one consumer | Records shared across consumers |
| Scaling limit | Number of partitions | Not tied to partition count |
| Progress tracking | Offset per partition | Acknowledgement per record |
| Retries | Built by the application | Delivery attempts counted by Kafka |
| Ordering | Preserved within a partition | Not the design goal |
| Best for | Event streams, CDC, aggregations | Independent tasks: jobs, notifications, AI calls |
If the order of records matters, use a consumer group. If each record is an independent piece of work, a share group is now a real option.
Where it fits well
- AI and enrichment workloads. Calling a model for each document or record is slow and variable. Share groups let you add workers to match throughput without repartitioning, and a slow call doesn't block the rest.
- Fan-out jobs from existing events. If the events already land in Kafka, for example from change data capture, processing them as tasks without copying them to another broker removes a hop.
- Bursty background work, where worker count needs to swing widely during the day, combined with autoscaling like that in Kubernetes scale-to-zero.
Where a separate broker still wins
- No Kafka today. Adopting Kafka just for a work queue is rarely worth it. A managed queue is simpler.
- Complex routing. Topic exchanges, header-based routing and per-message priorities remain stronger in dedicated brokers.
- Strict ordering with parallelism. If you need per-key order and parallel processing, keyed consumer groups remain the right model.
- Small clusters. Share groups store state in an internal topic that defaults to three replicas, so clusters with fewer than three brokers must adjust internal replication settings first2.
How I'd adopt it
- Get to a current release. Share groups need KRaft-based Kafka 4.x; plan the ZooKeeper migration first if you haven't5. Run the latest 4.2 patch release rather than 4.2.0.
- Pick one queue workload that already consumes from Kafka and has idempotent processing, using the patterns in idempotent pipelines. Per-record acknowledgement still means at-least-once delivery.
- Decide what happens after the last attempt. Route repeatedly failing records somewhere a person will see them.
- Monitor share-group lag with the new metrics and alert on it, as you would queue depth1.
- Check your managed service. Hosted Kafka offerings adopt new features and metrics at different speeds.
- Retire the old queue only after a full cycle of peak load and incident handling on the new one.
How I can help
I design event and data platforms that stay simple as they scale, including when to consolidate systems and when not to. As a fractional CTO I can own that architecture. Or analyse your challenge and bring the result to a call.
- 1Apache Kafka 4.2.0 release announcementApache Kafka · 2026
- 2Upgrading to 4.2 (Kafka documentation)Apache Kafka
- 3Apache Kafka 4.2 releaseConfluent · 2026
- 4Kafka 4.0 and the KRaft architecture (news report)InfoQ · Apr 2025
- 5Upgrading to 4.1 (Kafka documentation)Apache Kafka