What Is New in Apache Kafka 3.2
This release brings incremental improvements to the core broker, the Kafka Streams API, and the Connect framework. The focus is on stability, performance tuning, and adding flexibility for developers.
| Category | Key Highlights |
|---|---|
| New Features | KIP-710: Pluggable Raft-based Metadata Quorum; KIP-744: More metrics in Kafka Streams |
| Improvements | KIP-735: ZStandard compression performance; KIP-709: Cluster-wide dynamic reconfiguration |
| Bug Fixes | Numerous fixes for stability in replication, streams, and client behavior |
| Deprecations | Deprecation of ZooKeeper-based metadata management continues |
What are the major broker and core improvements?
The headline feature is the groundwork for a pluggable metadata quorum. KIP-710 lays the foundation for eventually replacing ZooKeeper with a Raft-based implementation, which is a massive step towards a ZooKeeper-less Kafka. This is a long-term project, but 3.2 starts making it a reality.
For daily operations, KIP-709 is a big deal. It enables cluster-wide dynamic reconfiguration of certain broker parameters without needing a full restart. This means you can tweak settings like the number of background threads on the fly across the entire cluster, which is great for tuning performance in a live environment.
How does Kafka Streams get better in 3.2?
Observability gets a significant boost with KIP-744, which adds a slew of new metrics for Kafka Streams. You now get much deeper insight into the state of your state stores, including detailed metrics for built-in and custom stores. This is invaluable for debugging performance bottlenecks and understanding exactly what's happening inside your streams applications.
There are also improvements to the KStream#process API and fixes for issues around state store restoration and standby tasks. These changes make the framework more robust and predictable under heavy load or during failure recovery scenarios.
What performance enhancements should I know about?
KIP-735 tackles ZStandard compression performance head-on. It significantly reduces the CPU overhead when using ZStandard, especially for smaller batch sizes. In practice, this means you can get that great compression ratio without taking as big of a hit on throughput, making ZStd a more viable default choice for many use cases.
There are also under-the-hood optimizations in the consumer group protocol and replica fetchers that reduce unnecessary network traffic and improve overall cluster efficiency. These are the kinds of changes that just make everything feel a bit snappier without you having to change any code.
Are there any changes for Kafka Connect?
Yes, the Connect framework gets its share of love. The most notable change is the official support for incremental cooperative rebalancing. This has been in the works for a while and helps minimize the downtime and disruption when connectors or tasks are added or removed from a Connect cluster.
There are also several bug fixes related to exactly-once semantics and source connector offset tracking. For anyone running mission-critical data pipelines, these fixes improve the reliability and correctness of data ingestion and egress.
FAQ
Is Kafka 3.2 finally ZooKeeper-free?
No, not yet. KIP-710 is the first major step towards a pluggable metadata quorum, but the default and only implementation in 3.2 still uses ZooKeeper. This release is about building the infrastructure so that a Raft-based implementation can be plugged in later.
Should I switch to ZStandard compression because of KIP-735?
It's a great time to test it. The reduced CPU overhead makes ZStandard much more competitive with snappy for high-throughput scenarios while offering a better compression ratio. Benchmark it with your specific data to see the trade-offs.
Do the new Kafka Streams metrics impact performance?
There is always a small overhead for collecting metrics, but the added visibility is almost always worth the minimal cost. The new metrics are focused on state stores, which are often the source of performance issues, so they pay for themselves in debugging time saved.
Can I dynamically change all broker configs now with KIP-709?
No, only a specific subset of configurations is supported for dynamic, cluster-wide updates. The classic per-broker dynamic configs still work as before. Check the documentation for the exact list of cluster-wide configs.
Is upgrading to 3.2 from 3.x straightforward?
For the most part, yes. It's a minor version upgrade with no breaking API changes. As always, test your clients and streaming applications in a staging environment first, paying close attention to the deprecated warnings if you're still using any ZooKeeper-based tools.