What Is New in Apache Kafka 3.1
This release brings incremental improvements focused on stability, performance, and developer experience. Here's a quick summary of the key changes.
| Category | Key Updates |
|---|---|
| Core Protocol & Broker | Support for ZStandard compression at the broker level, new AlterPartition RPC, and improved handling of offline log directories. |
| Kafka Streams | State store metrics, improved grace period handling for windowed operations, and a fix for a potential infinite loop during restoration. |
| Connect & Admin | Ability to alter client quotas via kafka-configs.sh and a fix for a Connect framework NPE. |
| Bug Fixes | Numerous fixes across the board, including issues with group coordination, replication, and the Raft implementation. |
How did broker-side compression get an upgrade?
Brokers can now use ZStandard compression for repressing messages from producers that used a different codec. This is a big deal for optimizing storage and network traffic without forcing all your producers to change their configuration.
You enable it by setting compression.type=zstd on the broker. In practice, this lets you centralize compression logic, making it easier to enforce efficiency standards across all data flowing into a cluster.
What's the new AlterPartition RPC for?
It replaces the old ZooKeeper-based path for updating partition metadata, which is a critical step towards fully removing the ZooKeeper dependency. This makes the controller more robust and simplifies the internal architecture.
For developers, this change is mostly under the hood. You won't interact with it directly, but your cluster benefits from a more stable and modern metadata management system.
What Kafka Streams improvements should I know about?
This release adds new metrics for state stores, like stream-state-store-avg-latency and stream-state-store-put-rate. These are invaluable for debugging performance bottlenecks in your stateful applications.
They also fixed a tricky bug where a state store restoration could get stuck in an infinite loop if an I/O exception occurred. This matters because it prevents a single bad segment from halting your entire application during startup.
How is log directory failure handling better?
Brokers are now smarter about handling log directories that go offline. Previously, an offline directory could cause widespread problems. The improved logic helps isolate the failure, preventing it from cascading and affecting other healthy directories.
This means your cluster has a better chance of staying operational and accepting messages on good disks even if one disk fails.
FAQ
Should I immediately upgrade my producers to use ZStandard because of the new broker support?
No. The broker-side feature is for re-compressing data. Your producers can continue using their current codec (like snappy or gzip). The broker will just convert it to ZStandard, giving you the storage benefits without a client-side rollout.
Does the AlterPartition RPC change how I administer topics?
No, this is an internal protocol change. All your existing tools and scripts, like kafka-topics.sh, will continue to work exactly as before.
I had a Streams app that sometimes hung on startup. Could this version fix it?
Possibly. The fix for the infinite loop during state store restoration (KAFKA-13646) directly addresses a class of startup hangs caused by I/O issues during recovery.
Are there any new metrics I should add to my dashboards for Kafka Streams?
Yes, definitely add the new state store metrics (e.g., stream-state-store-avg-latency). They provide much-needed visibility into the performance of your stateful operations, which is often a source of latency.
Is there a performance impact from the improved offline log directory handling?
There should be no negative performance impact. The improvement is about fault tolerance--it makes the broker more resilient to hardware failures without adding overhead during normal operation.