Apache Kafka 3.4: What's New, Release History & Support Lifecycle

What Is New in Apache Kafka 3.4

This release brings significant enhancements to KRaft, new metrics for better cluster observability, and several key API improvements. It's a solid step forward in the maturation of the KRaft metadata mode.

How has KRaft improved in this release?

KRaft, Kafka's built-in metadata system, received major operational features. The headline is full support for JBOD (Just a Bunch Of Disks) configurations in KRaft mode, which was previously only available for ZooKeeper-based clusters. This allows brokers to manage multiple log directories on different disks for better I/O handling and failure isolation.

KIP-858 introduces the ability to dynamically update the broker endpoints a cluster advertises to clients. You no longer need a full rolling restart to change security protocols or listener names. In practice, this makes scaling and reconfiguring clusters in cloud environments much smoother.

Under the hood, numerous fixes improve controller failover stability and correctness, making KRaft even more production-ready. These changes are critical for teams planning their migration away from ZooKeeper.

What new metrics should I start watching?

This release adds observability into several key internal operations. New metrics for partition reassignments (kafka.controller:type=KafkaController,name=PartitionReassignmentTimeMs) give you visibility into how long these operations are taking, which is vital for capacity planning.

For teams using Tiered Storage, new metrics track the time spent on various remote log management tasks (kafka.log.remote:type=RemoteLogManager,name=*TimeMs). This helps pinpoint bottlenecks when offloading data to cheaper storage.

You also get new metrics around consumer group deletion latency and the metadata log's end offset, providing a clearer picture of cluster health and metadata propagation times.

Are there any new APIs for managing groups?

Yes, the new DeleteGroups API allows you to delete multiple consumer groups in a single operation. This is more efficient than the previous method of deleting groups one by one, especially for cleaning up test environments or removing many stale groups.

Another API addition, DeleteOffsets, lets you remove the stored offsets for a group without deleting the group itself. This is useful for resetting a group's position without losing its current state and membership information.

These APIs provide finer-grained control over group management, moving beyond the basic describe and delete operations that were previously available.

What other notable fixes and changes are included?

Several fixes address specific pain points. A race condition in the kafka-topics.sh script that could cause incorrect partition counts has been resolved. This matters because it was a silent error that could lead to unexpected cluster topology.

For RocksDB-based state stores, a bug was fixed where iterators might not be properly closed, potentially leading to memory leaks over time. There's also a fix for an issue where state store changelog topics could be created with the wrong configuration.

On the consumer side, the handling of negative timestamps in the ListOffsets API was corrected. These cumulative fixes improve the overall stability and predictability of the platform.