What Is New in Apache Kafka 2.6
Kafka 2.6 is a feature-packed release focusing on incremental improvements to core components, client libraries, and the Connect framework. It delivers better stability and more control for operators.
| Category | Key Highlights |
|---|---|
| New Features | Kafka Streams exactly-once semantics for standalone mode, AdminClient support for listing producer/consumer metrics. |
| Improvements | Enhanced log cleaner, better KRaft controller performance, new metrics, Connect REST API extensions. |
| Bug Fixes | Numerous fixes across core, clients, streams, and connect for improved reliability. |
| Deprecations | Older message format versions and some ZooKeeper-based APIs are deprecated. |
What are the core broker and client improvements?
The log cleaner saw significant optimizations for handling tombstones and avoiding unnecessary retries, which reduces I/O overhead on the broker. This is a big deal for clusters with high compaction workloads.
For client applications, the AdminClient now exposes listProducerMetrics and listConsumerMetrics methods. This gives you a centralized way to monitor client performance directly from your operational tools instead of scraping individual JMX endpoints.
How did Kafka Streams get better?
The standout feature is bringing exactly-once (EOS) processing semantics to standalone mode. Before 2.6, EOS was only available in the fault-tolerant, multi-instance mode. Now, even a single standalone application can guarantee no duplicate processing after a restart.
They also added a new metric, skipped-records-rate, to help you track when your stream processing topology is dropping records due to deserialization errors or other issues.
What's new for Kafka Connect?
The Connect REST API got more powerful. You can now list and describe connector configurations and specific task configurations directly through the API. This makes automated management and debugging of your Connect cluster much easier.
There's also a new alterConnectLoggers endpoint in the AdminClient, allowing you to dynamically change log levels for Connect workers without a restart. This is a lifesaver for debugging production data pipeline issues on the fly.
Are there any notable deprecations?
Yes, the older message format versions 0 and 1 are now formally deprecated. You should be using message format version 2, which has been the default for a while. This is a nudge to ensure everyone is on the more efficient format.
Several ZooKeeper-based methods in the AdminClient, like describeConsumerGroup with ZK, are also deprecated in favor of their newer, broker-based alternatives. This aligns with the long-term move towards the KRaft metadata mode.
FAQ
Does the new exactly-once for Kafka Streams standalone mode work with any state store?
Yes, it supports all state store types, including RocksDB and the in-memory stores. The state is persisted to disk and recovered to guarantee exactly-once semantics after a restart.
I use the AdminClient for monitoring; do the new metrics listing methods work with older brokers?
No, the listProducerMetrics and listConsumerMetrics methods in the AdminClient require the brokers to be on version 2.6 or newer to function properly.
What's the practical impact of the log cleaner improvements?
You should see reduced CPU and disk I/O on your brokers, especially if your topics use compaction and have a high volume of tombstones (delete markers). This translates to better overall cluster stability.
Is there a performance hit when using EOS in standalone mode?
There is a slight overhead because it requires writing state to disk more frequently to persist transaction boundaries. However, for most use cases, the guarantee of no duplicates is worth the minimal performance cost.
Why are the older message formats being deprecated?
Message format v2 includes several efficiencies, like improved batch handling and better compression. Deprecating the old formats simplifies the codebase and ensures all users benefit from the most efficient storage format.