What Is New in Apache Kafka 2.3
This release brings incremental stability improvements, new metrics, and enhanced client functionality. Here's a quick summary of the key changes.
| Category | Key Updates |
|---|---|
| New Features | Sticky Partitioners, KIP-392, KIP-360, KIP-455 |
| Improvements | New metrics, Enhanced TLS support, Log cleaner improvements |
| Bug Fixes | Numerous fixes for replication, clients, streams, and connect |
| Deprecations | Deprecated Scala 2.11 support, some ZooKeeper configs |
What are the most impactful client-side improvements?
The producer gets a major boost with the sticky partitioner. This reduces latency by batching records more effectively for the same partition, instead of the old round-robin approach.
KIP-392 lets consumers fetch from the closest replica, which can significantly cut down on cross-data-center traffic. This is huge for geo-distributed clusters where network latency is a primary concern.
For those building admin tools, KIP-360 finally fixes the issue where broker IDs could appear negative, cleaning up those confusing metrics and logs.
How does Kafka 2.3 improve cluster stability and monitoring?
A slew of new metrics gives you better visibility. You now get new metrics for request handlers, replication throttling, and the log cleaner, making it easier to pinpoint bottlenecks.
The log cleaner was made more robust to handle corrupt indexes gracefully, preventing a single bad segment from halting the entire cleaning process. In practice, this means fewer late-night pages for log cleaner stalls.
TLS connections are now more reliable with support for the TLSv1.3 cipher suite and fixes for TLS handshake failures that previously caused connection drops.
What should developers know about the Streams API changes?
KIP-455 adds preliminary support for rack-aware task assignment, laying the groundwork for future optimizations that place tasks closer to their state stores.
There are key bug fixes for handling windowed stores and suppressing operators, which were prone to errors during certain restoration scenarios. This matters because it prevents data loss or duplication in stateful stream processing applications.
Is there anything deprecated that requires immediate action?
Support for Scala 2.11 is now officially deprecated. You'll want to start planning your migration to Scala 2.12 for building applications.
A few ZooKeeper configuration properties like zookeeper.set.acl and zookeeper.ssl.cipher.suites are deprecated in favor of their direct counterparts (zookeeper.client.secure and zookeeper.ssl.ciphersuites). Update your configs to avoid warnings.
FAQ
What exactly does the new "sticky partitioner" do for producers?
It batches records more efficiently. Instead of switching partitions per record (round-robin), it sticks to a single partition until the batch is full or a timeout occurs. This reduces latency and improves throughput by creating fuller batches.
Does KIP-392 (rack-aware fetching) work automatically?
Yes, once configured. You set a rack ID for each broker (broker.rack), and the consumer will automatically prefer reading from a replica in the same rack, minimizing costly cross-rack network traffic.
I saw negative broker IDs in metrics -- is that fixed?
Yes, KIP-360 fixes the bug that caused broker IDs to be represented as negative integers in some JMX metrics and logs. They now correctly show the positive broker ID.
Should I be concerned about the Scala 2.11 deprecation?
Not for immediate runtime operation, but for development. Your existing Kafka 2.3 cluster runs fine, but you should start compiling any custom code (like Streams apps or connectors) with Scala 2.12 to ensure future compatibility.
Were there any critical bug fixes for Kafka Connect?
Several. Key fixes include resolving an issue where the JSON converter could fail on decimal types and a bug that prevented task reconfiguration upon connector restart, improving the overall stability of data pipelines.