What Is New in Apache Kafka 1.0
Apache Kafka 1.0 is a significant milestone focused on production stability and operational maturity. This release consolidates numerous improvements from the 0.11.x series and introduces several key features that simplify stream processing and cluster management.
| Category | Key Changes |
|---|---|
| New Features | Exactly-Once Semantics (EOS), Kafka Connect Exactly-Once Support, Java 9 Compatibility |
| Improvements | Enhanced Metrics, TLS Performance, Log Compaction Stability |
| Bug Fixes | Over 200 fixes from the 0.11.0 release |
| Deprecations | Deprecation of Scala-based producers/consumers, Old consumer offset storage in Zookeeper |
What are the core streaming improvements?
The headline feature is full support for Exactly-Once Semantics (EOS). This allows producers to write to multiple partitions and topics atomically, ensuring no duplicates in the event of retries. It's a game-changer for financial transactions and other use cases where duplicate data is unacceptable.
Kafka Connect also gets EOS support, making it possible to build end-to-end exactly-once data pipelines from sources like databases into Kafka and out to sinks. In practice, this means your ETL jobs become far more reliable without the complexity of deduplication logic in your applications.
How does this release improve operational stability?
Kafka 1.0 includes critical fixes for log compaction, a process that manages topic retention. Previous versions could see compaction threads get stuck under heavy load, potentially leading to uncontrolled disk usage. These fixes make the broker much more predictable in production.
TLS handshake performance saw major optimizations. Earlier versions could experience high CPU usage during TLS connection storms, which now scales significantly better. This matters because it reduces the overhead of running encrypted traffic in high-throughput environments.
What should developers know about API changes?
The old Scala-based producer and consumer clients are now officially deprecated. You should be using the Java clients (org.apache.kafka.clients.producer.KafkaProducer and KafkaConsumer) for all new development, as they offer better performance and are the focus of all new features.
Storing consumer offsets in Zookeeper is also deprecated. The new standard is to commit offsets directly to a Kafka topic (__consumer_offsets), which is handled automatically by the Java consumer. This simplifies the architecture by reducing dependency on Zookeeper for client operations.
Is there better support for modern Java runtimes?
Yes, Kafka 1.0 adds compatibility with Java 9. While the core still targets Java 8, this ensures the broker and tools can run on newer JVMs without issues. You won't see new features requiring Java 9, but it prevents classpath conflicts and warnings that were present in earlier versions.
The metrics system was overhauled to be more consistent and comprehensive. This gives operators better visibility into broker performance, client activity, and network health, which is crucial for debugging complex streaming applications.
FAQ
Should I upgrade to Kafka 1.0 for the Exactly-Once Semantics?
Yes, if you have use cases that require strict data correctness, like financial processing. EOS eliminates duplicate messages from producer retries, which simplifies application logic. Be prepared to upgrade your producers and consumers to use the new transactional APIs.
Is this release backwards compatible with my 0.10.x clients?
For the most part, yes. The wire protocol maintains compatibility, so older producers and consumers can talk to a 1.0 broker. However, to use new features like EOS, you must upgrade your client libraries to version 1.0.
What's the biggest operational win in 1.0?
The log compaction stability fixes are a major operational win. They prevent a class of bugs that could cause disk space to grow indefinitely, which was a real headache in previous versions. This makes the broker more hands-off in production.
Why are the Scala clients deprecated?
The Java clients proved to be more performant and easier to maintain. The Scala clients lagged behind in features and bug fixes. All future development is on the Java clients, so migrating is essential for long-term support.
Does Kafka 1.0 require Zookeeper 3.5?
No, it does not. Kafka 1.0 maintains compatibility with Zookeeper 3.4.x. The deprecation of storing consumer offsets in Zookeeper is a client-side change and does not mandate a Zookeeper upgrade for the cluster itself.