What Is New in Elasticsearch 6.4
Elasticsearch 6.4 delivers significant enhancements across search, security, and data management. This release introduces cross-cluster replication, new SQL features, and a more robust security framework.
| Category | Key Updates |
|---|---|
| New Features | Cross-cluster replication (beta), SQL support for arrays, Painless scripting enhancements, JDBC driver |
| Security | Role-based access control for documents and fields, New built-in roles, PKI realm support |
| Improvements | Index lifecycle management preview, Frozen indices, Better disk usage for sparse fields |
| Deprecations | Deprecated the `_optimize` API in favor of the `_forcemerge` API |
How does cross-cluster replication work in 6.4?
Cross-cluster replication (CCR) enters beta, enabling indices to be automatically replicated from a remote cluster to a local one. This is a game-changer for geo-redundancy and disaster recovery scenarios.
You set up a follower index that pulls changes from a leader index on a remote cluster. In practice, this means you can keep a hot copy of your critical data in a different geographical location without manual intervention.
What SQL and scripting enhancements were added?
SQL support gets a major boost with the ability to handle arrays and the introduction of a dedicated JDBC driver. You can now run SELECT statements that return array types, making it much easier to work with complex data structures directly from SQL.
For Painless scripting, new methods and capabilities have been added. This includes better support for regular expressions and additional string manipulation methods, giving developers more power to write complex logic within their queries and ingest pipelines.
How is security improved in this release?
Security gets finer-grained with document and field level security. You can now define roles that restrict access not just to an index, but to specific documents within it and even specific fields within those documents.
New built-in roles like monitoring_user and remote_monitoring_agent simplify setting up monitoring. The PKI realm also allows for authentication using client certificates, providing a more secure alternative to basic auth for certain use cases.
What new data management features are available?
A preview of Index Lifecycle Management (ILM) is introduced, providing a framework to automate how indices are managed over time. While the full UI comes later, the APIs are available to start automating rollovers and other policies.
Frozen indices are a new way to handle read-only archival data. They consume significantly less heap memory by storing the index metadata in a more efficient manner, which is perfect for data you need to query infrequently.
Sparse Field Storage
The way sparse fields are stored on disk has been optimized. Indices with many fields where most documents only populate a small subset of them will see a reduction in disk usage, which directly lowers storage costs.
FAQ
Is cross-cluster replication production-ready in 6.4?
No, it is marked as a beta feature. It's recommended for testing and evaluation purposes but you should avoid using it for mission-critical production workloads until it is marked as stable.
Can I use the new SQL features with my existing BI tools?
Yes, the new JDBC driver enables connectivity with a wide range of BI and analytics tools that support the JDBC standard, allowing them to query Elasticsearch directly.
What is the main benefit of frozen indices?
Frozen indices drastically reduce heap memory usage for archival, read-only data. This allows you to keep more historical data searchable without needing to scale your cluster's memory resources proportionally.
How does document-level security work?
It uses a query defined in a role to restrict which documents a user can see. For example, a user role could have a clause like "term": { "department": "engineering" } so they only see documents where the department field matches.
Is the deprecated `_optimize` API immediately removed?
No, deprecation is a warning. The `_optimize` API continues to function but you should migrate to using the `_forcemerge` API, as `_optimize` will be removed in a future major version.