What Is New in Elasticsearch 6.6
Elasticsearch 6.6 delivers key enhancements in indexing performance, cross-cluster search, and SQL functionality. This release focuses on making distributed operations smoother and providing better tools for data analysis.
| Category | Key Updates |
|---|---|
| New Features | SQL support for ROLLUP, Frozen Indices, Cross-Cluster Search from SQL |
| Performance | Faster indexing with "index sorting" and "soft deletes" |
| Resilience | Improved cross-cluster search failover and recovery |
| Deprecations | Deprecation of the `_optimize` API |
How did indexing performance get a boost?
The indexing throughput saw significant improvements through two main features. Index sorting allows for physically organizing data on disk as it's written, which speeds up range queries. Soft deletes change how documents are marked for deletion, reducing the I/O overhead during heavy indexing workloads.
In practice, this means bulk indexing operations and time-series data ingestion can run much faster. The underlying changes reduce segment merging costs and make the entire process more efficient, which is crucial for high-volume logging and metrics pipelines.
What's new with SQL and ROLLUP?
This version introduces experimental SQL support for querying rolled-up data. You can now use familiar SQL syntax to run aggregations on your summarized indices created by the Rollup feature, bridging the gap between raw and pre-aggregated data analysis.
This is a big deal because it allows analysts to use standard tools to query both detailed and summarized historical data without switching contexts. The syntax integrates directly, so a query like SELECT HISTOGRAM(time, INTERVAL 1 HOUR) AS time, AVG(price) FROM my_rollup_index GROUP BY time becomes possible.
Can I search across clusters more reliably now?
Yes, cross-cluster search (CCS) received important resilience upgrades. The system now handles cluster failures more gracefully during a search operation. If a remote cluster is unreachable, the query will continue executing on the remaining available clusters instead of failing completely.
This matters for production setups where you're spanning multiple regions or environments. You get partial results and can still serve most of your query rather than presenting a total failure to your application users, improving overall system stability.
What are Frozen Indices and why should I care?
Frozen Indices are a read-only, high-density storage format for older, rarely accessed data. They allow you to keep more historical data searchable without consuming the resources of a fully loaded index. You query them using a special `_search` endpoint.
For managing large-scale time-series data like logs, this is a cost-effective way to maintain access to cold data. The trade-off is that queries will be slower, so it's best suited for archival data that you only need to occasionally investigate.
FAQ
Is the SQL support for Rollup ready for production use?
It's marked as experimental in 6.6. While powerful for analytics, you should test it thoroughly for your specific use case before relying on it for mission-critical production workloads due to potential changes in future versions.
How do I start using index sorting to improve performance?
You enable it at index creation time by defining the `index.sort.*` settings in your index mapping. This needs to be planned upfront as it cannot be changed on an existing index.
What exactly happens in a cross-cluster search if one cluster is down?
The search request will skip the unreachable cluster and return partial results from all other healthy clusters that were part of the request. It logs the failure but does not halt the entire operation.
Should I replace the `_optimize` API calls in my scripts?
Yes, the `_optimize` API is deprecated. You should migrate to using the `_forcemerge` API, which provides the same functionality under its new name.
Can I use SQL to join data between a regular index and a frozen index?
No, the SQL query execution in 6.6 does not support cross-index joins, and this limitation includes joins involving frozen indices. You would need to query them separately.