What is new in Elasticsearch 9.1
Elasticsearch 9.1 focuses on improving data reliability, query flexibility, and vector search performance. The release introduces the failure store for data streams, promotes several important features to general availability, and adds powerful new capabilities in ES|QL while delivering storage and security enhancements.
These updates help teams reduce data loss, build more advanced analytics pipelines, and run efficient semantic search at scale.
Key Highlights in Elasticsearch 9.1
| Feature | Description | Benefit |
|---|---|---|
| Failure Store for Data Streams | Captures documents that fail ingest or mapping validation in a dedicated failure index. | Prevents data loss and provides visibility into problematic documents without rejecting entire batches. |
| FORK Command in ES|QL (Tech Preview) | Allows branching a query into multiple parallel paths with a _fork discriminator column. | Enables complex conditional processing and multi-path analytics in a single query. |
| ES|QL Cross-Cluster Querying (GA) | Run ES|QL queries seamlessly across multiple clusters. | Simplifies unified analytics in multi-cluster environments. |
| Sparse Vector Token Pruning (GA) | Prunes irrelevant tokens during sparse vector queries. | Improves both speed and relevance for sparse vector search. |
| Lucene 10.2.2 Upgrade | Brings memory optimizations for HNSW graph building and several query fixes. | Reduces memory usage during indexing and improves search stability. |
| Synthetic Source Improvements | Enabled by default when synthetic source is active, with native array offset storage. | Significantly faster indexing performance and lower storage overhead. |
Failure Store for Reliable Data Ingestion
The failure store is a major reliability feature in Elasticsearch 9.1. When enabled on a data stream, documents that fail due to ingest pipeline errors or mapping conflicts are automatically stored in a separate failure index instead of causing the request to fail.
You can enable it through index templates or directly on existing data streams. Failed documents include the original source, error details, and timestamp for easy debugging.
To query the failure store, use the ::failures syntax:
POST logs-test-apache::failures/_searchThis change turns potential data loss into actionable insights while keeping your cluster stable.
Powerful New ES|QL Capabilities
ES|QL continues to evolve with several important additions:
- FORK command (Tech Preview): Splits the query stream into multiple branches for parallel conditional processing. Results include a
_forkcolumn to identify the source branch. - Cross-cluster querying (GA): Run ES|QL queries across remote clusters with full support for partial results.
- New functions and operators including
MATCH_PHRASE,LIKEwith lists,COMPLETION, and better support for parameters inLIMIT. - Performance optimizations such as constant keyword handling, join improvements, and aggressive memory management for shard contexts.
These enhancements make ES|QL more expressive and suitable for complex analytics workloads.
Vector Search Enhancements
Vector capabilities receive solid improvements in 9.1:
- Sparse vector token pruning is now generally available, delivering better performance and relevance.
- Added support for bit vectors in semantic text fields.
- New off-heap statistics for dense_vector fields in node and index stats APIs.
- Option to explicitly include or exclude vectors from
_sourceretrieval. rescore_vectoris now generally available with additional quantization options.- Panama-accelerated scalar quantization and improved HNSW filtered search.
These changes help reduce memory usage and speed up semantic and hybrid search operations.
Storage and Indexing Optimizations
Several storage-related improvements help lower costs and boost performance:
- Synthetic recovery source is enabled by default when synthetic source is active, resulting in faster indexing.
- Native storage of array offsets for many field types (boolean, keyword, numeric, etc.) when using synthetic source.
- Better handling of multi-value fields and reduced storage for certain text subfields.
- Lucene 10.2.2 brings reduced on-heap memory during HNSW graph construction.
Together, these optimizations make Elasticsearch more efficient for high-volume indexing scenarios.
Security and Operational Improvements
Security receives important updates:
- Permanent switch from Java SecurityManager to the new Entitlements system for better protection.
- Support for custom SAML attributes from Identity Providers.
- New Microsoft Graph delegated authorization realm.
- Enhanced JWT support and additional role privileges for Kibana system and entity analytics.
Other changes include safer S3 repository handling through the AWS SDK v2 upgrade and various ingest pipeline fixes.
How These Changes Help in Real-World Use
Observability and logging teams benefit greatly from the failure store, which protects against data loss from malformed events. Analysts gain more power with the FORK command and cross-cluster ES|QL for advanced data processing. Teams running semantic search see better performance and lower memory usage thanks to token pruning and vector optimizations. Infrastructure operators appreciate the storage gains from synthetic source and the more secure runtime model.
Most features are backward compatible or can be adopted gradually, making the upgrade to 9.1 smooth for production clusters.
FAQ
What is the failure store in Elasticsearch 9.1?
It is a dedicated mechanism that stores documents failing ingest or mapping validation in a separate index, preventing data loss and enabling easier troubleshooting.
How does the FORK command work in ES|QL?
The FORK command splits query execution into multiple branches. Each branch processes data independently, and results are merged with a _fork column identifying the origin branch.
Is cross-cluster querying in ES|QL production ready in 9.1?
Yes, ES|QL cross-cluster search is now generally available and supports partial results.
What changed for sparse vector search in 9.1?
Token pruning for sparse vectors moved from tech preview to general availability, improving query speed and relevance.
Does synthetic source improve indexing speed?
Yes. When enabled, synthetic recovery source is now on by default, and native array offset storage further boosts indexing performance.
Should I upgrade to Elasticsearch 9.1?
If you use data streams, ES|QL, or vector search, the reliability, performance, and new query features make 9.1 a worthwhile upgrade. Review the AWS SDK v2 changes for S3 repositories before migrating.