The latest Elasticsearch release version was made available on September 24, 2020, and contains several bug fixes and new features from the previous minor version released this past August. This article highlights some of the crucial bug fixes and enhancements made, discusses issues common to upgrade to this new minor version, and introduces some of the new features released with 7.9 and its subsequent patches. A complete list of release notes can be found on the elastic website.
EQL search is an experimental feature introduced in ELK version 7.9 that lets users match sequences of events across time and user-defined categories. It can be used for many common needs such as log analytics and time-series data processing but was implemented to fill a need in threat detection. Early articles about its use in Elasticsearch show how EQL can be used to help stop the adversarial activity.
When using EQL user-defined timestamp and event categories are used to refine queries to look for more complex data sequences. You can also use a timespan to define how far apart these events can be instead of requiring them to be sequential. This will check for two events that occurred within some time period, regardless of events in between. You can also still use filters with EQL, so sequences only contain events you want to include in the sequence.
Since the EQL was added to Elasticsearch as an experimental feature, the functionality can be changed or removed completely in future releases. Further documentation on how to implement EQL can be found here.
Workplace search was made generally available in ELK version 7.7. This tool allows users to connect data from multiple workplace tools (such as Jira, Salesforce, SharePoint, and Google Drive) into a single searchable format.
ELK version 7.9 brings many of the features of Workplace Search into the free tier, though some additional features such as searching for private sources like email are limited to the platinum subscription model. More information on Elastic Workplace Search on the Elastic website.
This issue is seen when upgrading from an earlier version to ELK version 7.9.0. The machine learning annotations index and the machine learning config index will have incorrect mappings. The error results in the machine learning UI not displaying correctly and machine learning jobs not being created or updated appropriately.
This issue is avoidable if you manually update the mapping on the older ELK version you are already using before updating the Elasticsearch release to 7.9.0, or if you update directly to ELK version 7.9.1 or 7.9.2 (skipping 7.9.0). If the mappings have already been corrupted due to the upgrade, you must reindex them to recover. Updating to a newer ELK version after corruption will not fix this issue.
ELK version 7.9.0 provides enhancements and new features in pipeline aggregation capability. New capabilities with pipeline aggregations include adding the ability to calculate moving percentiles, normalize aggregations, and calculate inference aggregations.
The field capabilities API, or _field_ caps API, which was introduced experimentally in ELK version 5.x, is used to get the capability of index fields using the mapping of multiple indices. As of Elasticsearch release 7.9.0, an index filter is available to use so results are limited to fields in certain indices. Effectively, rather than using the API to return all index mappings, the API can eliminate fields located in unwanted indices that may have the same mapping. More information on this new feature can be found in the Github issue.
The _field_caps API uses types to find if there are conflicts across identically named fields across indices. The types used in the API are refined so that users may detect conflicts between different number types for example. However, constant_keyword was removed from the type list as it was deemed equal to the keyword. The latter is the family type and should be used for description.
Dangling indices exist on the disk but do not exist in the cluster state. These can be formed in several circumstances.
Dangling indices are imported automatically when possible with some unintended effects like deleted indices reappearing when a node joins a cluster. While there are some cases where this import is necessary to recover lost data, in Elasticsearch release 7.9.0 the automatic import is deprecated, and disabled by default, and will be removed completely in ELK version 8.0. Support for user management of dangling indices is maintained in the present and future ELK versions to ensure the recovery can still be accommodated when necessary.
A security flaw has been present in all ELK versions since 6.8.12 with a fix present as of ELK version 7.9.0. An update to this version is required to fix the issue. The security hole is present when running a scrolling search with field-level security. If a user runs the same query that was recently run by a different, more privileged user then the search may show fields that should be hidden to the more constrained user. An attacker may use this to gain access to otherwise restricted fields within an index.
Global ordinals have been present in Elasticsearch since ELK version 2.0 and make aggregations more efficient and less time-consuming by abstracting string values for incremental numbering. A memory leak was found in the global ordinals or other queries that create a cache entry for doc_values are used with low-level cancellation enabled. The search memory leak was fixed in ELK version 7.9.2. Details of the bug and fix can be found on Github.
The Elasticsearch ELK version 7.9.0 is based on Lucene 8.6.0. This version of Lucene introduced a memory leak that would slowly become evident when a single document is updated repeatedly using a forced refresh. A new version of Lucene was released (8.6.2) and Elasticsearch’s ELK version 7.9.1. This may have appeared as a temporary bug for some users and should now be resolved.
In recent years, microservices have emerged as a popular architectural pattern. Although these self-contained services offer greater flexibility, scalability, and maintainability compared to monolithic applications, they…
Distributed microservices and cloud computing have been game changers for developers and enterprises. These services have helped enterprises develop complex systems easily and deploy apps faster….
Gaming apps are complex systems. They combine multi-function systems, like the game engine, to other resources such as server containers, proxies and CDNs in order to…
