At Coralogix we are committed to continuously improving the user experience. With this goal in mind we now provide you the ability to query Archived Logs directly from the Logs screen. This functionality is currently in Beta, and it can be accessed from the top-left portion of the Logs UI (see “Archive (BETA” in the image below).
Now you can query archived data along with the live data streams in the same screen, side by side with all the familiar Logs screen utilities.
The ‘Archive query’ allows you to query your data directly from your S3 archive using any text or Elasticsearch syntax query (In the future, SQL queries with presto functions will also be supported), even if the data was never indexed and without the usage of your daily quota.
This enables you to store more of your data in our monitoring and compliance priority levels (Read more here) and take advantage of Coralogix’s real-time analysis and remote storage search capabilities. This means you can use a shorter retention period and still be able to query all your data in less than 1 minute using the familiar ES syntax.
Now it is also possible to search for the surrounding archived data of indexed logs through the context menu:
Surrounding logs in the Archive Query mode will be available in the future.
In order to use the Archive Query from the Logs screen feature, make sure you have set Read/Write permission to your AWS S3 archive bucket (read more about enabling the Archive feature here)
If you don’t have such permission you will see the following screen:
‘Archive Query’ time range limit is up to 24 hours at a single query.
Running queries on archived data is a bit slower than regular Logs screen queries (recent logs may be delayed up to 5 minutes, as they need to be buffered, collected, zipped, and uploaded, before being made available for searching).
It is possible to use the same query syntax as queries run on the Archive Query page. (For more details, please refer to https://coralogix.com/tutorials/archive-query/).
Some of the functionalities of the Logs screen are not available yet for the Archive queries.
Results shown on the logs grid are capped at 10,000 raw logs: While aggregations are available across all the data, we only pull up to 10,000 raw logs to display in the main logs grid. (Kibana for example limits that number to 500 logs).
The Context Menus are reduced to the supported features (mainly searching logs).
Scanning Limitation: up to 200 GB of compressed archive data can be queried. As our archive compression is anywhere between 5x and 10x, this enables the scanning of 1-2 TB of data (above this amount an error message will appear). In the event that you would need this limit increased, please contact the Coralogix Customer Success Team for assistance.
Reindexing the data fetched with the ‘Archive Query’ is currently not available from the Logs screen. To reindex archived data please create an ‘Archive query’ under the TCO tab –> Archive query to reindex archived data, visit here for more info.
Exporting the data in the logs grid father an archive query is limited to the top 20 pages (100 logs per page) so you can export a top of 2000 logs. In order to export all archive query logs, please create an ‘Archive query’ under the TCO tab –> Archive query.
For any questions, please don’t hesitate to visit us via our Application Chat. We are here to help!