Coralogix ‘Archive query’ allows you to query your logs directly from your S3 archive using text syntax. This enables you to store more of your log data in lower-cost storage and take advantage of Coralogix’s real-time data scanning capabilities.
For example, setting logs to Medium priority (monitoring) still allows you to view and query them in the LiveTail, receive real-time alerts and anomalies on top of them, parsing rules, Loggregation, create Logs2mMetrics, and now query them without ever indexing the data at 40% of the cost.
In order to use this feature make sure you have set Read/Write permission to your AWS S3 archive bucket (read more about enabling Archive feature here)
If you don’t have such permission you will see the following screen:
Click on the ‘ARCHIVE QUERY’ button and the following dialogue box will open:
In the top section, you will fill the query name and description.
In the following section, you can enter a text search query to match a subset of logs from S3, note, we will not mount anything besides logs matching this query.
Choose the applications, subsystems, severity, and time frame criteria for the query. (Note: The time range limit is up to 24 hours)
After clicking on “ARCHIVE QUERY” button and wait till it is processed you will see your new query and the 3 options: Download TSV, Reindex, Logs preview
If you click on “Logs preview” you will be able to view your logs without never indexing that data.
Clicking on “Download TSV” will show you the following screen with the files you will download
Clicking on “Reindex” will show you the following screen and you’ll be asked to verify your selection as it will affect your daily quota.
After clicking ‘CONFIRM’ you will be taken back to the main Reindexing window. The window will show Reindexing tasks. Each task can be in one of these states:
- Processing – ‘REINDEXING …’ is displayed while the reindex process is underway
- Completed – Ready to view logs
- Failed – The reindexing process failed. You have the option to retry or create a new Reindex.
After processing is finished, clicking on “View Logs” will prompt you to the logs screen to view and analyze your reindexed data.
Reindexed Logs, like any other of your logs, will be automatically deleted after the account’s retention period has passed. They differ from other logs by having their original timestamp (which might be out of the retention period) and by not being processed through rules, alerts, archiving, ML and anomalies, LiveTail, and the enrichment engines. This keeps the operational integrity of the notifications, views, and analysis provided to you by Coralogix.
Eventually, after some time, the Archive query you created will expire so you can no longer view, download, or reindex the data. You can always create another similar “Query Archive”.