Archive Query

Coralogix ‘Archive query’ allows you to query your logs directly from your S3 archive using any text or Elasticsearch syntax query, even if the data was never indexed and without the usage of your daily quota. This enables you to store more of your data in our monitoring and compliance priority levels and take advantage of Coralogix’s real-time analysis and remote storage search capabilities. This means you can use a shorter retention period and still be able to query all your data in less than 1 minute using the familiar ES syntax.

For example, prioritizing logs at the monitoring level still allows you to view and query them in the LiveTail, receive real-time alerts and anomalies on top of them, leverage parsing rules, Loggregation, Logs2Metrics, and query them without ever indexing the data at 40% of the cost.

In order to use this feature make sure you have set Read/Write permission to your AWS S3 archive bucket (read more about enabling the Archive feature here)

If you don’t have such permission you will see the following screen:

Click on the ‘ARCHIVE QUERY’ button and the following dialogue box will open:

In the top section, you will fill the query name and description.

In the Search query section, you can enter a text search query or Elasticsearch syntax query to match a subset of logs from S3.
Note: we will not mount anything besides logs matching this query.
Query examples:

  1. A query to find logs with the field ClientIP_geoip.continent_name:”Europe” and the field ClientIP_geoip.country_name with values other than: Czechia, United Kingdom or Germany:
    ClientIP_geoip.continent_name:”Europe” NOT (ClientIP_geoip.country_name:”Czechia” OR ClientIP_geoip.country_name:”United Kingdom” OR ClientIP_geoip.country_name:”Germany”)
  2. A query to find logs with words status and get:
    status get
  3. A query to find only logs with HTTP method post:
    “http_method”:”post”

Choose the applications, subsystems, severity, and time frame criteria for the query.

(Note: The time range limit is up to 24 hours)

After clicking on “ARCHIVE QUERY” button and wait till it is processed you will see your new query and the 3 options: Download TSV, Reindex, Logs preview

If you click on “Logs preview” you will be able to view your logs without ever indexing that data.

Clicking on “Download TSV” will show you the following screen with the files you will download

Clicking on “Reindex” will show you the following screen and you’ll be asked to verify your selection as it will affect your daily quota.

After clicking ‘CONFIRM’ you will be taken back to the main Reindexing window. The window will show Reindexing tasks. Each task can be in one of these states:

  • Processing –  ‘REINDEXING …’ is displayed while the reindex process is underway
  • Completed – Ready to view logs
  • Failed – The reindexing process failed. You have the option to retry or create a new Reindex.

After processing is finished, clicking on “View Logs” will prompt you to the logs screen to view and analyze your reindexed data.

Reindexed Logs, like any other of your logs, will be automatically deleted after the account’s retention period has passed. They differ from other logs by having their original timestamp (which might be out of the retention period) and by not being processed through rules, alerts, archiving, ML and anomalies, LiveTail, and the enrichment engines. This keeps the operational integrity of the notifications, views, and analysis provided to you by Coralogix.

In case you want to create another Archive query similar to one query you already created you can easily duplicate it by clicking on the Clone button.

If you wish to share an Archive query with another teammate click on your query and on the top click again on the chain-link icon. This will copy to your clipboard the link to that same Archive query.

Eventually, after some time, the Archive query you created will expire so you can no longer view, download, or reindex the data. You can always create another similar “Query Archive”.