Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!

Start Free Trial Request Demo
Back to All Docs

Archive Query Archive Query

Last Updated: Jan. 13, 2023

Coralogix ‘Archive query’ allows you to query your logs directly from your S3 archive using any text, Elasticsearch syntax query, SQL syntax query, or DataPrime syntax query, even if the data was never indexed and without the usage of your daily quota. This enables you to store more of your data in our monitoring and compliance priority levels and take advantage of Coralogix’s real-time analysis and remote storage search capabilities. This means you can use a shorter retention period and still be able to query all your data in less than 1 minute using the familiar ES syntax.

For example, prioritizing logs at the monitoring level still allows you to view and query them in the LiveTail, receive real-time alerts and anomalies on top of them, leverage parsing rules, Loggregation, Logs2Metrics, and query them without ever indexing the data at 40% of the cost.

Archive keeps all Coralogix logs after applying rules. It means that:

  • Low
  • Medium
  • High
  • Logs blocked by block rules but with the checked option “View blocked logs in Livetail and archive to S3”

Only blocked logs are not sent to the archive.

Lets starts with Archive query

  1. Click on  Data flow > Archive Queries


In order to use this feature make sure you have set Read/Write permission to your AWS S3 archive bucket (read more about enabling the Archive feature here)

 2. Click on the ‘ARCHIVE QUERY‘ button and the following dialogue box will open:

In the top section, you will fill the query name and description.

In the Search query section, you can enter a text search query, Elasticsearch, SQL, or DataPrime syntax query to match a subset of logs from S3.Note: we will not mount anything besides logs matching this query. Each query will return up to 1000000 logs in the results.

Select the source from below option ( Please check the option which you have selected while setting up Archive), It use fetch data from it.

  1. CSV
  2. CX-Data

CSV is the legacy archive type. It keeps logs on S3 bucket in the CSV format. 

CX-Data is new, recommended archive type. It keeps logs on S3 bucket in the Parquet format.

Note: You can store logs in the archive in one or in both formats. SQL query syntax is not available on CX-Data.

Query examples:

  1. A query to find logs with the field ClientIP_geoip.continent_name:”Europe” and the field ClientIP_geoip.country_name with values other than: Czechia, United Kingdom or Germany:
    ClientIP_geoip.continent_name:”Europe” NOT (ClientIP_geoip.country_name:”Czechia” OR ClientIP_geoip.country_name:”United Kingdom” OR ClientIP_geoip.country_name:”Germany”)
  2. A query to find logs with words status and get:
    status get
  3. A query to find only logs with HTTP method post:
    “http_method”:”post”

Choose the applications, subsystems, severity, and time frame criteria for the query.

(Note: The time range limit is up to 3 Days)

3. After clicking on “RUN ARCHIVE QUERY” button and wait till it is processed you will see your new query and the 4 options: Download TSV, Reindex, Logs preview, Clone.

Logs Preview:
If you click on “Logs preview” you will be able to view your logs without ever indexing that data.

Download Archive data:
Clicking on “Download TSV” will show you the following screen with the files you will download

Reindex:

Coralogix ‘Reindex’ feature allows you to bring back to your index specific sets of data out of your logs from your S3 archive.

Clicking on “Reindex” will show you the following screen and you’ll be asked to verify your selection as it will affect your daily quota.

After clicking ‘CONFIRM’ you will be taken back to the main Reindexing window. The window will show Reindexing tasks. Each task can be in one of these states:

  • Processing –  ‘REINDEXING …’ is displayed while the reindex process is underway
  • Completed – Ready to view logs
  • Failed – The reindexing process failed. You have the option to retry or create a new Reindex.

After processing is finished, clicking on “View Logs” will prompt you to the logs screen to view and analyze your reindexed data.

Reindexed Logs, like any other of your logs, will be automatically deleted after the account’s retention period has passed. They differ from other logs by having their original timestamp (which might be out of the retention period) and by not being processed through rules, alerts, archiving, ML and anomalies, LiveTail, and the enrichment engines. This keeps the operational integrity of the notifications, views, and analysis provided to you by Coralogix.

Clone:
In case you want to create another Archive query similar to one query you already created you can easily duplicate it by clicking on the Clone button.

If you wish to share an Archive query with another teammate click on your query and on the top click again on the chain-link icon. This will copy to your clipboard the link to that same Archive query.

Eventually, after some time, the Archive query you created will expire so you can no longer view, download, or reindex the data, but you can click on clone and duplicate the same query with the same criteria instead of recreating the query from scratch.

Note: Max results you can fetch is 1000000 while running Archive query, Need to refine your query if its exceeds for better results.

On this page