Our next-gen architecture is built to help you make sense of your ever-growing data. Watch a 4-min demo video!

Back to All Docs

Archive Query Archive Query

Last Updated: Jun. 05, 2024

Archive Query enables you to directly query your logs and spans from your S3 archive using any text or a wide range of syntax queries. Query data irrespective of priority, daily quota, or its time frame.

Overview

Archive Query enables you to directly query your logs and spans from your S3 archive.

Use this feature to:

  • Take advantage of various querying options. This includes text-based queries or specific query languages such as Lucene or DataPrime.
  • Enjoy multiple data sources. Using our innovative DataPrime syntax language, query both logs and spans.
  • Query data irrespective of priority and daily quota. Only blocked logs are not sent to the archive.
  • Query data with unlimited time frames. There are no restrictions on how far back in time your data can go.
  • Save on costs while maintaining interactive query times. Store more of your data in our Monitoring and Compliance priority levels. By prioritizing logs at the Monitoring priority level, for example, you can view and query them in LiveTail, receive real-time alerts, utilize parsing rules, Loggregation and Events2Metrics, and query them without indexing your data. All of this is available at 40% of the cost.

Prerequisites

Archive Query Setup

STEP 1. In your Coralogix toolbar, navigate to Data Flow > Archive Queries.

Archive Query Coralogix

STEP 2. Click ARCHIVE QUERY.

Archive Query Coralogix

STEP 3. Define a New Archive Query.

Archive Query Coralogix
  • Enter a Query Name and Description.
  • Search Query. Enter a text search query, Lucene, or DataPrime syntax query to match a subset of logs from your S3 bucket.

Notes:

  • Only the data matching the query will be presented.
  • Using our innovative DataPrime syntax language, query both logs and spans.

Query Examples


1. A query to find logs with the field ClientIP_geoip.continent_name:”Europe” and the field ClientIP_geoip.country_name with values other than: Czechia, United Kingdom or Germany:ClientIP_geoip.continent_name:”Europe” NOT (ClientIP_geoip.country_name:”Czechia” OR ClientIP_geoip.country_name:”United Kingdom” OR ClientIP_geoip.country_name:”Germany”)

2. A query to find logs with words status and get:status get

3. A query to find only logs with HTTP method post:“http_method”:”post” </aside>

  • Source. Select CX-Data.
  • Select the Timeframe, Applications, Subsystems, and Severity for the query.

STEP 4. Click RUN ARCHIVE QUERY. Once you have set up and run your query, a test will be run to validate your setup.

View Archive Query Results

View your query results in one of three formats: Logs Preview, Download TSV, or Clone.

Archive Query Coralogix

Logs Preview

This option allows you to view your logs without ever indexing your data.

Archive Query Coralogix

Download TSV

Download a TSV file to view query results.

Archive Query Coralogix

Clone

Duplicate your current query by clicking on the Clone button.  In the new duplicated query, click RUN ARCHIVE QUERY.

Archive Query Coralogix

If you wish to share an archive query with another teammate, click on the chain-link icon in the query of choice. This will copy to your clipboard the link to that same archive query.

Archive Query Coralogix

After some time, the archive query you created will expire so you can no longer view or download the data. Click Clone and duplicate the same query with the same criteria instead of recreating the query from scratch.

Limitations

The limitations placed on queries are described below.

LimitationDescription
Bytes processedUp to 30% of daily ingested bytes
Parquet files scannedUp to 500k files
Clone resultsUp to results 1M results while running Archive Query
Time outUp to 5 min of query execution
Query time frameMaximum query time frame is up to 192 hours (8 days)

Warnings

Once a limit is reached, a warning message is displayed. Refine your query results to avoid reaching a limit.

Refine Your Query Results

Refine your query results using any of the following methods:

  • Apply more selective filters to your queries (for example, on application or subsystem).
  • If using the Dataprime extract operator and subsequently filtering its results, create a parsing rule and filter on the parsed field instead.
  • Avoid regular expressions or wildcards in filters.
  • In Dataprime, switch from using the contains operator on strings to the free text search operator (~).

Additional Resources

DocumentationArchive Query from the Explore Screen

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at support@coralogix.com

On this page