INTEGRATIONS

Fastly logs Integration through HTTPS streaming

fastly logs

Fastly’s Real-Time Log Streaming feature provides the ability to send Fastly logs to any HTTPS endpoint. This will allow you to optimize your Fastly services.

Fastly supports real-time log streaming of data that passes through it. Fastly supports a number of protocols that allow you to stream logs to a variety of locations, including third-party services, for storage and analysis. The following tutorial will cover how to set up your HTTPS endpoint to Coralogix from your Fastly account. For more details, on Log streaming through HTTPS, you can find the official documentation here: https://docs.fastly.com/en/guides/log-streaming-https.

NOTE: The logging endpoint for HTTPS streaming is disabled by default. To enable the endpoint for your account, you will need to contact support@fastly.com to request it.

Setting up Fastly logs Configuration

1 – After logging into your Fastly account, you will see a listing of all configured services. Click on Configure at the top.

coralogix fastly integration, list of all configured services

 

2 – Choose which service you wish to provision. You can switch between existing services or create a new one. Click on “View active configuration” to set a new endpoint.

coralogix fastly integration choose service to provision

 

3 – Click on “Clone” to access editing.

coralogix fastly integration clone button

 

4 – On the left panel, click on “Logging”.

coralogix fastly integration click logging

 

5 – Click on “Create Endpoint”.

coralogix fastly integration create endpoint

 

6 – Scroll down the page and choose HTTPS, and click on the associated “Create Endpoint” button.

coralogix fastly integration https endpoint

 

7 – Next you will configure your endpoint.

coralogix fastly integration endpoint configuration

Here, enter the relevant information for each of the requested fields.

  • Name: Name your endpoint
  • Log format (Recommended): (enter the following data)
    {
      "timestamp":%{time.start.msec}V, 
      "applicationName":"fastly", 
      "subsystemName":"coralogix.com", 
      "severity": 3, 
      "json": { 
        "time": { 
            "start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
            "end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
            "elapsed":%D
        },
        "cdn_server": { 
            "ip_ipaddr":"%A",
            "code":"%{server.datacenter}V",
            "hostname":"%{server.hostname}V",
            "region_code":"%{server.region}V",
            "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
            "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
            "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
            "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
            "h2_stream_id":"%{fastly_info.h2.stream_id}V"
        },
        "client": { 
            "city_name":"%{client.geo.city.utf8}V",
            "country_code":"%{client.geo.country_code}V",
            "country_name":"%{client.geo.country_name}V",
            "continent_code":"%{client.geo.continent_code}V",
            "region":"%{client.geo.region}V",
            "ip_ipaddr":"%h",
            "name":"%{client.as.name}V",
            "number":"%{client.as.number}V",
            "connection_speed":"%{client.geo.conn_speed}V",
            "location_geopoint": { 
                "lat":%{client.geo.latitude}V,
                "lon":%{client.geo.longitude}V
            }
        },
        "response": { 
            "status":%>s,
            "content_type":"%{Content-Type}o",
            "age":"%{Age}o",
            "cache_control":"%{Cache-Control}o",
            "expires":"%{Expires}o",
            "last_modified":"%{Last-Modified}o",
            "tsv":"%{TSV}o",
            "header_size":%{resp.header_bytes_written}V,
            "body_size":%B
        },
        "request": { 
            "host":"%{req.http.host}V",
            "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
            "backend":"%{req.backend}V",
            "service_id":"%{req.service_id}V",
            "url":"%{cstr_escape(req.url)}V",
            "url_ext":"%{req.url.ext}V",
            "header_size":%{req.header_bytes_read}V,
            "body_size":%{req.body_bytes_read}V,
            "method":"%m",
            "protocol":"%H",
            "referer":"%{Referer}i",
            "user_agent":"%{User-Agent}i",
            "accept_content":"%{Accept}i",
            "accept_language":"%{Accept-Language}i",
            "accept_encoding":"%{Accept-Encoding}i",
            "accept_charset":"%{Accept-Charset}i",
            "connection":"%{Connection}i",
            "dnt":"%{DNT}i",
            "forwarded":"%{Forwarded}i",
            "via":"%{Via}i",
            "cache_control":"%{Cache-Control}i",
            "x_requested_with":"%{X-Requested-With}i",
            "x_att_device_id":"%{X-ATT-Device-Id}i",
            "x_forwarded_for":"%{X-Forwarded-For}i"
        },
        "socket": { 
            "cwnd":%{client.socket.cwnd}V,
            "pace":%{client.socket.pace}V,
            "nexthop":"%{client.socket.nexthop}V",
            "tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
            "tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
            "tcpi_rtt":%{client.socket.tcpi_rtt}V,
            "tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
            "tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
            "tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
            "tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
            "tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
            "tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
            "ploss":%{client.socket.ploss}V
        }
      }
    }

    The first five fields are mandatory:

    • Timestamp – The format should not change.
    • applicationName – Enter the name of the application.
    • subsystemName – Enter the name of the subsystem. 
    • Severity – Apply the severity to all logs, using the following choices: 1-debug, 2-verbose, 3-info, 4-warning, 5-error, 6-critical. This can be changed later using an extract rule, as described below.
    • JSON (object) – Fields can be added or removed. Static fields can be added.  Nested JSON formats are supported including any fields described in the Fastly VCL reference: https://docs.fastly.com/vcl/variables/.

    The response.status field sends the request status. This is a recommended field. Then, using the Coralogix parsing rules, you may set a JSON extract rule to extract the status code from the request into Coralogix severity. Define the severity to automatically determine the importance of the type of log. Note: in Coralogix we automatically map HTTP status codes into a severity tag as appropriate. For example, status code 200 will set the Coralogix severity as “INFO”, status code 4xx will set Coralogix severity as “ERROR”, etc.

Coralogix rule configuration

This is how the rule will look in Coralogix.

coralogix fastly integration json extract rule

8 – Under advanced options, enter the following data.

coralogix fastly integration advanced options

 

Set the options as:

Content type – application/json

Custom header name – private_key

Custom header value – YOUR CORALOGIX PRIVATE KEY (your unique private key, which can be found under settings → send your logs, at the top left of the screen)

Method – Choose “POST” (default)

JSON log entry format – Choose “Array of JSON”

Select a log line format – Choose “Blank” (default)

Placement – Choose “Format Version Default” (default)

Leave the rest of the options empty and click on “Create” (or “Update” if you are updating an endpoint). To finish setting up your service, click on the green “Activate” button on the right to activate the new/updated endpoint.

9 – After activation, under “Domains” on the left panel,  click on “Test domain” to verify your configuration. A test log should appear in your Coralogix account.

coralogix fastly integration test domain

 

10 – This is how it should appear in Coralogix.

coralogix fastly integration log example in coralogix

If you see your test log in Coralogix, it means that you have successfully configured the integration.

To get all the Coralogix dashboards and alerts, contact our support on our website/in-app chat. We reply in under 2 minutes!

Still, have questions? check our website and use the in-app chat for quick help from a Coralogix professional.

Start solving your production issues faster

Let's talk about how Coralogix can help you better understand your logs

Managed, Scaled and Compliant ELK Stack

No credit card required

Get a personalized demo

Jump on a call with one of our experts and get a live personalized demonstration