Fastly Logs via HTTPS Streaming

Fastly’s Real-Time Log Streaming feature provides the ability to send Fastly logs to any HTTPS endpoint. This will allow you to optimize your Fastly services.

Fastly supports real-time log streaming of data that passes through it. Fastly supports a number of protocols that allow you to stream logs to a variety of locations, including third-party services, for storage and analysis. The following tutorial will cover how to set up your HTTPS endpoint to Coralogix from your Fastly account. For more details, on Log streaming through HTTPS, you can find the official documentation here: https://docs.fastly.com/en/guides/log-streaming-https.

NOTE: The logging endpoint for HTTPS streaming is disabled by default. To enable the endpoint for your account, you will need to contact [email protected] to request it.

Configuration

https://docs.fastly.com/en/guides/log-streaming-coralogix

After logging into your Fastly account, you will see a listing of all configured services. Click on Configure at the top.

Choose which service you wish to provision. You can switch between existing services or create a new one. Click on “View active configuration” to set a new endpoint.

Click on “Clone” to access editing.

On the left panel, click on “Logging”.

Click on “Create Endpoint”.

Scroll down the page and choose HTTPS, and click on the associated “Create Endpoint” button.

Next you will configure your endpoint.

Here, enter the relevant information for each of the requested fields.

• Name: Name your endpoint
• Log format (Recommended): (enter the following data)

{
"timestamp":%{time.start.msec}V,
"applicationName":"fastly",
"subsystemName":"coralogix.com",
"severity": 3,
"json": {
"time": {
"start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
"end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
"elapsed":%D
},
"cdn_server": {
"ip_ipaddr":"%A",
"code":"%{server.datacenter}V",
"hostname":"%{server.hostname}V",
"region_code":"%{server.region}V",
"is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
"cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
"is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
"is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
"h2_stream_id":"%{fastly_info.h2.stream_id}V"
},
"client": {
"city_name":"%{client.geo.city.utf8}V",
"country_code":"%{client.geo.country_code}V",
"country_name":"%{client.geo.country_name}V",
"continent_code":"%{client.geo.continent_code}V",
"region":"%{client.geo.region}V",
"ip_ipaddr":"%h",
"name":"%{client.as.name}V",
"number":"%{client.as.number}V",
"connection_speed":"%{client.geo.conn_speed}V",
"location_geopoint": {
"lat":%{client.geo.latitude}V,
"lon":%{client.geo.longitude}V
}
},
"response": {
"status":%>s,
"content_type":"%{Content-Type}o",
"age":"%{Age}o",
"cache_control":"%{Cache-Control}o",
"expires":"%{Expires}o",
"last_modified":"%{Last-Modified}o",
"tsv":"%{TSV}o",
"header_size":%{resp.header_bytes_written}V,
"body_size":%B
},
"request": {
"host":"%{req.http.host}V",
"is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
"backend":"%{req.backend}V",
"service_id":"%{req.service_id}V",
"url":"%{cstr_escape(req.url)}V",
"url_ext":"%{req.url.ext}V",
"header_size":%{req.header_bytes_read}V,
"body_size":%{req.body_bytes_read}V,
"method":"%m",
"protocol":"%H",
"referer":"%{Referer}i",
"user_agent":"%{User-Agent}i",
"accept_content":"%{Accept}i",
"accept_language":"%{Accept-Language}i",
"accept_encoding":"%{Accept-Encoding}i",
"accept_charset":"%{Accept-Charset}i",
"connection":"%{Connection}i",
"dnt":"%{DNT}i",
"forwarded":"%{Forwarded}i",
"via":"%{Via}i",
"cache_control":"%{Cache-Control}i",
"x_requested_with":"%{X-Requested-With}i",
"x_att_device_id":"%{X-ATT-Device-Id}i",
"x_forwarded_for":"%{X-Forwarded-For}i"
},
"socket": {
"cwnd":%{client.socket.cwnd}V,
"pace":%{client.socket.pace}V,
"nexthop":"%{client.socket.nexthop}V",
"tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
"tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
"tcpi_rtt":%{client.socket.tcpi_rtt}V,
"tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
"tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
"tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
"tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
"tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
"tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
"ploss":%{client.socket.ploss}V
}
}
}

The first five fields are mandatory:

• Timestamp – The format should not change.
• applicationName – Enter the name of the application.
• subsystemName – Enter the name of the subsystem. 
• Severity – Apply the severity to all logs, using the following choices: 1-debug, 2-verbose, 3-info, 4-warning, 5-error, 6-critical. This can be changed later using an extract rule, as described below.
• JSON (object) – Fields can be added or removed. Static fields can be added.  Nested JSON formats are supported including any fields described in the Fastly VCL reference: https://docs.fastly.com/vcl/variables/.

The response.status field sends the request status. This is a recommended field. Then, using the Coralogix parsing rules, you may set a JSON extract rule to extract the status code from the request into Coralogix severity. Define the severity to automatically determine the importance of the type of log. Note: in Coralogix we automatically map HTTP status codes into a severity tag as appropriate. For example, status code 200 will set the Coralogix severity as “INFO”, status code 4xx will set Coralogix severity as “ERROR”, etc.

Coralogix rule configuration

This is how the rule will look in Coralogix.

• URL:  Select the logs endpoint associated with your Coralogix domain.
  Based on the ending of your Team’s URL, please choose the API Cluster Endpoint that best matches your environment:

• Maximum logs: Leave as default
• Maximum bytes: 2000000

Under advanced options, enter the following data.

Set the options as:

Content type – application/json

Custom header name – private_key

Custom header value – Your CORALOGIX PRIVATE KEY

Method – Choose “POST” (default)

JSON log entry format – Choose “Array of JSON”

Select a log line format – Choose “Blank” (default)

Placement – Choose “Format Version Default” (default)

Leave the rest of the options empty and click on “Create” (or “Update” if you are updating an endpoint). To finish setting up your service, click on the green “Activate” button on the right to activate the new/updated endpoint.

After activation, under “Domains” on the left panel,  click on “Test domain” to verify your configuration. A test log should appear in your Coralogix account.

This is how it should appear in Coralogix.

If you see your test log in Coralogix, it means that you have successfully configured the integration.

To get all the Coralogix dashboards and alerts, contact our support on our website/in-app chat. We reply in under 2 minutes!

Still, have questions? Check our website and use the in-app chat for quick help from a Coralogix professional.