Cloudflare

Last Updated: Aug. 07, 2022

Cloudflare Enterprise customers have access to Logpush service which allows you to forward logs to cloud service providers like AWS. In this tutorial, you will find all steps to send logs to Coralogix.

Send logs directly

Prerequisites

A Private Key – You can find your key in your Coralogix account under ‘Data Flow’ -> ‘API Keys’ -> ‘Send your logs’.
Cloudflare ZoneID / AccountID – Find the relevant id in the cloudflare dashboard under ‘Websites’ -> ‘<Your-site>’ -> ‘API (scroll down)’.

Configuration

To start sending data directly to coralogix we first need to pick the type of logs (data-sets) and fields that will be sent, Cloudflare Data sets.

To create the logpush job we need to call the API using the terminal

curl -s https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/jobs -X POST \
-H "Content-Type:application/json" \
-d '{
  "name": "logpush-to-coralogix",
  "logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=unixnano",
  "destination_conf": "https://cdn-ingress.coralogix.com/api/v1/cloudflare/logs?header_Authorization=Bearer%20<Send_your_data_key>&header_timestamp-format=UnixNano&header_dataset=HTTPRequests",
  "max_upload_bytes": 5000000,
  "max_upload_records": 1000,
  "dataset": "http_requests",
  "enabled": true,
  "frequency": "low"
}' \
-H "X-Auth-Email: <Your_Auth_Email>" \
-H "X-Auth-Key: <Your_API_Key>"

Notes:

Replace <zone_id> with your site zone id, <Send_your_data_key> with your coralogix private key and ‘X-Auth-Email’ ‘X-Auth-Key’ with your cloudflare credentials.
To change the dataset sent change the ‘dataset’ field and `header_dataset` inside the ‘destination_conf’ field using the table below.
To change the fields sent change the ‘logpull_options’ field, each dataset has different fields.
Each dataset has a different ‘timestamp’ key, providing it in the ‘logpull_options’ field is required, also using the unixnano format for the timestamp is also required.
To configure Account-scoped datasets use ‘https://api.cloudflare.com/client/v4/accounts/<Account_ID>/logpush/jobs’ .

Dataset name	Header name	‘Timestamp’ key	Scope
dns_logs	DNSLogs	Timestamp	Zone
firewall_events	FirewallEvents	Datetime	Zone
http_requests	HTTPRequests	EdgeStartTimestamp	Zone
nel_reports	NELReports	Timestamp	Zone
spectrum_events	SpectrumEvents	Timestamp	Zone
audit_logs	AuditLogs	When	Account
gateway_dns	GatewayDNS	Datetime	Account
gateway_http	GatewayHTTP	Datetime	Account
gateway_network	GatewayNetwork	Datetime	Account
network_analytics_logs	NetworkAnalyticsLogs	Datetime	Account

After creating the logpush you can view it in the terminal

curl -s https://api.cloudflare.com/client/v4/zones/<Zone_ID>/logpush/jobs -X GET \
-H "X-Auth-Email: <Your_Auth_Email>" \
-H "X-Auth-Key: <Your_Auth-Key>"

Or in the dashboard itself under ‘Websites’ -> ‘<Your-site>’ -> ‘Analytics’ -> ‘Logs’.

Send logs via S3 bucket

Prerequisites

AWS S3 bucket – Follow the tutorial to send logs from the S3 bucket to Coralogix: https://coralogix.com/integrations/data-collection-s3/

Configuration

To enable the Clouflare Logpush service:

Log in to the Cloudflare dashboard
Ensure the Enterprise domain you want to use with Logpush is selected
Select the Analytics app in the top menu
Select the Logs section in the secondary menu
Select Connect a service. A modal window opens where you will need to complete several steps.
Under Select service, pick your Amazon S3 and click Next.
Enter or select the following:
- Bucket path
- Daily subfolders
- Bucket region
Copy the policy from the field below “Grant Cloudflare access to upload files to your bucket”
In new window go to S3 > your bucket name > Permissions > Bucket Policy
Click on the Edit button at the Bucket Policy.
Paste the copied policy and save changes.
In the Cloudflare dashboard, click Validate access.
Follow the on-screen instructions to enter the Ownership token (included in a file Cloudflare sends to your Amazon S3 bucket).

Open a file with token on your Amazon S3 bucket and paste the token into Ownership token field:
Click the “Prove ownership” button.
Next in the Customize log screen, select the data set in which you’re interested (currently HTTP requests or Spectrum events).

You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in Logs > Logpush (select the wrench icon).
To finish enabling Logpush, click Save and Start Pushing