Cloudflare

Cloudflare Enterprise customers have access to Logpush service, which allows you to forward logs to cloud service providers like AWS. This tutorial demonstrates how to send your logs to Coralogix.

Send Logs Directly

Prerequisites

• Coralogix Send-Your-Data API key
• Cloudflare ZoneID / AccountID – Find the relevant id in the cloudflare dashboard under ‘Websites’ -> ‘<Your-site>’ -> ‘API (scroll down)’.

Configuration

To start sending data directly to Coralogix, select the type of logs (data-sets) and fields that will be sent here.

To create the logpush job, call the API using the terminal. Input your Coralogix domain to create your endpoint URL: https://ingress.<cx_domain>/cloudflare/v1/logs.

curl -s https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/jobs -X POST \
-H "Content-Type:application/json" \
-d '{
  "name": "logpush-to-coralogix",
  "logpull_options": "fields=BotScoreCloudflare,BotScoreSrc,BotTags,CacheCacheStatus,CacheResponseBytes,CacheResponseStatus,CacheTieredFill,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientMTLSAuthCertFingerprint,ClientMTLSAuthStatus,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestPath,ClientRequestProtocol,ClientRequestReferer,ClientRequestScheme,ClientRequestSource,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,ClientTCPRTTMs,ClientXRequestedWith,EdgeCFConnectingO2O,EdgeColoCode,EdgeColoID,EdgeEndTimestamp,EdgePathingOp,EdgePathingSrc,EdgePathingStatus,EdgeRateLimitAction,EdgeRateLimitID,EdgeRequestHost,EdgeResponseBodyBytes,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseContentType,EdgeResponseStatus,EdgeServerIP,EdgeTimeToFirstByteMs,FirewallMatchesActions,FirewallMatchesRuleIDs,FirewallMatchesSources,JA3Hash,OriginDNSResponseTimeMs,OriginIP,OriginRequestHeaderSendDurationMs,OriginResponseBytes,OriginResponseDurationMs,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseHeaderReceiveDurationMs,OriginResponseStatus,OriginResponseTime,OriginSSLProtocol,OriginTCPHandshakeDurationMs,OriginTLSHandshakeDurationMs,ParentRayID,RayID,RequestHeaders,ResponseHeaders,SecurityLevel,SmartRouteColoID,UpperTierColoID,WAFAction,WAFFlags,WAFMatchedVar,WAFProfile,WAFRuleID,WAFRuleMessage,WorkerCPUTime,WorkerStatus,WorkerSubrequest,WorkerSubrequestCount,ZoneID,ZoneName&timestamps=unixnano",
  "destination_conf": "<https://ingress.<coralogix_domain>/cloudflare/v1/logs>?header_Authorization=Bearer%20<Send_your_data_key>&header_timestamp-format=UnixNano&header_dataset=HTTPRequests",
  "max_upload_bytes": 5000000,
  "max_upload_records": 1000,
  "dataset": "http_requests",
  "enabled": true,
  "frequency": "low"
}' \
-H "X-Auth-Email: <Your_Auth_Email>" \
-H "X-Auth-Key: <Your_API_Key>"

Notes:

• Replace <zone_id> with your site zone id, <Send_your_data_key> with your coralogix Send-Your-Data API key and ‘X-Auth-Email’ ‘X-Auth-Key’ with your cloudflare credentials.
• To change the dataset sent change the ‘dataset’ field and `header_dataset` inside the ‘destination_conf’ field using the table below.
• To change the fields sent change the ‘logpull_options’ field, each dataset has different fields.
• Each dataset has a different ‘timestamp’ key, providing it in the ‘logpull_options’ field is required, also using the unixnano format for the timestamp is also required. Find out more about log fields here.
• To configure Account-scoped datasets use ‘https://api.cloudflare.com/client/v4/accounts/<Account_ID>/logpush/jobs’.
• By default, the integration will set application_name as Cloudflare, and subsystem_name as the data set name. To overwrite these parameters, add the following:
  • header_CX-Application-Name – application name override
  • header_CX-Subsystem-Name – subsystem name override

After creating the logpush, view it in the terminal.

curl -s https://api.cloudflare.com/client/v4/zones/<Zone_ID>/logpush/jobs -X GET \
-H "X-Auth-Email: <Your_Auth_Email>" \
-H "X-Auth-Key: <Your_Auth-Key>"

Or in the dashboard itself under ‘Websites’ -> ‘<Your-site>’ -> ‘Analytics’ -> ‘Logs’.

Send Logs via S3 Bucket

Prerequisites

AWS S3 bucket – Follow the tutorial to send logs from the S3 bucket to Coralogix: https://coralogix.com/integrations/data-collection-s3/

Configuration

To enable the Clouflare Logpush service:

1. Log in to the Cloudflare dashboard
2. Ensure the Enterprise domain you want to use with Logpush is selected
3. Select the Analytics app in the top menu
4. Select the Logs section in the secondary menu
   
5. Select Connect a service. A modal window opens where you will need to complete several steps.
   
6. Under Select service, pick your Amazon S3 and click Next.
   
7. Enter or select the following:
   • Bucket path
   • Daily subfolders
   • Bucket region
     
8. Copy the policy from the field below “Grant Cloudflare access to upload files to your bucket”
9. In new window go to S3 > your bucket name > Permissions > Bucket Policy
10. Click on the Edit button at the Bucket Policy.
11. Paste the copied policy and save changes.
12. In the Cloudflare dashboard, click Validate access.
13. Follow the on-screen instructions to enter the Ownership token (included in a file Cloudflare sends to your Amazon S3 bucket).
    
    Open a file with token on your Amazon S3 bucket and paste the token into Ownership token field:
    
14. Click the “Prove ownership” button.
15. Next in the Customize log screen, select the data set in which you’re interested (currently HTTP requests or Spectrum events).
    
    You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in Logs > Logpush (select the wrench icon).
    
16. To finish enabling Logpush, click Save and Start Pushing

Once connected, Cloudflare lists the provider you just configured under Logs > Logpush. This is where you can make changes or remove the provider.

If all steps were executed properly, you should see files in your S3 bucket and also in Coralogix.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].