Ship OpenTelemetry Data to Coralogix via Reverse Proxy (Caddy 2)
George Pickers & Maya Keren-Abraham
April 4, 2023
It is commonplace for organizations to restrict their IT systems from having direct or unsolicited access to external networks or the Internet, with network proxies serving as gatekeepers between an organization’s internal infrastructure and any external network. Network proxies can provide security and infrastructure admins the ability to specify specific points of data egress from their internal networks, often referred to as an egress controller.
This tutorial demonstrates how to leverage open-source telemetry shippers in conjunction with an open-source network proxy to create a hub-and-spoke architecture that sends your data to Coralogix with a single specified point of data egress.
STEP 3. Define any global options that apply to the entire Caddy server, including which HTTP protocols to support. The h2c scheme allows us to translate gRPC requests into HTTPS onward requests to Coralogix.
protocols h1 h2 h2c
STEP 4. Define the parameters of the reverse proxy, including the address and port for the inbound traffic coming from our OpenTelemetry Collectors. This allows us to successfully forward inbound gRPC traffic from our OpenTelemetry Collectors to Coralogix ingress via HTTPS.
STEP 5. Instruct Caddy 2 to publish Prometheus-format metrics of the Caddy 2 server itself. This step allows us to use our OpenTelemetry Collectors to scrape these metrics and actively monitor our egress controller without deploying any additional components into our telemetry stack.
STEP 6. To apply the configuration for the first time and start the Caddy server, use the following command:
STEP 7. To make any changes to the Caddyfile, reapply the configuration with the following command:
STEP 8. To view the logs generated by Caddy 2 in stdout, use the following command:
sudo journalctl -u caddy -f
Now that have implemented our Caddy 2 server, update the configuration of our OpenTelemetry Daemonset to send the gRPC traffic to the reverse proxy listening address.
Use this example values.yaml file with Helm to apply the new configuration to our OpenTelemetry Collectors.
As this is a tutorial environment, we have added tls: insecure_skip_verify: true configurations to each of the endpoints (traces, metrics, logs) for the Coralogix Exporter.
The setting insecure_skip_verify: true allows us to send the data using unencrypted gRPC (without TLS verification) to our Caddy 2 egress controller. Caddy 2 then handles the TLS handshake with Coralogix ingress over HTTPS.
Important note folks! This is for a non-production environment. If you have a valid SSL/TLS architecture available, we recommend you secure the traffic between the OpenTelemetry Collectors and Caddy 2 using TLS.
Here we add a configuration in our OpenTelemetry Collector configuration that leverages the Prometheus receiver to scrape the metrics published by Caddy 2. All we need to do here is change <caddy_proxy_address> to the address of our Caddy 2 server.