[Live Webinar] Next-Level O11y: Why Every DevOps Team Needs a RUM Strategy Register today!

Optimized Traffic Mirroring Examples – Part 2

  • Yuval Khalifa
  • February 15, 2022
Share article

In a previous post, we looked at an example of a fictional bookstore company and recommended mirroring strategies for that specific scenario. In this post, we’ll be looking at a fictional bank and recommended mirroring strategies for their network traffic.

For a list of the most commonly used strategies, check out our traffic mirroring tutorial.

GoldenBank

In this scenario, the client is running two different services, in two different private VPCs. The two VPCs cannot communicate with each other. The reverse proxies handle the majority of the traffic and perform basic request validations such as URIs and web methods and possibly also parameters type validation and then forward the request to the frontend servers.

Unlike the previous scenario, the connection between the reverse proxy and the frontend servers is also encrypted in TLS. The frontend servers perform business context validation of the requests and then process them by sending requests to the backend servers.

AWS region

Close-up App1

app1 close up

Close-up App2

app2 close up

The backend servers in app1 rely on a storage server, while the backend servers of app2 rely on an AWS RDS service.

Following are the details of the security groups of all instances in both applications’ VPCs:

VPC App1

Rev Proxy

(No public IP)

FromToServiceStatus
anyanyHTTP/tcpAllow
anyanyHTTPS/tcpAllow
10.0.0.0/16FrontendHTTPS/tcpAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16DNSDNS/udpAllow
10.0.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Frontend

(No public IP)

FromToServiceStatus
Rev. Proxy10.0.0.0/16HTTPS/tcpAllow
10.0.0.0/16BackendCustom ports setAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16DNSDNS/udpAllow
10.0.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Backend

(No Public IP)

FromToServiceStatus
Frontend10.0.0.0/16Custom ports setAllow
10.0.0.0/16StorageNFS/udpAllow
10.0.0.0/16StorageNFS/tcpAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16DNSDNS/udpAllow
10.0.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Storage

(No Public IP)

FromToServiceStatus
Backend10.0.0.0/16NFS/udpAllow
Backend10.0.0.0/16NFS/tcpAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16DNSDNS/udpAllow
10.0.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

DNS

(No Public IP)

FromToServiceStatus
10.0.0.0/16anyDNS/udpAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Package Cache

(No Public IP)

10.0.0.0/16anyHTTP/tcp, HTTPS/tcpAllow
10.0.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.0.0.0/16DNSDNS/udpAllow

Here are our visibility recommendations for similar networks:

Reverse Proxies and Front-end servers

Outbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Monitor everything *acceptAll protocols0.0.0.0/00.0.0.0/0

* Even though most of the traffic is encrypted since it would include responses to client requests, the outbound traffic is still extremely valuable. It can indicate the size of the data being sent to the client and assist in detecting large data leaks, it can detect connections originating from the reverse proxy itself and help detect command and control connections, it can also detect connections or connection attempts to other servers other than the frontend servers which can detect lateral movement.

Inbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Drop HTTPS*rejectTCP4430.0.0.0/010.0.0.0/16
200Monitor everything elseacceptAll protocols0.0.0.0/00.0.0.0/0

* Since the incoming traffic on HTTPS is encrypted, the only thing it can indicate is its volume over time which can indicate certain types of DoS or DDoS and also the validity of the certificates used. So if the traffic volume is expected to be high, we would recommend excluding this traffic.

Backend servers

Since the frontend servers are contacting these servers with requests that are supposed to be valid after the checks done by the proxy and frontend servers, the traffic volume is expected to be significantly lower than traffic volume to the proxy and frontend servers.

Also, since this is the last chance to detect malicious traffic before it reaches the database, we believe that all traffic from these servers should be mirrored. However, if this is still too expensive for your organization, we would recommend that you use the following mirroring configuration to at least be able to tell if a backend server is behaving abnormally which may indicate that it has already been breached:

Outbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Exclude file service activity **rejectTCPNFS10.0.0.0/1610.0.0.0/16
200Exclude file service activity **rejectUDPNFS10.0.0.0/1610.0.0.0/16
300Monitor everything *acceptAll protocols0.0.0.0/00.0.0.0/0

* – Although this rule will also mirror the server’s responses to frontend requests we do recommend that you mirror it since it might also contain HTTPS connections initiated by malware running on the proxy servers to command and control servers. Also, this might include unintended responses that will indicate that an attack (such as XSS or SQL Injection) took place.

** – Since the filesystem activity is usually heavy, we would recommend you avoid mirroring this type of traffic unless it is highly valuable in your application or network architecture.

Inbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Monitor everything elseacceptAll protocols0.0.0.0/00.0.0.0/0

Storage Servers

Outbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Exclude file service activity *rejectTCPNFS10.0.0.0/1610.0.0.0/16
200Exclude file service activity *rejectUDPNFS10.0.0.0/1610.0.0.0/16
300Monitor everythingacceptAll protocols0.0.0.0/00.0.0.0/0

DNS server

Since this server generates a very minimal amount of traffic that is so extremely valuable for detecting so many types of attacks we recommend that you would mirror all traffic in and out of this server.

Package cache server

Since this server is responsible for packages installations on all other servers, the data from and to this server can answer questions like who installed what and when, which can be crucial in a forensic investigation and for detecting installation of malicious tools on the server.

Also, the traffic volume from and to this server is expected to be quite low. Therefore, we recommend that you would mirror all traffic in and out of this server.

Bastion server

Since this server is serving as the only way to manage all other servers, provided that it is being used to update the code on the various servers, install required packages, etc., the traffic volume should be relatively low and the value of the data it can provide is extremely high and therefore we recommend that you would mirror all traffic in and out of this server.

VPC App2

Rev Proxy

(No Public IP)

anyanyHTTP/tcpAllow
anyanyHTTPS/tcpAllow
10.1.0.0/16FrontendHTTPS/tcpAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16DNSDNS/udpAllow
10.1.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Frontend

(No Public IP)

Rev. Proxy10.1.0.0/16HTTPS/tcpAllow
10.1.0.0/16BackendCustom ports setAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16DNSDNS/udpAllow
10.1.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Backend

(No Public IP)

Frontend10.0.0.0/16Custom ports setAllow
10.1.0.0/16RDSDB portsAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16DNSDNS/udpAllow
10.1.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

RDS

(No Public IP)

Backend10.1.0.0/16DB portsAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16DNSDNS/udpAllow
10.1.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

DNS

(No Public IP)

10.1.0.0/16anyDNS/udpAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16Packages CacheHTTP/tcp, HTTPS/tcpAllow

Package Cache

(No Public IP)

10.1.0.0/16anyHTTP/tcp, HTTPS/tcpAllow
10.1.0.0/16anyNTP/udpAllow
BastionanySSH/tcpAllow
10.1.0.0/16DNSDNS/udpAllow

This network architecture is quite complex and would require an approach that would combine multiple data sources to provide proper security for it. In addition, it involves traffic that is completely encrypted, storage servers that are contacted via encrypted connections, and also AWS services that cannot be mirrored to the STA.

Here are our visibility recommendations for similar networks:

Reverse Proxies and Front-end servers

Outbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Monitor everything *acceptAll protocols0.0.0.0/00.0.0.0/0

* Even though most of the traffic is encrypted since it would include responses to clients requests, the outbound traffic is still extremely valuable. It can indicate the size of the data being sent to the client and assist in detecting large data leaks, it can detect connections originating from the reverse proxy itself and help detect command and control connections, it can also detect connections or connection attempts to other servers other than the frontend servers which can detect lateral movement.

Inbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Drop HTTPS*rejectTCP4430.0.0.0/010.1.0.0/16
200Monitor everything elseacceptAll protocols0.0.0.0/00.0.0.0/0

* Since the incoming traffic on HTTPS is encrypted, the only thing it can indicate is its volume over time which can indicate certain types of DoS or DDoS and also the validity of the certificates used. So if the traffic volume is expected to be high, we would recommend excluding this traffic.

Backend servers

Outbound

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Monitor everything *acceptAll protocols0.0.0.0/00.0.0.0/0

* – Since this server is expected to communicate with the RDS service, we would recommend mirroring all the traffic from this instance as this will allow the detection of database-related attacks. In addition, the RDS security group should be configured to allow connections only from the backend server (this can be more strict by configuring the RDS itself to allow connections only if they were identified by a certificate that exists only on the backend server) and an alert should be defined in Coralogix based on CloudTrail logs that would fire if the security group is modified.

Also, by forwarding the database logs from the RDS instance, an alert should be defined in Coralogix so it would be triggered if the database detects connections from other IPs.

Inbound

Since the traffic volume here should be quite low due to the validations done by the proxies and front-end servers, and due to the fact that this is the last point we can tap into the data before it is saved into the DB we recommend that you mirror all incoming traffic from this server

Rule numberDescriptionRule actionProtocolSource port rangeDestination port rangeSource CIDR blockDestination CIDR block
100Monitor everything *acceptAll protocols0.0.0.0/00.0.0.0/0

DNS server

Since this server generates a very minimal amount of traffic that is so extremely valuable for detecting so many types of attacks we recommend that you mirror all traffic in and out of this server.

Package cache server

Since this server is responsible for packages installations on all other servers, the data from and to this server can answer questions like who installed what when, which can be crucial in a forensic investigation and for detecting installation of malicious tools on the server. Also, the traffic volume from and to this server is expected to be quite low. Therefore, we recommend that you mirror all traffic in and out of this server.

Bastion server

Since this server is serving as the only way to manage all other servers, provided that it is being used to update the code on the various servers, install required packages, etc., the traffic volume should be relatively low and the value of the data it can provide is extremely high and therefore we recommend that you mirror all traffic in and out of this server.

Summary

Just like in any other field of security, there is no real right or wrong here, it’s more a matter of whether or not it is worth the trade-off in particular cases. There are several strategies that can be taken to minimize the overall cost of the AWS traffic monitoring solution and still get acceptable results.

For a list of the most commonly used strategies, check out our traffic mirroring tutorial.

Where Modern Observability
and Financial Savvy Meet.

Live Webinar
Next-Level O11y: Why Every DevOps Team Needs a RUM Strategy
April 30th at 12pm ET | 6pm CET
Save my Seat