Managing Apache Logs and Server Performance: A Real-World Case Study
Adam C. |

Server performance is critical for any web application, but it’s common to encounter issues caused by factors like excessive log growth, abusive IP traffic, or internal requests. In this blog post, we’ll walk through a real-world example of addressing these issues and optimizing server performance.

Photo by Joe Hernandez on Unsplash

The Problem

1. Excessive Log Growth

The other_vhosts_access.log file was growing at an alarming rate, reaching hundreds of megabytes in a short span of time. This rapid growth was making it difficult to analyze logs and increasing disk I/O, potentially affecting overall server performance.

2. Abusive IP Traffic

A monitoring script revealed multiple IPs repeatedly generating requests resulting in 403 Forbidden responses. These IPs seemed to be attempting to access unauthorized resources, putting unnecessary load on the server.

3. Internal Requests Flooding Logs

Internal requests (from private IPs) were also being logged, contributing significantly to the size of the log file. These requests were not useful for analysis but were consuming valuable resources.

The Solution

To address these issues, we took a series of targeted steps:

1. Handling Large Log Files

Analyzing Log Size

We started by analyzing the size and growth rate of the other_vhosts_access.log file using the following command:

wc -l /var/log/apache2/other_vhosts_access.log

This revealed tens of thousands of entries within minutes, highlighting the need for optimization.

Implementing Log Rotation

To prevent logs from growing indefinitely, we ensured logrotate was configured to rotate and compress logs:

cat /etc/logrotate.d/apache2

Example configuration:

/var/log/apache2/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        systemctl reload apache2 > /dev/null
    endscript
}

2. Blocking Abusive IPs

Identifying Abusive IPs

Using the monitoring script, we identified IPs generating repeated 403 requests:

awk '{print $1}' /var/log/apache2/other_vhosts_access.log | sort | uniq -c | sort -nr | head

Adding IPs to Firewall

To block these IPs, we used iptables:

sudo iptables -I INPUT 1 -s <IP> -j DROP

The -I INPUT 1 ensures the rule is added at the top of the chain, so the traffic is blocked before further processing.

We verified the rules:

sudo iptables -L INPUT -n -v --line-numbers

3. Excluding Internal IPs from Logs

Updating Apache Configuration

Internal IPs like 10.x.x.x and 127.0.0.1 were generating many log entries. To exclude them, we updated the Apache configuration:

SetEnvIf Remote_Addr ^10\. no_log
SetEnvIf Remote_Addr ^127\. no_log
SetEnvIf Remote_Addr ^192\.168\. no_log
SetEnvIf Remote_Addr ^172\.16\. no_log
CustomLog /var/log/apache2/other_vhosts_access.log combined env=!no_log

This configuration ensures requests from internal IPs are ignored in the logs.

Reloading Apache

We applied the changes by reloading Apache:

sudo systemctl reload apache2

Verifying the Results

After these changes, we monitored the logs:

grep '10\.' /var/log/apache2/other_vhosts_access.log | wc -l

Requests from internal IPs no longer appeared in the logs, significantly reducing their size.

4. Optimizing IP Blocking Rules

Adding a large number of individual IP blocking rules to iptables can potentially slow down performance, especially if your server processes a high volume of incoming traffic. Here’s a detailed explanation:

How iptables Processes Rules

  1. Rule Evaluation:
    • iptables evaluates rules sequentially, starting from the top of the chain.
    • For each incoming packet, it traverses the rules until a match is found or the packet reaches the default policy.
  2. Impact of Many Rules:
    • As the number of rules increases, the time taken to evaluate each packet also increases.
    • If there are hundreds or thousands of DROP rules, each packet may take slightly longer to process, which can add up under heavy traffic.

When Does It Affect Performance?

  1. Large Number of Rules:
    • Having hundreds or thousands of IP blocking rules can cause noticeable delays.
    • If you’re only adding a few dozen rules, the performance impact is negligible.
  2. High Traffic Volume:
    • On a busy server with thousands of incoming requests per second, even small delays from rule evaluation can accumulate.
  3. Resource Constraints:
    • On resource-limited servers (low CPU/memory), the overhead of processing many rules can become significant.

Solutions for Better Performance

If you need to block many IPs efficiently, consider these alternatives:

  1. Use IP Sets
    • IP sets allow you to manage large groups of IPs efficiently. Instead of adding individual rules, you create an IP set and reference it in a single rule.
    • Steps:

Install ipset (if not already installed):

sudo apt install ipset

Create an IP set:

sudo ipset create blacklist hash:ip

Add IPs to the set:

sudo ipset add blacklist 47.128.111.209
sudo ipset add blacklist 47.128.39.234
sudo ipset add blacklist 111.225.148.31
sudo ipset add blacklist 110.249.201.41
sudo ipset add blacklist 111.225.148.17

Create an iptables rule to drop traffic from the IP set:

sudo iptables -I INPUT -m set --match-set blacklist src -j DROP

Save the rules:

sudo ipset save > /etc/ipset.conf
sudo iptables-save > /etc/iptables/rules.v4
  • Advantages:
    • The IP set is managed in memory, making it much faster.
    • Adding or removing IPs is efficient and doesn’t require modifying the iptables rules.
  1. Use a Web Application Firewall (WAF):
    • A WAF like Cloudflare or AWS WAF can block abusive IPs before they reach your server.
    • These services can block entire regions or IP ranges with minimal performance impact.
  2. Block by IP Range:
    • If abusive IPs come from a small number of IP ranges, block the entire range instead of individual IPs.

Example:

sudo iptables -I INPUT -s 111.225.148.0/24 -j DROP

Conclusion

  • A Few Dozen IP Rules: Won’t significantly affect performance.
  • Many Rules (Hundreds/Thousands): Use ipset for better performance.
  • High Traffic: Consider a WAF or rate-limiting to mitigate abuse efficiently.

Let me know if you’d like help setting up ipset or exploring other options!

The Results

After implementing these solutions:

  • Log Growth Reduced: The other_vhosts_access.log file size was drastically reduced, making it easier to manage and analyze.
  • Abusive Traffic Mitigated: Blocking abusive IPs reduced unnecessary load on the server.
  • Improved Performance: Excluding internal IPs and reducing disk I/O improved overall server performance.

Lessons Learned

  1. Monitor Logs Regularly: Keep an eye on log growth to identify unusual traffic patterns early.
  2. Block Abusive IPs: Use tools like iptables to immediately block suspicious traffic.
  3. Optimize Logging: Exclude irrelevant requests, like internal traffic, from logs to reduce overhead.
  4. Automate with Scripts: Monitoring scripts can help detect and respond to issues faster.

Conclusion

By addressing excessive log growth, blocking abusive traffic, and optimizing logging, we significantly improved server performance. These steps are essential for maintaining a healthy and efficient web application infrastructure.

If you’re facing similar issues, follow these strategies to optimize your server and keep your application running smoothly.