Web Server abused as Proxy

One Monday morning we discovered the PostgreSQL database process had crashed on one of our Linux/Apache/PHP web-sites. We restarted the server and the site worked as normal, but we were concerned as to why the crash happened.

We started looking through all the various logs for clues. There was nothing unusual in the postgres logs, or the Apache error log, or other system logs. Disk-space, CPU and memory utilization were all within acceptable boundaries.

There was something strange in the Apache access_log however. Namely, lots of requests along the lines of: - - [06/May/2013:05:50:19 +1000] "GET HTTP/1.1" 200 1108 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
The strange thing here is that a client from IP address was making GET requests on our server back to their own IP on a different port. We had around 10 of these requests per second spanning over a six hour period.

To check for this type of thing in your own access logs, you can try grep "GET http" var/log/apache2/access* | less

The issue is that the HTTP spec allows a full URL in the GET request, which the web-server can choose to either retrieve for the client, or respond with some form of denial. This can be useful for proxy servers, and can be enabled in Apache by turning on mod_proxy. By default, with mod_proxy disabled, Apache will simply respond with the default VirtualHost main index page.

An easy way to check this for your own domain is to use telnet. For example:

  • telnet yoursite.example.com 80
  • GET http://www.yahoo.com/ HTTP/1.1 
  • Host: www.yahoo.com

(NOTE the two blank-lines at the end, i.e. pressing enter-enter to mark the end of the request)

This should return a response, which you can inspect to see what's happening. In our case, this was the default index of our site.

Now 10 requests per-second is not a very high load for large sites, but it was more than what we're configured for. As the index page runs multiple DB queries, this was enough to overwhelm our server (which probably points to some other configuration issue as to why it lead to a crash).

One way to try and prevent against leaking server resources to such spam attacks is to update the default VirtualHost to respond with a 403 Forbidden message for any requests that do not match any explicit VirtualHost Aliases. For example, adding the following VirtualHost directive achieved this for us:

  ServerName default.only
    Order allow,deny
    Deny from all

(NOTE that you need to ensure the ServerAlias directive in your main VirtualHost has all the domains or IP addresses that you DO want to server, including localhost if needed).

It's common for people to probe your site and try to use you as a proxy. Configuring the VirtualHost as above means you don't waste resources on this type of attack. However, somebody could just as easily request your index page 1000 times a second, which you'd need to deal with differently.


Post a Comment

Popular posts from this blog

Wkhtmltopdf font and sizing issues

Import Google Contacts to Nokia PC Suite

Can't delete last blank page from Word