Hobbit DDOS Attack Detection (submission)
list Charles Jones
I had a hobbit-monitored site become the victim of a DDOS syn-flood attack. To help detect this in the futrue and also aid information-gathering, I did the following, which I am sharing:
1. Made the following additions to hobbit-clients.cfg
# Detect more than 100 half-open connections (possible syn-flood attack)
PORT STATE=SYN_RECV MIN=0 MAX=100 COLOR=red TRACK="SYN"
# Detect more than 2000 established connections (possible simple DDOS http get attack)
PORT STATE=ESTABLISHED MIN=0 MAX=2000 COLOR=red
2. Made a small modification to hobbit-linux.sh, specifically to the [ports] section. I crammed it all into a single line.
echo "[ports]"
echo "SYN_REC Quick Stats:";SYNs=`netstat -pant 2>/dev/null | grep SYN | awk -F: '{print $2;}' |awk {'print $2"\t"$1'}| sort | uniq -c |sort -n`;if [ -n "$SYNs" ]; then echo " # Address Port";printf "$SYNs\n";echo =============================;else echo "No SYNs Found";echo =============================;fi
# Bug in RedHat's netstat spews annoying error messages.
netstat -ant 2>/dev/null
#1 allows Hobbit to detect and alert for 2 common DDOS attack signatures (syn floods and plain old http overloading), as well as creating graphs via the "track" feature.
#2 prefixes the netstat output you see in the "ports" column with a table of IPs that have half-open connections. It shows the address, IP, and what port they are "attacking". Here is an example (IPs and ports masked of course):
SYN_REC Quick Stats:
# Address Port
1 289.122.3.20 80
1 213.102.135.60 80
1 200.120.152.6 8080
1 201.192.9.130 443
1 174.231.84.4 443
2 191.136.92.135 80
2 216.122.32.240 8080
=============================
The prefixing of this data in the client message has no effect on the Hobbit server, as it is only looking for the netstat output in the ports section, and so ignores that extra data (but still displays it). So if the site gets a syn flood, you can check the ports column of your web host(s) and easily see the offending IP(s) that are performing the attack. Note that most "floodbots" are coded such that they spoof their IP addresses, so the information you gather may not actually help you determine who is attacking, but is still useful for temporarily firewalling or whatever other steps you need to take to block it.
Note: this works on RHEL, so should work on RedHat/CentOS/Fedora. I havn't checked to see if debian/ubuntu has the same netstat parameters I used, so YMMV :)
-Charles
list Josh Luthman
What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks...
▸
On 1/25/08, Charles Jones <user-e86b4aeade4e@xymon.invalid> wrote:
I had a hobbit-monitored site become the victim of a DDOS syn-flood
attack. To help detect this in the futrue and also aid
information-gathering, I did the following, which I am sharing:
1. Made the following additions to hobbit-clients.cfg
# Detect more than 100 half-open connections (possible syn-flood attack)
PORT STATE=SYN_RECV MIN=0 MAX=100 COLOR=red TRACK="SYN"
# Detect more than 2000 established connections (possible simple DDOS
http get attack)
PORT STATE=ESTABLISHED MIN=0 MAX=2000 COLOR=red
2. Made a small modification to hobbit-linux.sh, specifically to the
[ports] section. I crammed it all into a single line.
echo "[ports]"
echo "SYN_REC Quick Stats:";SYNs=`netstat -pant 2>/dev/null | grep SYN
| awk -F: '{print $2;}' |awk {'print $2"\t"$1'}| sort | uniq -c |sort
-n`;if [ -n "$SYNs" ]; then echo " # Address Port";printf
"$SYNs\n";echo =============================;else echo "No SYNs
Found";echo =============================;fi
# Bug in RedHat's netstat spews annoying error messages.
netstat -ant 2>/dev/null
#1 allows Hobbit to detect and alert for 2 common DDOS attack signatures
(syn floods and plain old http overloading), as well as creating graphs
via the "track" feature.
#2 prefixes the netstat output you see in the "ports" column with a
table of IPs that have half-open connections. It shows the address, IP,
and what port they are "attacking". Here is an example (IPs and ports
masked of course):
SYN_REC Quick Stats:
# Address Port
1 289.122.3.20 80
1 213.102.135.60 80
1 200.120.152.6 8080
1 201.192.9.130 443
1 174.231.84.4 443
2 191.136.92.135 80
2 216.122.32.240 8080
=============================
The prefixing of this data in the client message has no effect on the
Hobbit server, as it is only looking for the netstat output in the ports
section, and so ignores that extra data (but still displays it). So if
the site gets a syn flood, you can check the ports column of your web
host(s) and easily see the offending IP(s) that are performing the
attack. Note that most "floodbots" are coded such that they spoof their
IP addresses, so the information you gather may not actually help you
determine who is attacking, but is still useful for temporarily
firewalling or whatever other steps you need to take to block it.
Note: this works on RHEL, so should work on RedHat/CentOS/Fedora. I
havn't checked to see if debian/ubuntu has the same netstat parameters I
used, so YMMV :)
-Charles
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer
list Charles Jones
▸
Josh Luthman wrote:
What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks...
I admin and consult on numerous Hobbit installations. This particular one is a small web cluster whose dedicated hosting provider has no IDS or decent firewall, as they get hit by these sorts of attacks and the only way to stop them is to request the blockage of specific IPs. I didn't mean to imply the "patch" I submitted was for everyone, just those who might find it useful :) -Charles
list Josh Luthman
Two different worlds, two different solutions =( Thanks a whole bunch for the notes, though. I'm sure I'll come back to use it someday!
▸
On 1/25/08, Charles Jones <user-e86b4aeade4e@xymon.invalid> wrote:Josh Luthman wrote:What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks...I admin and consult on numerous Hobbit installations. This particular one is a small web cluster whose dedicated hosting provider has no IDS or decent firewall, as they get hit by these sorts of attacks and the only way to stop them is to request the blockage of specific IPs. I didn't mean to imply the "patch" I submitted was for everyone, just those who might find it useful :) -Charles
-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
list Henrik Størner
▸
On Fri, Jan 25, 2008 at 01:59:08PM -0700, Charles Jones wrote:
I had a hobbit-monitored site become the victim of a DDOS syn-flood attack. To help detect this in the futrue and also aid information-gathering, I did the following [...]
Quite off-topic, but since this was on a Linux system you might want to look into enabling syn-cookies on these servers. sysctl net.ipv4.tcp_syncookies=1 somewhere in a startup script will do that. It has no impact on the system unless the SYN connection queue fills up. There's a thread discussing this on the Red Hat mailing list: http://www.redhat.com/archives/rhl-devel-list/2005-January/msg00447.html but the discussion is generic to Linux in all versions. Alan Cox (one of the main Linux kernel developers) also kills some of the myths about this setting in http://www.redhat.com/archives/rhl-devel-list/2005-January/msg00483.html It won't help on the simple connection flooding (2000 active connections to your webserver), but there are other ways of stopping that - e.g. with Linux' built-in firewall (iptables): http://www.debian-administration.org/articles/187 The example is for fending off SSH brute-force attacks, but the same mechanism can be used for protecting webservers - although you will probably want to raise the limits from the 3 connections/second that the example allows. Regards, Henrik