Reaching the open files limit of a Qpid server with simple Nmap requests

I recently noticed a curious behavior of Apache Qpid Messaging server 0.20 while performing some simple nmap checks:

while true; do
  echo "$(date +'%Y/%m/%d-%H:%M:%S') $(nmap -p 5672 qpid_server | grep '^5672\/tcp')" 
  sleep 0.5
done

Qpid’s log started growing very fast and very soon it was flooded with many (really, many: hundreds of thousands) lines like this one:

debug Exception constructed: Too many open files (qpid/sys/posix/BSDSocket.cpp:219)

So I checked the resources used by qpidd daemon and I realized that it already reached its open files limit:

# lsof -p 29529 | awk '{ if ( $1 == "COMMAND" || $1 == "qpidd" ) print $0 }'
COMMAND     PID      USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
qpidd   29529 root   29u  sock       0,6        0t0 888124153 can't identify protocol
(...)

# lsof -p 29529 | awk '{ if ( $1 == "qpidd" ) print $0 }' | awk '{ print $5 }' | sort | uniq -c | sort -nr | awk 'BEGIN { total=0 } { total=total+$1 ; print $0 } END { printf "%7s TOTAL\n",total }'
 998 sock
 41 REG
 7 FIFO
 4 CHR
 3 IPv4
 2 DIR
 1 IPv6
 1056 TOTAL

# ulimit -a | grep 'open files'
open files (-n) 1024

Other simple TCP checks also changed from:

# /usr/lib64/nagios/plugins/check_tcp -H qpid_server -p 5672
TCP OK - 0.001 second response time on port 5672|time=0.000820s;;;0.000000;10.000000

to

TCP OK - 0.997 second response time on port 5672|time=0.997461s;;;0.000000;10.000000
CRITICAL - Socket timeout after 1 seconds

and the server’s performance became much worse as the minutes went by. I created a chart with the following tests and here is the result:

Chart: seconds versus memory, cpu and sockets percentage

Chart: seconds versus memory, CPU and sockets percentage

So, after ten minutes the open files limit was reached and since that moment the CPU usage was increased in almost 2 % each minute until I stopped the daemon (less than 15 minutes after, with 30 % of CPU usage).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: