Akismet Update and Server Problems

After monitoring the comments that Akismet blocked very carefully, I can report that I’ve had no false positives for nearly a week. I’m not quite sure what changed to fix things.
With a dynamic system like Akismet, things will change over time. That is the nature of the beast. I don’t know whether Matt and the crew tweaked something, or whether a concerted poisoning attempt stopped being effective, but I’m glad I can start trusting it again.
I suspect it was the former because the change back was very dramatic, though I’m sure Automattic would not want to admit to it.

Server Trouble

In the meantime, at around one this morning, my server went down, or rather my blog stopped working. After a quick investigation, I determined that the database server was complaining of too many connections. I checked and there were a large number of httpd processes running. Presumably each, or most had a database connection open. Static files were being served ok, but anything involving the database was failing.

I restarted the Apache and that seemed to cure it. I started checking through log files to see if I could determine the culprit, but found nothing suspicious. Fifteen minutes later the site was down again. I then spent the next two hours monitoring the situation. A quick script allowed me to watch the process count:

ps -ef | grep httpd | wc -l

It was growing quite rapidly from an initial 16 to over 100, though the site would start failing at about 80. In the end I gave up when the process count stayed stable for 20 minutes. Though when I checked after a few hours sleep, it had gone down again and was down for over 5 hours. I’m presuming it was an attack of some kind.
It has since gone down again, but the growth in number of processes seems to take a much longer time. I didn’t find anything obvious in the logs that I checked, but maybe it is one of the lesser sites which is being attacked. I will continue to investigate…

13 thoughts on “Akismet Update and Server Problems

  1. I’ve had similar problems on my server for quite a while. Sometimes I get these bursts of connections that would grind my server to a halt. I eventually modified my apache configuration to limit the maximum number of running servers, until I found a level that the box seems to be able to handle.

  2. In the past week nothing has changed on the algorithim side, so it might just have been that the system finally learned to identify the strain of spam and ham you were getting after it got more data. Spam is a lot like a virus and as new strains pop up it can take some time (anywhere from 2-10 comments) for it to be identified and squashed.

  3. Mike,
    I’m working on a website and using your journalized winter theme, real cool! How do I set up the right column so I upload links from the admin screen in WordPress.

    If I use the classic theme, I can upload links through the admin screen, but with your theme, they do not show up.

    Thanks,
    Mike

  4. Mike, consider running a cron job such as

    killall -HUP httpd
    httpd

    This will periodically disable your Web daemons for a few milliseconds and restart a single session. Choose frequency in accordance with server strain.

  5. Hi Roy,
    That’s a good idea but it would be much ‘cleaner’ to use the proper start up script to do something like that. In my case

    /etc/init.d/httpd restart

    But I’d still rather do that only having detected something wrong like too many connections

    Mike

  6. I assume that restarting httpd will send a signal to one or all of the existing daemons. Why not add a conditional statement (e.g. ps -ef | grep httpd | wc -l > 2). Nothing to lose here… if the state is morbid, killall -HUP httpd; /etc/init.d/httpd restart; (if that’s the path in your distribution)

  7. Roy,
    It is the ‘killall’ command that I regard as not clean. For instance, I sometime run more than one web server on my machine. I wouldn’t want to kill all of them and only re-start one.

    The restart option for the apache start-up script will first stop the process it started before starting another.

    Thus my cron script might look like

    if [ `ps -ef | grep httpd | wc -l` -gt "25" ]
    then /etc/init.d restart
    fi

    (I haven’t tried this, so the syntax might be wrong)

    Mike

  8. Indeed, I got the same attack at my weblog last week. I had to scramble and tone a lot of system services down, since the attacker hit my httpd, and it was configured with a maxclients setting too high, causing a load of over 266.

Leave a Reply

Your email address will not be published. Required fields are marked *