After monitoring the comments that Akismet blocked very carefully, I can report that I’ve had no false positives for nearly a week. I’m not quite sure what changed to fix things.
With a dynamic system like Akismet, things will change over time. That is the nature of the beast. I don’t know whether Matt and the crew tweaked something, or whether a concerted poisoning attempt stopped being effective, but I’m glad I can start trusting it again.
I suspect it was the former because the change back was very dramatic, though I’m sure Automattic would not want to admit to it.

Server Trouble

In the meantime, at around one this morning, my server went down, or rather my blog stopped working. After a quick investigation, I determined that the database server was complaining of too many connections. I checked and there were a large number of httpd processes running. Presumably each, or most had a database connection open. Static files were being served ok, but anything involving the database was failing.

I restarted the Apache and that seemed to cure it. I started checking through log files to see if I could determine the culprit, but found nothing suspicious. Fifteen minutes later the site was down again. I then spent the next two hours monitoring the situation. A quick script allowed me to watch the process count:

ps -ef | grep httpd | wc -l

It was growing quite rapidly from an initial 16 to over 100, though the site would start failing at about 80. In the end I gave up when the process count stayed stable for 20 minutes. Though when I checked after a few hours sleep, it had gone down again and was down for over 5 hours. I’m presuming it was an attack of some kind.
It has since gone down again, but the growth in number of processes seems to take a much longer time. I didn’t find anything obvious in the logs that I checked, but maybe it is one of the lesser sites which is being attacked. I will continue to investigate…

I’ve just spent a rather painful 45 minutes recovering legitimate comments from my Akismet admin panel. Painful because Akismet had over 1400 comments marked as spam from the last week.

That number is not excessive for my blog: Akismet has caught over 12,000 spam comments since I installed it; but I’ve not been keeping on top of the list this week. Unfortunately checking for false positives is impossible once you have more than 150 spam comments.

The Akismet plugin displays the newest 150 comments each with a check box to allow you to separate the legitimate ones from the rubbish (the ham from the spam). That’s great: check the boxes, push the “not spam” button.

However, the only other action is to delete all the comments that Akismet has determined are spam. But if you have legitimate comments that are not in the most recent 150, you cannot see them to rescue them.

Luckily for me, I’m technical enough that I can figure out how to get round problems like this, but most people are not.

In the end, I rescued somewhere between 40 and 50 comments. I’m not sure of the exact number because I wasn’t paying attention, and releasing them from Akismet’s clutches doesn’t trigger the email notification so I can’t count the emails either. Not one single comment was let through in the last 6 days. I don’t know whether this is a minor hiccup from Akismet or the start of an alarming trend.

I then spent another 20 minutes responding to some of them. Oh yeah, and I’ve spent another 30 minutes writing this post!

I think I will have to look at enhancing the Akismet plugin. Either by adding a ‘delete just spam in the list’ button or by adding pagination to the list of spams. The former sounds far easier than the latter.

Although it has been available for a few days now, WordPress 2.0 ‘Duke’ is now official.

In fact has already had been downloaded more than 33,000 times as I write this! You can download it from the revamped WordPress.org web site.

There is a lot of new code in this release, most of which is concerned with either the Administration interface, especially the WYSIWYG write interface, or under the hood stuff which will benefit plugin and theme developers. There are a whole bunch of bug fixes too.

There are some very compelling reasons to upgrade to this new version, but there are a whole bunch of reasons to hold off too: Continue reading →