Oddhead Blog

For the second time, a hacker (in the swine sense of the word) broke in and defaced Oddhead Blog. Once again, I’m left impressed by the ingenuity of web malefactors and entirely mystified as to their motivation.

Last week several readers notified me that my rss feed on Google Reader was filled with spam (“Order Emsam No RxOrder Emsam Overnight DeliveryOrder… BuyBuy…”).

The strange part was, the feed looked fine when accessed directly on my website or via Bloglines. Only when Google requested the feed did it become corrupted, thus mucking up my content inside Google Reader but not on my website.

(Hat tip to Anthony who diagnosed the ailment: calling curl http://blog.oddhead.com/feed/ yielded clean output, while the same request masquerading as coming from Google, curl -A ‘Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 10 subscribers; feed-id=12312313123123)’ http://blog.oddhead.com/feed/, yielded the spammed-up version.)

In the meantime, Google Search had apparently deduced that my site was compromised and categorized my blog as spam. Look at the difference between these two searches. Nearly every page containing the query terms, no matter how tangential, takes precedence over blog.oddhead.com in the results. [2009/06/23 Update: This is no longer the case: Apparently Google Search has reconsidered my blog.]

So began a lengthy investigation to find and eradicate the invader. The offending text did not appear anywhere in my WordPress code or database. Argg. I found that my plugins directory was world-writeable: uh oh. Then I found a file named remv.php in my themes directory containing a decidedly un-automattic jumble of code. Apparently this is an especially nasty bugger:

I’ve never seen a hack crop up with the tenacity of “remv.php” tho. Seriously, it’s kind of scary.

I’m still not sure how or even if an attacker used remv.php to corrupt my feed in such a subtle way. I decided on surgery by chainsaw rather than scalpel. I exported all my content into a WordPress XML file, deleted my entire installation of WordPress, reinstalled WordPress, then imported my content back in. I restored my theme and re-entered some meta data, but I still have many ongoing repairs to do like importing my blogroll and other links.

The attack was clever: a virus that sickens but does not kill the patient. The disease left my web site functioning perfectly well, making it less likely for me to notice and harder to track down. The bizarre symptom — corrupting the rss feed but only inside Google Reader — led Chris to wonder if the attacker knew I was a Yahoo! loyalist. That seems unlikely. I don’t think I have enemies who care that much. Also, the spammy feed appeared in Technorati as well. Almost surely I was the victim of an indiscriminate robot attack. Still, after searching around, I couldn’t find another example of exactly this form of RSS feed “selective corruption”: has anyone seen or heard of this attack or can find it? And can anyone explain why?

What did I learn? I learned to listen to Chris and not make him mad.

I also found a bunch of useful WordPress security tips, resources, and plugins that might be useful to others including my future self:

• WordPress, remv.php and you
• 3 must apply security tips for WordPress
• Hardening WordPress
• 5 plugins to keep WordPress secure
• Anatomy of a WordPress hack (“The kicker? All these sites were on Dreamhost.”)
• Did your WordPress site get hacked?
• DreamHost: Troubleshooting hacked sites
• Dealing with a hacker on DreamHost
• Docs on WordPress feeds
• AskApache plugin to display all the internal WordPress URL rewrite rules (example use) (I couldn’t discern how to interpret the output)
• WordPress exploit scanner plugin (I didn’t use after this question spooked me)
• Secure WordPress plugin
• AskApache password protect plugin

I’m planning to take the World Blogging Month (WoBloMo) challenge in March. Join me!

The goal is simple: blog at least every other day from March 1 to March 31. Post something — anything — on every odd day of the month and you win. Skip any day not divisible by 2 and you lose.

Many bloggers already write every day or nearly so. More power to them. For the rest of us, who blog infrequently and spend copious time arguing with their inner editors, ludicrous and artificial pretenses can be a good thing.

WoBloMo resembles the write-a-novel-in-a-month contest NaNoWriMo and other timed artistic challenges prefaced on the idea that quantity and quality can be friends. By suppressing the Spock-like perfectionist inside you, you can bring out your inner Kirk and “just do it”. Agonizing over details always has diminishing returns and sometimes, perversely, can make things worse. Or so the theory goes. You be the judge once (if) my WoBloMo fountain erupts.

Added 2009/02/26: Full disclosure.

As I alluded to previously, I seem to be getting “intelligent spam” on my blog: comments that pass the re-captcha test and seem on-topic, yet upon further inspection clearly constitute link spam: either the author URI or a link in the comment body is spam.

Here is one of the most clear cases, received on January 9 as a comment to my post on the CFTC’s call for proposals to regulate prediction markets:

Date: Fri, 9 Jan 2009 01:28:01 -0800
From: Matt.Herdy
New comment on your post #71 “A historic MayDay: The US
government’s call for help on regulating prediction markets”
Author : Matt.Herdy
Comment:
Thanks for that post. I’ll put a note in the post.

1. It’s nothing new. The CFTC will just formalize the current
status quo.
2. We are prisoner of the CFTC regulations and the US Congress’
distaste of sports “gambling”. As for the profitability of prediction
exchanges in that strict environment, I don’t see how you can deny that
HedgeStreet went bankrupt even though it was well funded. Isn’t that a
hard fact?
3. You’re right, but all “pragmatists” should follow a business
plan and make profits. See point #2. Pragmatists won’t make miracles.

<a href=”http://www.stretch-marks-help.com/”>Removing stretch marks</a>

At first blush, the comments seems to come from a knowledgeable person: they refer to HedgeStreet, an extremely relevant yet mostly unknown company that’s not mentioned anywhere else in the post or other comments.

It turns out the comments seem intelligent because they are. In fact, they’re copied word for word from Chris Masse’s comments on his own blog.

Chris Masse’s page has a link to my page, so it could have been discovered with a “link:” query to a search engine.

Though now I understand what this spammer did, I remain puzzled exactly how they did it and especially why.

1. Are these comments being inserted by people, perhaps hired on Mechanical Turk or other underground equivalent? Or are they coming from robots who have either broken re-captcha or the security of my blog? (John suspects a security breach.)
2. Is it really worth it economically? All links in blog comments are NOFOLLOW links anyway, and disregarded by search engines for ranking purposes, so what is the point? Are they looking for actual humans to click these links?

In any case, it seems an intriguing development in the spam arms race. Are other bloggers getting “intelligent spam”? Does anyone know how it’s done and why?

I joined the quantcast audience measurement service. It took about two minutes to sign up and initiate tracking. I’m impressed with the ease of use, the utility, and the inroads the company has made in the year or so since former Yahoo Mike Speiser first showed it to me.

Looks like I’m getting about 1000 visitors a month, roughly 3/4 that of Chris, 1/6 of Robin, 1/10 of Lance, 0.00079% of my employer, and 0.00073% of my employer’s frenemy.

I also joined the scribd document hosting service (“Youtube for documents”) and used it to embed a PDF in my previous post. Again, from signup to service took a matter of minutes. (I think scribd could be great for hosting my publications which are in need of both a content and interface update.)

Probably there’s some sort of business axiom here, probably already blogged and book-ed: the two minute rule of successful web services.

Sorry folks, I just released 26 comments from purgatory where they had been sitting for as long as 58 days. All pending comments have now been approved and posted. I’ll try to go through them soon and respond where appropriate.

About two months ago I changed my WordPress configuration and it turns out that comments were piling up for moderation without email notification, and I failed to spot the growing queue until now.

Since I’m using re-captcha and have turned off trackbacks, I shouldn’t need to moderate comments going forward, so I’ve turned off moderation (fingers crossed).

Here’s an update on my fractured relationship with The Economist magazine.

To my pleasant surprise, Alan Press, Vice President of Marketing & Circulation at The Economist actually posted a comment on my blog agreeing to cease and desist their renewal scare tactics!

We agree, the language is bad. We are discontinuing the use of this letter going forward, and will replace it with a message that makes clear how much we value readers like you.

(I didn’t notice the concession at first, as his comment got stuck in my Akismet spam folder for several days.)

I thought this was a stand-up gesture. I temporarily felt all warm and fuzzy about the good old days when The Economist and I first met. In all seriousness, I do appreciate the public comment and the prompt/effective action.

So are we getting back together?

That’s none of your business!

In any case, I’m happy to see blogplaining/freedbacking actually have an effect.

I‘ve been blogging for about 33 weeks and this is my 31st post. Of those, I’d say roughly 12 are meals¹ and 19 are snacks. So I’m clocking in a bit below one post per week, 1.5 meals per month.

If you feel that’s too few, or if you have any other comments or recommendations let me know. I’ll see what I can do. Without satisfied readers I’m just a tree falling in the woods 0.94 times a week.

In the meantime, if you’re craving more, you’re welcome to subscribe to the RSS feed of my shared bookmarks.^2,3 There you can track me goofing off — er, conducting vital industry research. My bookmarking pace is closer to daily and I try to annotate each site with a revealing sentence or two.

Here’s an example of what the feed looks like in bloglines.⁴

Yesterday I was hacked.

More specifically, my web hosting service DreamHost was hacked and Oddhead Blog was hijacked. Someone stole a bunch of passwords and methodically replaced index.* files, including the index.php file in my wordpress directory. If you visited this blog yesterday, this is what you saw.

Also yesterday, I noticed that I’ve been splogged. (Splog = spam + blog.)

The strange part about both incidents is that neither attack has an obvious motive. I don’t see any blatant ads or links or any real benefit that the attacker gained. Probably SEO related, but I just don’t see it.

Tough neighborhood, these Internets.

I’ve been MarginalRevolutioned and kottked [?]. Welcome!

Some administrivia:

1. I was previously publishing the wrong RSS feed URL, which was truncating posts. If you’d like to subscribe to this blog, the correct RSS feed URL, with unabridged posts, is: http://blog.oddhead.com/feed/
2. I’m trying out a new wordpress theme: “Tiga”. So far I am very happy with it: easy to customize, clean, bug free, etc.
3. One among several bugs in my previously theme was that comments were broken. Comments now seem to be working fine.