[Previous entry: ""] [Main Index] [Next entry: "Dilbert"]

11/30/2003 Archived Entry: "Spam, spam, spam, and spam"

Something of a milestone: last week I received over 2000 junk emails. Spam seems to be on an accelerating growth curve. What can be done?

The first rule in dealing with spam is this: never, under any circumstances, for any reason, buy something from an email advertisement. Right now, if only 50 in every million people respond to a spam, the spammer is making a profit. The only long term solution to spam is to eliminate this profit, which means, don't buy the product, and don't follow the links.

Also, never reply to the email or use the "unsubscribe" link. These are usually phony, and simply confrm to the spammer that he's found a "live" email address which he can then sell on to others.

When I say "don't buy," I include "legitimate" commercial email too. I don't care if they're trying to sell fake Viagra or sex videos or software updates or holiday vacations -- if email advertising is profitable, it will only increase. The US government in its ridiculous attempt to legislate spam away is leaving plenty of loopholes for lobbyists -- er, businessmen -- to send you "unsolicited commercial email." Nor do I trust "opt-out" lists; there's no evidence that they work, and I don't want to add my name to yet another government-collected list.

No, the problem of spam is one that must be fought on an individual level. Fortunately there are lots of actions you can take to prevent, block, or ameliorate spam.

If you are not receiving spam, take steps now to ensure that you don't. You must never make your email address visible on the web...because there are computer programs constantly searching the web for email addresses, building databases that they can sell to spammers. Therefore:

* Never put your email address on a web page, either as plain text or as a mailto: link.

* Never post your email address in an online bulletin board. This is the same as putting in on a web page, as far as the robots are concerned. Be careful because some BB software will automatically put a mailto: link on your posts, if you supply an email address when you register. (Most BBs allow you to block this "feature.")

* Never post your email address in an Internet newsgroup, either in the text or in the Reply: field. If you need people to contact you, garble the address with some nonsense characters, and then in the text tell people how to get your real address. For example, if you're joe@yahoo.com, tell people your email address is joe1234@yahoo.gov, and then in the text tell people "remove the numbers and change gov to com". Humans can understand this, but robots can't.

If you're getting spammed, but you don't really need a "public" email address, change your email address. Tell your family and friends and anyone else who needs to contact you, and shut the old address down. Then stick to the rules above.

If, like me, you need to have a public email address, you're going to get flooded. But you can arm yourself with some tools.

* Filter out. Almost all modern email programs let you "filter" messages into mailboxes based on Subject, From, or To. You can use this, for example, to send to the trash any message with "Viagra" in the subject line. (One of my filters trashes any message with a $ in the subject line.) The downside is that spammers are hip to this, so they've started saying "V1agra" instead of "Viagra" and similar tricks. You need to constantly keep updating your filters.

* Filter in. A much more effective use of filters is to look for email of interest, and filter that into a special mailbox. For example, when I receive email from certain clients, it automatically gets filtered into a mailbox with their company name. This also serves to bring it to my attention. I handle those mailboxes first, and "email from strangers" in my In box gets read later. You can also filter for subjects you're interested in, mail lists you belong to, and so on.

* Rule-based filters. There are now some very sophisticated filtering programs, such as SpamAssassin, which use a combination of rules to "score" a message. These rules can look for clues like a phony date or a bogus return address. The rules need to be constantly updated, but some of these programs will link to a central repository which tracks the latest spam trends. (Be warned, SpamAssassin is not easy to install. It's really designed for mail servers, not for individual users.)

* Bayesian filtering. A new weapon in the anti-spam arsenal is the "Bayesian" filter. This is an adaptive filter that learns to distinguish spam from non-spam messages based on their content. ("Bayesian" refers to the statistical math that's used to evaluate a message.) Because they are constantly being "trained," they learn new spam patterns quickly. Also, the filter learns to recognize not only messages you don't want, but also messages you do want. Several email programs, including Mozilla and Eudora now have this built-in, making it easy to use. There are also add-on programs like PopFile for other email clients.

* Blacklists. Some spam filters try to keep a "blacklist" of computers on the Internet which are known to send spam. Unfortunately, I've had bad experiences with these -- my ISP has been listed, causing my perfectly legitimate emails to be rejected by the recipient (and you can't send them an email to tell them of the problem!). So I don't recommend blacklist filters.

* Whitelists. A more useful spam filter is the "whitelist." This is a list of senders that you expect to receive email from, so it's accepted as non-spam. (Senders not on the whitelist aren't rejected; their email just gets extra scrutiny.) A nice variant on this is being used by some spam filtering services; when they receive an email from an unknown sender, they reply with instructions that a human being (not a robot) must follow, in order to authorize your email. Once a real human replies, the sender's address is added to the whitelist. This seems to be quite effective and is only a minor inconvenience for your correspondents.

You might be wondering, what's the difference between a whitelist and "filtering in"? If you're running the spam filter on your machine, perhaps none. But it's much more effective to filter your email at the mail server...which usually means at your ISP. The better ISPs are now offering spam filtering; so is Yahoo mail. Blocking it there means that you don't waste bandwidth downloading it (especially important with dial-up connections). It also means that someone else will take responsibility for keeping the spam rules updated. If your ISP doesn't offer this, there are also services you can subscribe to (for a price).

We use a combination of approaches. One of our ISPs offers filtering throught the Postini service, which uses a combination of rules, whitelist, and blacklist, and seems to be quite effective. I was truly impressed that Postini caught the SoBig virus within hours of its release...not everyone did. (On the other hand, we have to disable their "sexuallly explicit" rules or most ifeminist news doesn't arrive.) But Postini sells their services only to ISPs, not to individuals, so you'll have to ask your ISP if you want this service.

I've been using Mozilla 1.4 for email, which includes a Bayesian spam filter and a whitelist. The Bayesian filter actually does quite a good job...I'd say that it captures about 95% of the spam messages that I receive. The remaining 5% get dumped into my In box, where I can see from the subject line that they're spam, and a single click sends them to the Junk mailbox and updates the filter. I've been seeing about one "false positive" (legit email incorrectly tagged as spam) a week, so I still glance at the Junk before deleting it, but I think before long that will no longer be necessary. Even now it takes only a few minutes a day to deal with spam. Mozilla is available for Windows and Linux.

Other email programs with spam filters include Eudora (Windows only, paid version only) and Opera (Windows, Linux, and others). I believe Eudora uses a Bayesian filter; I don't know what Opera uses. For the more technically inclined, Paul Graham has a fairly comprehensive index of spam filtering software.

Here's the bad news: as people get more aggressive filtering their spam, spammers are going to get more aggressive sending it out. Hence the recent drastic increase. Don't wait for someone else to fix this for you. You have to take action to defend yourself.

brad