[Previous entry: "DailyKos blogawarms Greenspan"] [Main Index] [Next entry: "What you'll wish you'd known"]
03/06/2005 Archived Entry: "Sender Policy Fuc---er, ahem, Framework"
Bad news on the email front. Our local ISP has said that they're going to implement Sender Policy Framework (SPF) to block spam. Unfortunately, this is likely to block a large amount of our legitimate email as well. I'm already devising contingency plans.
SPF will not be as bad as I had feared. I got it wrong last year when I described how SPF will work. Well, I got one thing wrong, anyway, but it's a crucial distinction: the difference between the "From:" header and the "Mail From" address.
When you send an email, your return address is identified two ways. When you compose your email, you provide a "From:" line. (Most likely your email program does this for you automatically.) This is part of the "header" of your message, and while it looks all technical and official and stuff, it's really just part of the text of your message. (The email program at the far end will simply extract this bit of text and display it in a "From" box.) You can think of these headers like the "letterhead" of a paper letter, that has your address and maybe phone number and other information. Most of this is under your control.
But when you send the message, it's going to go out through an SMTP (Simple Mail Transfer Protocol) server, to another SMTP server for the intended recipient. SMTP servers don't read email headers, but they do need to know where the email came from, in case they need to send a bounce message. So in the SMTP protocol, the MAIL command (to transmit a message) includes a FROM location. This is supplied by your SMTP server and usually you have no control over it. In an email message it will show as part of the "return path".
You can think of the MAIL FROM address as the return address on an envelope. The addresses on the envelope, not the addresses typed in the letter, determine where the letter will be sent and where undeliverable letters will be returned. For this reason the MAIL FROM address is often referred to as the "envelope sender".
Remember, the "From:" that you see is part of the letterhead. Now, it would be perfectly possible for me to fake up a piece of IBM letterhead, showing myself as president, but then mail it to you in a plain envelope that with "P.O. Box 5, Snowball, Ontario" as a return address. The same is true of email.
(In case you're wondering, by the way, this is how BCC's are sent by email. The BCC recipient's address is left off the "letter," and only put on the "envelope." Most mailing lists work this way too.)
Which brings us back to SPF.
SPF -- and here is where I went wrong before -- doesn't care squat about your From: address (your "letterhead" address, so to speak). SPF exists in order to verify your envelope address. It wants to be sure that a message claiming to come from smtp.yahoo.com really does come from smtp.yahoo.com....because spammers usually forge this information to make their messages harder to trace. So when an SPF system receives a mail with a MAIL FROM address (envelope sender) at yahoo.com, it makes note of the numerical IP address of the sender, and then sends a query to yahoo.com saying "is this one of your email servers?"
This is all transparent to you, and unless you are actually running an SMTP server, you shouldn't have to worry about it. Your ISP or your web hosting service (if you use their SMTP server) should take care of the SPF verification. If you're an "ordinary" Internet user, you should never have to deal with this.
Except for two eensy little problems.
1. What if your ISP or web host hasn't implemented their half of the SPF service? (The verification part, where they confirm that yes, the email really did come from their server.) If your recipients start rigorously enforcing SPF -- and remember, this is up to their ISPs or web hosts, not them -- then suddenly your email messages will be refused, because their ISP can't verify your SPF data. (You should receive a bounce message...but you might not, depending on how careful the implementers are.)
2. What if you forward your email? To stretch the analogy a bit further, forwarding email is like slapping a new To: address label on the outside of the envelope and putting it back in the mailbox. The problem is, the SMTP server that's doing the forwarding (putting it back in the mail) does not match the return address on the envelope. SPF will reject forwarded email.
(You might ask, why not change the return address too? The problem is, then, if the email doesn't get delivered, it gets returned to the forwarding service instead of the sender...and the forwarding service no longer has the sender's address.)
The SPF proponents have been hammered with this question over and over during the development of SPF, and now they'll simply tell you flat: SPF breaks forwarding. They don't care. (I think that most of them control their own mail servers, so this will never be an issue for them. What this group needed was a few more "ordinary users" on the design team. Too late now.)
There are some attempts being made to develop new mail forwarding schemes which work with SPF. This will require every web host that offers mail forwarding to replace their software, so expect a lot of confusion during the transition. Some web hosts simply won't bother.
From what I'm hearing, SPF is going to be "rolled out" by more and more ISPs this year. The bottom line for you is twofold:
1. Expect email to become unreliable (or more unreliable) this year. You may suddenly find that people you've been emailing for years suddenly can't receive your email. It would be prudent to establish a backup account on one of the webmail services like Yahoo -- who are likely to support SPF early -- so that you have another way to send email to "problem recipients".
2. If you're forwarding several different email addresses to your one email account, check with your ISP (or web host) to see if they'll be implementing SPF. If so, then you can expect your mail forwarding to simply stop some day, with no warning. (And you won't get a message when this happens -- email will just stop arriving.) You'll need to make other arrangements for your forwarded addresses. I'll write more about this later.
So the brain-damaged SPF, while not as bad as I had feared, is definitely coming. The sad thing is, it will do little to reduce the flow of spam. Every "zombie PC" that's been captured by a virus can pump out spam emails all day long, which SPF won't catch.
Plan now for the disruption.
brad
P.S. Looking back at my earlier blog on the subject, I see that I was wrong about problems #2, #4, and #5, due to my confusion of From: and MAIL FROM addresses. #1 is still correct. #3 is still correct but now largely irrelevant. #6 is still relevant, but fortunately this was spiked by the Internet standards committees when they refused to adopt a proposal that had restricive "intellectual property" requirements.