Email System Changes, June 22 2011

(This is a copy of an email that got sent out to all market managers on June 22, 2011)

Hello! This is Eric Wagoner from locallygrown.net.

This letter is to tell you about some major changes to the system, mostly related to the handling of email, and mostly behind the scenes. There are also some new features, including perhaps the most common request I've received over the years, so read down through the bottom for that.

First off, what prompted the changes. Email is a tricky thing. About 95% of all the email sent through the world is junk mail. If your email providers didn't try to filter out all that junk, email as a whole would be totally unusable. Think about all the junk mail you do get, and keep in mind that for every one that gets through to you, a hundred or more were blocked by your provider. They continue to get more strict, they talk to each other, and they're just looking for any reason at all to block emails from getting to you. There are many things they are looking at, but there were three in particular that had become trouble for locallygrown.net emails.

1. Recipients flag email as junk. This is the biggest, and it was happening a lot. Your customers would find your market, create an account, and then when you would send out your market's newsletter, they'd file a complaint with their email provider. Even more puzzling to me, sometimes even your growers would file complaints about your mails. This just drove me crazy, as their complaints would lead to hundreds or thousands of locallygrown accounts not receiving emails they wanted. In the last two weeks, three major providers started blocking all of our emails from all of our markets.

2. Emails bounce because the customer changed email addresses but never changed their locallygrown.net account, or they mistyped their address to begin with. When you send an email out to a non-existent address, the mail servers on the other end send back a notice stating that address doesn't exist. If you then continue to send emails to that address, finally the mail servers will stop replying and just block you. This was starting to happen.

3. When emails get sent from one mail server, but have a "from" address of another server, those emails can look suspicious. Generally that alone's not enough to cause a block, but in conjunction with the points above, it can make them more likely to pull that blocking trigger.

All of our markets have grown to the point where my mail servers are sending out about 10,000 emails a day (sometimes twice that). When a large provider like AT&T or Hotmail starts blocking us, that's a lot of emails that simply go missing. And that was starting to happen more and more, so I had to make some drastic changes. What I ended up with is so much better than before, in ways I'll describe in just a moment, but it was a vary tough thing to do. It was made more the tougher because I had to do it without taking the whole system offline, and without losing any of your emails. I completed it yesterday, and all went relatively well. The only glitch was for a couple hours late Sunday night, emails were running about an hour behind as many of the largest markets all sent out their newsletters at once. I've scaled up that architecture, so I don't expect that to happen again.

Here are the changes:

1. I've set up a stand-alone mail server that only handles email. Not only should this make it easier to maintain going forward, but it also unloads work from the web servers, making the site as a whole snappier.

2. Before, newsletters were sent out as a single email, with all of the recipients added as "blind carbon copies". Now, everyone gets their own copy, with their address in the "To:" field. This makes email providers much happier.

3. I can now monitor junk mail complaints (when the providers share that information, and most do). When one of your customers or growers files a complaint about one of your emails, their account will immediately get marked as inactive. They will no longer be able to place orders, and they will no longer receive your emails. Email providers treat this very seriously, and their complaint endangers the entire system, so there are no second chances. Well, that's not quite true. You can go into their account and mark it as active again, but they will not be able to do this themselves.

4. Almost every email going out now has an "Unsubscribe from this market" link at the very bottom of it. If the recipient clicks that link, and then confirms they want to unsubscribe, their account will again get marked as inactive. They will no longer be able to place orders, and they will not receive any of your emails. Again, only you can reverse that for them. This is similar to a junk mail complaint, but don't go through the recipient's email provider and thus don't count as a black mark against us.

5. I can now monitor bounced email addresses. When an email bounces back (usually because the address doesn't exist, but sometimes because we're being blocked), then that person's locallygrown account will be set to not receive emails. The person can fix the problem on their own, by correcting their account's email address and clicking the link that they'd like to start receiving emails again. They can do both of those on the "Your Account" page.

6. I have a secondary backstop on the new mail server that duplicates items 3, 4, and 5 above. I'll clear that out at least once a week. That does mean that if a person files a complaint against your emails, but then you go in and re-activate their account, it may still be a week or so become they start receiving emails again. This "cooling off" period is to protect all the rest of us.

7. I'm being more explicit in how the "From:" fields are being set, to make it clear to the providers that what we're doing is legitimate. Most people will never notice, but some email programs (most notably Microsoft Exchange) will start displaying things like "From: locallygrown.net On Behalf Of manager@yourmarket.com" (fill in the addresses with what you are actually using). They may see it, but they can still reply and it'll go to you and all of that. It just looks odd when you're seeing it for the first time.

All of these changes are fantastic. It'll be harder for providers to block us, and I'll know right away if any do. Customers are in more control of what mailings they get. The system runs faster. And so on. It was really expensive, though. In one weekend, I committed at least $5000/year to gets the servers and all the associated services that go with it. The expense is volume based, so the more emails go out, the more it'll cost. So that brings to the last item:

1. You can now get your very own locallygrown.net email address!

I've been asked for this by just about every market that's started up, and now I can finally say "yes". In fact, it's all automated, and you can start using it now. To do so, if you choose, go to the Your Account page of your market. You'll see a big link at the top of the Market Manager section called "New Email Features". That'll take you to the "email" section of the "edit your market's information" form.

If you want to start using your locallygrown.net email address for all of your mailings, just check the first box. If your market's web address is yourmarket.locallygrown.net, then your email address will be yourmarket@locallygrown.net. Please not, this is an optional feature with an additional cost. It's only an additional 0.15%, and that will get added to your software fee. If your market does $1000/week in sales, this would cost you $1.50. It's not much, but if most of you choose to use it, then that'll pretty much pay for the new infrastructure.

The default is any mail sent to that address and any replies to the mail you send out will get forwarded to all of your market managers. This is probably what most of you will want to do, but if you already have a single email address for your market you are using and want your @locallygrown.net mail to get sent only to that one address, check the next box and fill in the custom email blank with the address you want to use. That field is already filled in with the address you are currently using for the market, so if you want all mail to get forwarded there, just check the box.

Finally, if you don't want to use an @locallygrown.net address, and want to keep things exactly as they are, don't do anything. This new feature is opt-in so you're not going to suddenly get changed for it unless you want to start using it.

And that's pretty much it! Many of you are only now figuring out that emails from several weeks ago were getting blocked. I've managed to get all the major blocks lifted, but there are still a few individual small domains that are blocking us. I can now easily check that, so if ask me if a customer suspects they're not getting mail. Here are the troubleshooting steps to take:

1. Make sure that the customer's account is set to receive emails. If that box is not checked, then that means their address is bad, or their provider is blocking us. The customer can correct their address and reset it themselves, or you can do it for them. In about a week, they should start getting emails again.

2. If their account is set to inactive, then either they unsubscribed or they files a complaint against us. Only market managers can reset the account as active again, and again it'll take about a week before we'll try to sent them email again.

3. If their account is active and set to receive emails, then we are sending them out and haven't received any bounce messages back. They should check their junk mail folder to be sure they're not stuck there.

4. The new mail server is logging the heck out of everything, so I can check the status of any address, see what was sent and when came back. If you have any questions, let me know and I can tell you what the server thinks is going on.

And of course, if you have any questions at all, don't hesitate to ask. This was one of the biggest infrastructure changes I've had to make since I opened the service in 2007, and I'm happy to report that it went about as good as it could have. I haven't had as much sleep as usual in the last few days, but I'll be able to sleep better from here on out :)

Thanks!
-eric