- Image via Wikipedia
Many clients expect email to be sort of an automatic thing. The thinking is “I should hit ‘Send’ and it should arrive within moments.”
So, sometimes those of us who work in email hear comments like the following:
I understand that these delays may not be your fault, but we can’t have delays.
and:
It’s just not acceptable that they may rate limit us due to their own capacity issues.
With it now being “Cyber Monday” and with clients doing their best to ramp up holiday sales, we are are to the time of the year when these kinds of comments become more common. So common, in fact, that I’m going to head things off early this year by giving you the hard truth about email.
Email is like anything else on the Internet: It is a finite resource. How do you know when your mail campaign is going a little too well? Perhaps when the clicks pour in so quickly that your IT staff reports that the web server is starting to shake just a little bit? Or maybe when you hear the IT staff talk about the need to purchase extra capacity and bandwidth at the holiday planning meeting? Well, that need is there because there are finite resources available to handle those demands.
When we send email, we tend to consider it a “fire and forget” mechanism. “I hit ‘send’ and the message arrives. It just works!” is how the thought goes. But, the same thing is true for your webserver. Your customers put in your URL and they expect to see your website. But that doesn’t always happen, does it? It does happen often enough that people expect it to work without giving any thought to the fact that it may not. That’s amazing reliability, but discounts such terrible things as coding errors that got put into production without QA, denial of service attacks, the failure of aging equipment, or that everyone in the world wants to get to your website RIGHT NOW!
And the same is true for email. ISPs and other receivers build their data centers with the needs of their customers in mind, and you are usually not their customer. When every relative your customer has wants to send mail inviting them to Christmas dinner (all at the same time) and you and every other marketer in the world want to send them mail inviting them to shop at your all-important sale (all at the same time), it overloads the available capacity of your customer’s ISP. While they do build their systems (more or less) robustly and with many normal, seasonal surges in mind, you have to understand that sometimes something happens that means that your email will not arrive instantaneously.
Sometimes email gets held up because:
- a hard drive on an aging server has finally given up the ghost
- some botnet herder in Eastern Europe has decided that they hate the ISP’s anti-spam systems so much that they will retaliate by sending the ISP 100 million messages per hour
- some intermediate link between your (ESP’s) mailserver and the ISP has gone dead
- that new filter that they’re using didn’t undergo enough QA and has either
- caused the server to lock up tighter than 8 day clock or
- caused all incoming mail to be delayed, filtered, or blocked
They way to deal with these issues is the same: spend more money. They can spend money on new servers, more bandwidth, and/or more personnel to code, do QA, and/or monitor and run the equipment. And, unlike WalMart, they really can’t do seasonal hires or 90 day equipment leases for this. It takes a long time to become proficient in how an existing setup works, and setting up a new server is not usually something that anyone relishes doing during the busy holiday period.
Email is a store-and-forward mechanism that can deal with lots of things. But, the hard truth about email is that the system cannot deal with everything so sometimes an email won’t go through at all, and sometimes it takes longer to get there. And unless you want to spend the money to buy an ISP more personnel or equipment, it’s not likely to ever change.
[...] This post was mentioned on Twitter by LashBack, Spamtacular.com. Spamtacular.com said: New blog post: The hard truth about email http://bit.ly/8RzNLB [...]
Social comments and analytics for this post…
This post was mentioned on Twitter by spamtacularcom: New blog post: The hard truth about email http://bit.ly/8RzNLB...
Mick,
Great post and comments I read referred from a linkedin.com article by e-mail marketing consultant John Caldwell. The more important question here is what have any anti-spam product or technology leaders done to lower spam levels or drive spammers away (besides Idalis Software's Anti-Spammer product but this IS NOT self promotional, read on). Being a former spammer and having driven spam levels down with our IP and URL reputation products (cut off the sending/money spammers need to survive), I see no support in the industry for this because high spam levels are justifying the sale of additional appliances, hosted SaaS services, and of course the back-end loss management to handle missed malicious messages and time spent sorting through pending authorization quarantines. These missed messages also grow botnets which contribute to the problem but that is an entire different discussion altogether.
Technology is supposed to make our life easier and more secure, so why are all the approaches doing nothing to 'spammers'? By combining 100 percent guaranteed organization shut downs with non-probability blocking, e-forensics, and law enforcement spammers can be taken down (see the recent 4 year prison sentencing of Alan Ralsky, who quit spamming our networks last year because he lost his entire class B CIDR within a couple days of identifying him.) Enough tooting my own horn…but why hasn't a deterrent or source fix of the problem approach been spread, rather than the self-justified fighting of the growing problems symptoms?
It appears to me that anti-spam and e-mail is almost a racket. I challenge responses to refute what I've proven since it has and can be performed, to dozens or hundreds of spam gangs. We have statistics, testimony, and e-forensic evidence to prove so. Now don't get me wrong, 10 to maybe 20 percent of spam won't completely ever go away, but losses, costs, and the majority of the problem can and as you stated the problem is in the growth of the spam levels and unwillingness to scale to accommodate or lower the problem itself.
Tom Caldwell
Idalis Software
http://www.idalissoftware.com
Was that a comment, or a press release?
Al,
If directed towards me, it was a comment and question in regards to high spam levels, lack of focus on spammers causing high spam levels, and I mentioned it wasn't a product pitch or solicitation in the second sentence but can be performed (proving a point then asking why it isn't the focus to solving the problem Mick pointed out).
FYI, I just found this blog/resource, it's great for anti-spammers, e-forensics, and of course marketing consultants, who should also be protected.
I hope that clears up any shameless plug accusations?
Thanks Al/Readers,
Tom
Tom, you're right that email marketers and deliverabilty wonks aren't doing anything about botnets, but they're far from the only players. There's a long list of "primary sources" links along the left side of boxofmeat.net which may provide some helpful information on what the rest of the industry is doing.
@Tom: Nope. Saying it wasn’t a product pitch doesn’t make it not a product pitch. A spammer that says his mail isn’t spam doesn’t magically make it not-spam. You could have made your point without mentioning your product.
Where you get the idea that ISPs are doing nothing about spam, and that they’re doing it because they want to spend money (what money??) on new toys is a puzzlement. It certainly doesn’t jibe with my experience in the industry.
Senders can't do anything about botnets except Don't Use Them. Now, there are some fairly large brands that are outsourcing their email acquisition programs far enough that mail advertising their products is sent using botnets. However, if it doesn't go over network I control I can't do a whole lot about it.
Tracking spammers, IDing botnets, IDing command and control, linking real world IDs with bot herders… that's some fairly specialized work. In some cases it requires international law enforcement involvement. The ISPs can't do this because it's often out of their jurisdiction and the costs are extensive. Law enforcement can't always do it for the same reason.
Look, many of the botnets, while infecting US computers, are actually run by criminal gangs in foreign countries responsible for botnets. It's neither easy nor cheap to track them down. It's also not always easy to make them stop just by suing them or arresting them. Look at Al Ralsky, he's lost court cases, he's been convicted of fraud previously, and yet he was still able to spam with impunity for almost 10 years. Same thing with Sanford Wallace. He was one of the first spammers sued by an ISP (ELN, back in the late 90's), and just lost a court case a couple months ago brought by Facebook.
Identifying and suing spammers isn't as effective as you make it sound like it is.
@J.D. I was going to quote some numbers from the Guys at Return Path regarding the 7 percent of spam coming from ESP's that give out free accounts however I found out that you might know them. Larry Ellis came over and was my POC. I pinged you off-Spamtacular so we can chat all things processed ham elsewhere and post them here if worthy.
@Annalivia, no I had to prove it can be done, and it is being done, because it is a different approach. Furthermore where did I ever mention that ISP's are doing nothing? I specifically stated the "anti-spam" vendors, NOT ISP's or even ESP's are doing very little to "SPAMMERS", not spam, because you have people behind one and digital electrons in a certain encoding format in another. . BIG DIFFERENCE between people and electrons, right. Let's be honest, read the latest "ISP's EAT YOUR SPAM POST" . It confirms everything to my point. In the future please read the post, and don't put out of context words into my mouth as you wouldn't want me to do that to you. I've worked for two ISP/ESP's and consulted with 4 more in the 500-1 million user range, so I know the environment pretty well.
@Been there done that, *sarcasm coming* yes it isn't technologically easy to mine through billions of e-mails, blacklist all the URLs, use those URLs to track botnets or spam supporting networks, attribute the URL and thousands of others to the same person, host , redirect, click-through target, then capture all of that data and attribute the bots being used to the URL's being sent and who they belong to (business intelligence and data mining with some simple whois and data caputre, even when forged). Nope totally impossible to do that…I mean who'd ever think up such a thing.
All of this is in a forensic database which FBI/DoJ can easily log into and use the spam sample, forensic testimony, screen-shots of the sites with context matched to the malicious message and then a subpoena of course to search the credit card owners premises, their financial records, and who they've been paying for botnet time and spam sale commissions.
I WORKED IN A SPAM RING, THAT'S EXACTLY HOW IT WORKS, even when multiple parties are doing the spamming, botnet harvesting, domain registration, hosting, and laundering the payment methods through another shell company to even make it look legit.
So if you have the data, system to easily match it, and attribute it to a person seamlessly plus your Deputy Assistant Director in IC at FBI can convince Congress to fund going after these people then YES, it can all be done and will be done. However if you ruin the spammers, no one in the industry will want to support you because they thrive on more spam being sent, it's how they're making money. What would happen to police if all crime dropped by 80 percent?
You were unclear in your writing, then. I will not put words in your mouth if they are clear on the page. You said "technology leaders" were doing nothing and from where I sit, that's primarily the ISPs. I know the one I work for has developed an amazing array of tech to deal with the problem and we have been considered industry leaders in the anti-spam world for quite some time. You specifically stated exactly nothing, and scolding me like a delinquent child for misreading your misleading and unclear writing does not behoove you. In the future, please write clearly. Pursuant to that – what does "non-probability blocking" mean? and how does one achieve "100 percent guaranteed organization shut down"? If all this data is at the fingertips of the Feds, why arent they using it?
Annalivia Ford from AOL has a response up to a comment made on this blog yesterday. It’s a great read! Go check it [...]
[...] about everything the ISPs have to consider when making hardware and buildout decisions in his post The hard truth about email on Spamtacular. When, like on cyber Monday, there’s a sharp increase in the volume of email, [...]
[...] Ford who wrote that fantastic poem from last week, Laura Atkins from Word to the Wise, and Mickey Chandler who is the go-to for all things related to spam law all talk more about this on their own [...]
[...] The hard truth about email. This post, which comes in forth on the top 10 list for page views comes in first for the number of comments (and we love comments!). It’s a reminder for senders that the ISPs’ resources are limited and tend to get overloaded during the holiday season. [...]