Interview with Hrvoje Dogan about spam

Hrvoje Dogan is a Systems Engineer for Eastern Europe and Russia at IronPort Systems, a Cisco business unit. Starting his career in IT at academic network environment some 15 years ago, he spent a significant amount of time in network research labs and snooping around various network operating systems. Moving on to the harsh commercial and banking environments, he stopped by as a pre-sales and support engineer at an IT Security solutions distributor, until finally settling at IronPort. Hrvoje’s vocational interests include IT security, networking and systems integration. As it is his mission to make things work, he believes in life-long learning and using every opportunity to broaden his horizons. In his free time, he tries to spend as much time possible to explore the wonderful undersea world. Hrvoje lives and works in Rijeka, a city on the Adriatic coast of Croatia. _

JB: Current internet “dark” community is no longer about prestige and coding (like writing viruses in 1990s). It’s all about business, there is commercial malware and so it is with spam. How do people make money on spam?_

HD: Basically, what we see is the whole “moneymaking” scene behind spam evolving along with the sophistication of the messages themselves. Right now, there is a whole multi-layer infrastructure, which works exactly the same as the “regular” global economy works. There are plants manufacturing (fake) products, such as pills and watches. It’s quite obvious - they make money from producing a cheap product in high quantities. Then, there is a network of different b2c sites that handle logistics and act as intermediaries between the producers and the spammers, often even running regular payment gateways - so they can accept credit card transactions on behalf of the spammers. These make money on commisions for each transaction (just like the real e-commerce gateways!). Next, there come the spammers themselves - they take most of the risk for legal prosecution, and thus make money from simply offering the product. To enable them to send spam, there is an elaborate network of botnets, providing sending and hosting infrastructure for spammers. And finally, there is a huge number of malware authors, writing different exploits, cracking tools, spyware agents and other malicious code, selling it to spammers. Of course, the malware authors themselves are also using the services of the payment gateway network mentioned earlier.

Unfortunately, this is a multi-layered model, where literally everyone in the chain can make enough money for a decent living.

JB: How many large spam groups are out there? Is the spam community interconnected or integrated somehow?

HD: It’s not so much about the spam groups themselves. The biggest problems are botnets. We know of several very large botnets in existence today, that are basically leasing their nodes to whoever has the money and the goal. The goal doesn’t necessarily need to be to send spam or sell something - it can be a DoS attack, it can be data analysis, or anything else that can be run on a large network of distributed systems. Often, different spammers will be sharing common infrastructure.

These guys cooperate - of course they do, after all they’re fighting a common enemy. It’s very similar to us on the other side - although there are many different anti-spam vendors and groups fighting for their market niche, we all cooperate and share information to achieve the common goal. Often you will also see them sharing same ISP space: Not too many providers are keen on hosting their infrastructure sites (C&C centers, content servers…), and those botnets that are not as smart as e.g. Storm (whose C&C center is using P2P protocol between infected node to form a gigantic, distributed C&C center), have to rely on a small number of ISPs wanting to have them.

There is also a whole bunch of community sites where participants of the spam economy hang out, exchange information, and offer their services - spammers, malware authors, botnet owners…

JB: The problem with e-mail is that we use old protocols (esp. SMTP) and do unauthenticated mail sending. Spam forced us to do lots of countermeasures like forcing reverse DNS entries to some more obscure measures like greylisting or hashcash. There has been some efforts to authenticate at least the sender’s domain using Sender Policy Framework and Domain Keys. How effective are these countermeasures?

HD: I’m glad you touched some technological blunders in this question :-) Actually, reverse DNS lookups, as well as greylisting, are one of the worst and least effective methods to stop spam. Unfortunately, when invented (especially greylisting), they were declared as the holy grail of spam combat. In reality, it took spammers probably a few hours of coding effort to adapt. Reverse DNS is bad simply because on today’s Internet, a host can (and SMTP gateway usually does) have multiple hostnames. Since MX records can’t point to a CNAME, you need to have several A and PTR records associated with each host. Now, the PTR records problem is where reverse DNS matching fails: If one host is presenting to come from (for example) 10 different domains (which it will do, because some people actually check if the connecting domain corresponds to the arguments of HELO/EHLO and MAIL FROM commands), this means it has 10 PTR records. Which, in turn means that you will either fail your match, or have to do up to 10 PTR lookups (depending on DNS implementation and probability) to match it. Imagine the delay this introduces. And finally, all you have established is that the spammer sending these spams is smart enough to choose a sending host with a correct DNS entry, and fakes his HELO and MAIL FROM in accordance to that.

Greylisting introduces unwanted delay in a similar way, however the delay is *much* longer (up to 4 hours at most default MTA installations!). The idea behind greylisting was that spammers usually try to send a spam mail once, and if it fails, they give up. So, if for every unknown sender, we reject the first message with a soft error, then spammers will give up, and regular senders will simply requeue the message and resend it later. There are two consequences of such approacHD: First, spammers figured it out, and changed the behavior of their sending hosts, so for every reject they get, they can retry at a later time (if the spam bot is still up), and spam will get through; And second - much graver - you will actually *delay* all valid e-mail from good senders, simply based on the fact that you haven’t heard from them before.

The right approach to solve the problems that greylisting and reverse DNS matching try to solve is to use technologies like SPF and DKIM. But, as every other new technology on the Internet, there is a big problem of adoption. Although these two complementary technologies exist, and although they are the right tool for the job, a lot of effort has to be spent now by the IETF and technical community to convince every single sender to publish SPF records and sign messages with DKIM, and to convince the receivers to actually verify those and discard non-matching messages. Most “big” players already use both technologies, but it’s the huge number of small senders that actually make up for most of spam traffic.

And again, SPF and DKIM only solve one part of the problem: They will establish authenticity of the message, and make it impossible for spam senders to fake identities. But if the message contains spammy content and matched SPF and DKIM settings - it will still make it to the recipient’s inbox. The positive consequences, though, will be that we will eliminate phishing, and sources of spam will be easier to identify and track down.

JB: There are list of “known spam” IP addresses, dynamic IP ranges and other “real time blacklists”. It’s a two-way weapon, ISPs often get to these lists, because their customers’ infected computers or even dedicated servers send out spam. Should the ISP be responsible for their customers’ use of e-mail protocols?

HD: This is an ongoing debate in the Internet community. The typical argument agains ISP responsibility is that the manufacturer of a revolver or a machine gun is not responsible for any murders commited with the weapon - after all, it takes somebody’s finger to pull the trigger. But weapons producers do take good care that they don’t sell their products to risk groups. Moreover, the malicious content on the Internet is usually sent through infected computers, where the owner of the infected PC has no idea, and especially no intention, to commit the crime. So, the finger that pulls the trigger, doesn’t do it out of its own free will.

Of course, ISPs should be forced into taking responsibility for content coming out of their network. One good move towards it is a widespread adoption of reputation services and blacklists. This sends a clear message from the Internet community to all of the irresponsible ISPs there. But again, there are too many service providers that just don’t care, and often e-mail receivers are adding them to whitelists - risking more spam from them - and contribute to solving the wrong problem. Reputation databases are rarely wrong.

_JB: Q: So providers basically have two options: they can sell a non-filtered pipe with and make customer responsible contractually, so when their server, network or computer starts sending out spam, regardless of reason (hack, deliberate sending), the customer is cut off. _

_Or they can add some active filter, which tries to block spam coming out from customers’ networks, thus increasing cost for them as an ISP and adding responsibility both ways: If the filter is wrong and blocks legitimate outgoing e-mail, they’ll be responsible to the customer, if the filter misses spam going out, they’ll be forced to solve it on network level. In this case, this added responsibility of detecting spam and filtering it out is added to the cost of connection. _

_It seems, that being ISP gets more difficult these days, it’s not just having a decent internet pipe, splitting it and selling parts of it to customers. There are compliance laws, that require logging, these filtering techniques for spam or outgoing DDoS attacks. _

_Is there something else that ISP can do? _ HD: Unfortunately, for the service providers, it does get more complex. Actually, ISPs have had it easy so far. If you just take a look at any other communal service (like electricity, gas, water… provisioning), each of those providers has to follow very strict rules on what to install at the customer’s premises, how to connect it to their distribution network, there are very strict parameters of product (water cleanliness, gass calorie value, power voltage and frequency…) they have to deliver. Now, same conditions are coming to the ISPs, except it’s a bit trickier, because we’re dealing with information content.

Regarding what SPs should do – both things you mentioned should be applied, but for different customers. Business customers could be given just a data pipe‚ and responsibility for compliance and “cleanliness” of their traffic is transferred to them. However, residential customers usually don’t have means or knowledge to control their traffic, so their traffic should be delivered clean – ideally with an option for residential users to “opt out” of the generic traffic cleansing, but then they take their own responsibility for their content. If ISPs think clearly, this is definitely a win-win situation: Their residential customers will be happy because they don’t get junk in their traffic, and the ISP can make additional revenue by providing “clean pipe” traffic to business customers who don’t want to bother with content security, and outsource it to the ISP.

JB: Anti-spam measures are quite complicated. If we look inside any modern anti-spam filter, we find use of databases like real time blacklists, distributed checksum databases, heavy use of statistic Bayes filters or even neural networks, most modern anti-spam systems even contain optical character recognition software for finding text embedded in pictures. With all these countermeasures, there’s a huge imbalance – sending spam massively is very easy, you need an infected computer and lots of e-mail addresses, which you can buy relatively cheaply online. On the other hand, blocking the spam requires huge computing capacity and thus cost on the mail server operator’s side. Yet, the spammers seem to be always one step ahead, or at least on the same track as anti-spam solutions. How do we turn this? Is there a way to make spamming difficult and fighting spam easier?

HD: Yes, of course there is. It has been described a few times in this article already: Widespread DKIM and SPF adoption, providers taking responsibility of their traffic, networks installing holistic security solutions to monitor for infections. Also, there are non-technological measures like working for a common legislation for computer security breaches, increasing comunity awareness… The whole point is that sending spam should be made *unprofitable* to the spammers. That is when fighting it will become easier, and it actually has to be done by other means than increasing the sophistication of anti-spam engines. It has to target the spammers’ infrastructure, not the result.

JB: Spam is no longer about e-mail, we often see spamming of forums, these times even social networks and instant messaging is target of spammers. It has the same problem – spamming is easy, fighting gets difficult. Captcha pictures have been a popular choice by forum administrators, while quite a burden for people, who actually wanted to post to these forums. These days, computers are better at solving Captchas than people themselves (for example can break most of todays Captchas). How do we fight web spam?

HD: This is a direct consequence of evolution of anti-spam technology for e-mail. Pushing spam through e-mail is becoming increasingly harder, and spammers have slowed down the efforts to fight the technology, but have turned to different battlefields. Web 2.0 and all of the different social networking sites didn’t make their work harder. What they introduced is a functional equivalent of e-mail - but with no anti-spam technology implemented! It was a dream come true for spammers. With a small investment in technology (CAPTCHA breakers and mass account creation tools), they can have a huge infrastructure to send spam - and no engines to block it. They basically have the situation they had with SMTP 15 years ago, and 15 years of spamming technology development experience. This is a fact that none of the Web content producers were aware of. Even worse is that with the development of Web spamming technology, there was a collateral victim of enabling spammers to mass-open accounts on different (Web-based) e-mail providers like yahoo, gmail and hotmail, and send out e-mail spam via Webmail. Only, this time these spam messages are coming from valid, very strong e-mail providers. By doing this, they are undermining the key principle of fighting spam today: Reputation. Using different reputation mechanisms (from simple DNS blacklists to highly advanced reputation networks like IronPort’s SenderBase), receivers can cut off 50-95% of unwanted mail at the connection level. High volumes of spam from well-known sites bring the efficacy of reputation filtering down, and false positives rate up.

JB: Fighting spam on technological grounds is only the tip of the iceberg. We should try to find and prosecute spammers themselves, discover the flow of money…

HD: Of course. Unfortunately, this is very hard to do. Internet is a collection of unrelated networks all around the world, and there is no common law platform to fight bad content. Spammers hide their tracks and operate from countries that don’t prosecute them, or at those that do, they find holes in the law, and hide behind them. Since spam e-commerce chain today includes several layers, even if you manage to track down some of it, sooner or later you will run into authorities in some faraway country that are simply not willing to cooperate.

Sometimes the authorities want to listen, and that is when things like shutting down of McColo happen. What we should do is increase awareness, and make internet accountable. If ISPs and enterprises took care of their security, if governments passed legislation to prohibit illegal advertising, scam marketing and fraud, and if sites providing support to illegal activities would be shut down on the regular basis, the bad guys would lose their foothold.

_ JB: One of a common DDoS technique we’ve seen in Slovakia is, that someone starts sending spam on behalf of attacked domain, which effectively takes down either the mail server or ISP’s pipe, because of the returning delivery error messages. Added to that, it also ruins the domain owner’s reputation, the world seems to start thinking, that they are sending the spam themselves, because it is difficult to explain to common people, that sender domain can be faked. _

_Of course SPF can help, but there’s a problem you mentioned previously: several smaller mailservers do not support or enforce SPF, or the SPF verification takes place after target e-mail address verification (which effectively sends out delivery error because of unknown user at the target domain sooner than SPF verification takes place, because it’s more effective to refuse the mail early). Any thoughts on this subject? _

HD: Actually, it’s great you mentioned this. This is a two-fold problem. The first part of the problem relates to a DoS condition caused by returning delivery errors (these are called “misdirected bounces”). The solution for the first part has been in existence for a while! It’s called Bounce Address Tag Validation (BATV), and has been published as an Internet Draft ( which expired Nov 2008, but the new status is not yet known. I’m happy to say that IronPort was among the first vendors to implement the modified BATV scheme, and all of our customers are protected by misdirected bounces. Unfortunately, there are some bogus anti-spam techniques that break BATV, and there are some MTAs that break the SMTP RFC and will reject BATV-tagged messages, however these problems are fairly easy to overcome.

The second part of the initial problem has to do with sender reputation. First of all, it is easily solved with SPF and DKIM. Even for the hosts that don’t support SPF or DKIM verification, there will be no issue with reputation decrease, as SMTP reputation databases are based on the actual sender, and not the information in the SMTP envelope or headers. Any anti-spam solution that calculates reputation of SMTP senders vs. reputation of IP addresses sending is just doing it wrong. Once more unfortunately, if somebody chooses not to support SPF and/or DKIM, it’s their own choice not to protect themselves. There is not too much we can do, other than spread awareness of the problem, and of the solution(s). So far, the problem awareness part is doing fine, as the problem is really visible. We have to try harder to promote the (correct!) solutions.


Written by Juraj Bednár //