We recently revamped our website. Our old one was a wrap-around to an actual site, so whenever they would have issues it would negatively impact our ability to sell. So we decided (a little late BTW) to finally pay some people to design a new site. It looks good now, and the search functions work the way they’re supposed to. AND if we ever have any issues, we can just fix it ourselves and not have to wait for another site’s IT force.

Unfortunately, the decision makers decided that we should send a mass email out to all of our customers telling them of our new site and how to log in. I mentioned that since we have hundreds of thousands of contact email addresses that maybe we should send them out a few at a time instead of all at once.

They agreed and wrote a program that will send emails out at a rate of 60 a minute (1/second). Preliminary testing showed that everyone received the primary “welcome” email along with a duplicate. So every Welcome To Our Site showed up with another Welcome To Our Site. Yay! Good thing we tested in-house.

Fixed that issue, tested again. Good to go. Tested on 5000 customers – test went just fine. But then, just today, they tried to send to another 5000 customers.

I say “tried” because we succeeded a little too much. A fellow employee mentioned to me that he received several of the Welcome emails. I said that’s probably because he has several accounts. But after he received his 8th email I knew something was up.

Log into the SPAM scanner and what do I see? A queue that’s usually less than 50 emails is over 8000. Ut oh.

Looks like one of the servers is sending out 8, 10, up to 14 emails out to the same customers.

I quickly tell the web guys to turn off the emailing service, but the queue is already full. So I then block IP access and issue NDRs to the remaining emails. Unfortunately the entire subnet was in the whitelist, so I had to remove that and readd the actual mail server.

All in all it looks like roughly 1200 email accounts were sent a total of 170 thousand emails. I blocked all but 15000 of them. So 12.5 emails each. Lucky for us it looks like almost 25% of them are no longer working email addresses. Even still, we received quite a few phone calls and emails saying people were never going to do business with us again. Yippie.