“The Era Of Distributed, Independent Email Servers Is Over”

Imagine the Internet had begun its life as a proprietary network from a major software vendor rather than evolved as a distributed network shared by researchers. It’s a future that almost came to pass for consumers in the 1990s when walled gardens such as AOL or the original incarnation of MSN were all the rage, but thankfully the world took the Internet course.

Though there are many continuing threats to Internet freedom we can still mostly use the network our way, but with sadness we note that one piece of Internet freedom may have drawn to a close. [Carlos Fenollosa] has written a lament about how the outlook for anyone running their own mail server now looks bleak.

At its heart is spam, or indeed the heavy-handed measures taken by large email providers to combat it. Spotting and canning spam is computationally expensive, so the easiest way to stop a spammer is to recognize their activity and block it at the network level. Thus a large email provider will instantly block large IP ranges when it detects they hold a spammer, with the collateral damage of also blocking any legitimate email servers in the same range such that their mail just doesn’t get through. Since spam is such a widespread problem, as [Carlos] points out it’s less of a case of if your server has this problem, but when. This functions essentially as something of a racket, in which large email providers have the power to ensure that any email not generated from amongst themselves is unlikely to reach any of the millions of addresses under their care, and the only recourse an operator of a small email domain has is to use the services of one of them.

He has something of a manifesto as to how this problem can be addressed, and we think that it’s important enough that you should take a look. Maintaining email as something beyond the control of large providers is too important not to.

Thanks [Thomas Steen Rasmussen] for the tip.

Header image: RRZE, CC BY-SA 3.0.

138 thoughts on ““The Era Of Distributed, Independent Email Servers Is Over”

  1. The article’s author has a very narrow view of the email ecosystem and his experience really isn’t sufficient for such generalisations.

    Email is older than the internet, some problems stem from that, but the rest is a result of natural evolution of the ecosystem. It’s not a racket, it’s that distributed trust is just incredibly difficult. Most proposals to “solve spam” unfortunately fall short (short template for testing anyone’s proposal https://craphound.com/spamsolutions.txt).

    Self-hosting email has never been easier, but also at the same time never required one to be this careful at establishing **and** maintaining trust. The big providers give you tools and information about what they check, it really is the small players that are the most finicky, unpredictable and opaque. There are many small, obsolete and low-quality blacklists (like UCEProtect or Backscatterer) that only small players use.

    We shouldn’t ring the death knell without reason, like this now.

    This has also been discussed in-depth on Hacker News https://news.ycombinator.com/item?id=32720234 / https://news.ycombinator.com/item?id=32715437 / https://news.ycombinator.com/item?id=32722240

      1. You may be something of an outlier. After a half dozen frustrating experiences with various providers, including running my own twice, I noticed that my yahoo.com and gmail.com addressed never had unexpected problems or required frequent maintenance to maintain interoperability with basically every other email service out there. So, I stopped fighting it and haven’t looked back. This was a little over 20 years ago and I don’t believe the situation has improved much in the meantime. As they say: YMMV.

        1. You’ve simply outsourced the maintenance to a third party. Yahoo or Gmail aren’t basically immune. Me I pay for email hosting from a small provider run by a 2 person company, and have for over two decades and the only problem has been a very legit mistake of a www Blocklist accidentally being applied to the wrong server.

          1. Yes, Google are immune. If your mail blocks Gmail, because they don’t fit your security standards, it will be considered a bug and you’ll have to fix it or lose all users.
            As such they make the rules. And you have to obey them. Of course they have a huge vested interest in making it as difficult as possible so you pay for your business to use one of the big mail services.

        2. 100% agree. I left the self hosting email server service years ago. If you did get blacklisted it was a huge time sink to get it fixed. It’s unfortunate but it’s a fact of life in the world of email.

          1. In over 20 years of running my own mail server (which handles not only my personal email, but also sends notifications from some websites hosted on the same server), I’ve only been snagged once by a blocklist, and that was resolved in maybe a couple of days by having Linode contact Microsoft to get my site unblocked. (I only learned about it when I heard that a friend’s @outlook.com (or whatever) email address wasn’t receiving any email from me.)

            The OP’s fears are overblown.

      1. That song has been sung over and over for 20 years or more. There must be something about email that keeps it the universal common channel while all the “innovative” would be replacements struggle but never achieve the same market penetration.

    1. Spam will die as soon as the world agrees to a mechanism to just charge users a small fee to send email, like we buy a stamp to send a letter by regular postal mail. It doesn’t have to be a large fee to make it prohibitively expensive for spammers. Just a thought.

      1. I’d think it only would take a tiny fraction of a cent ($0.0001, say, maybe much less) per message to dissuade mass spammers. I’d gladly pay what that would cost me per year to be rid of even a substantial portion of the burden of spam messages. Hell, I’d even be willing to pay it on legitimate incoming messages, but that opens up all kinds of issues.

      2. Completely agree with this concept. Has to be expensive enough to stop many thousands of emails to be sent yet cheap enough that it doesn’t materially affect the typical user.

      3. Proof of work, perhaps? My Postfix server is set up to generate Hashcash for all outbound email. It takes maybe a second for each mail, and the work could be done at either the client level or on the server. Other mail servers can validate the stamp quickly as part of their anti-spam filtering. I have a low-volume notification system with maybe a few dozen subscribers; Hashcash doesn’t slow it down appreciably. 1 message per second would be unacceptable to spammers, however, so they’d never adopt it.

        http://www.hashcash.org

      4. Now imagine after having cuppa coffee in Stbks you got your credentials leaked and then a massive and thunder-like spam attack done from your email, and you gotta pay for it in this case, so it won’t kill the spam, it will just make more people be a victim of email credentials theft.

        1. What you say is true if the system charged back to the email address owner. That approach has several issues. I think the better way to do it is at the smtp server where messages enter the delivery network. We generally already have a business relationship with our smtp server through our ISP, so the traffic can be clocked there. What do you think?

        2. What you say is true if the fee is charged back to the email address owner. That has several problems as you point out. However, suppose the fee was charged at the smtp server, where traffic can be clocked where it is presented to the network for delivery. The entity logging into the SMTP server would pay, not the email address holder themselves. In most cases, the business relationship already exists with your ISP to use their SMTP server.

  2. My domain never once sent spam, but large ‘free’ email service providers rank it’s reputation as poor. There’s nothing I can do to prevent them flagging my outgoing mail as spam. I think that’s code for ‘we own your information, have monopolised access to it, and will mine it to our ends, and heart’s content’. Go read your Gmail TOS.

          1. From the article: “I didn’t clarify this at first because I didn’t want this article to turn into an instruction manual. This is what I implemented: DKIM, DMARC, SPF, reverse DNS lookup, SSL in transport, PTR record. I enrolled on Microsoft’s JMRP and SNDS, Google postmaster tools. I verified my domain. I got 10/10 on mail-tester.com. Thanks to everybody who wrote suggesting solutions, but I did not have a configuration issue. My emails were not delivered due to blacklists, either public or private.”

          2. > How long before DKIM, SPF and DMARC don’t cut it anymore and there is a new protocol you must also wrangle into your service?

            Hard to predict. There are a few existing ones you didn’t list. I’d guess when the majority of mailops see the need. Deprecating plaintext between MTAs will probably be the next collective step.

            The three big players certainly don’t have unanimous power to enforce anything (Gmail doesn’t even dare DMARC p=reject), so it would take a while.

          3. > How long before DKIM, SPF and DMARC don’t cut it anymore and there is a new protocol you must also wrangle into your service?

            That’s part and parcel of running a server. It’s just as valid as asking “How long until SSL2 doesn’t cut it?” Or asking why no-one in their right mind has XP systems exposed to the internet anymore.

            You are of course free to ignore best practices. And other hosts are free to ignore your mailserver. If you want them others to play well with you, you should expect to play well with them too.

          1. I even only do SPF, and the only blacklists my server has been on since are the idiotic UCEProtect L2 and L3 lists. I did have to request removal at first for UCEP L1 and a few others because my server evidently inherited an IPv4 address formerly occupied by a spammer, but once those were cleared it’s been smooth sailing.

            I also only let my server send for trusted (read: my own) domains, and even sending requires authentication over an SSL connection.

        1. Without SPF, you can’t say that your domain has never sent spam. Really, you need all 3. Otherwise anyone can send anything “from your domain”. Much like slapping whatever return address catches your fancy on a letter.

      1. Yes I am for all the domains I manage. But I have a similar problem.
        There is masses of spam sent with one of my domain names, it’s a 4 letter TLD which I grabbed by sheer fluke one late night many years ago – used to belong to a bank but is our family initials.
        But I can send a message from it or from another name to the same end address and often the 4letter wont get through. Which is purely down to the big providers filtering it and adding spam points.

      2. I am.

        And I still have problems with messages sent from my server being silently marked as spam by gmail. Google apparently blocks all “small” senders and there’s nothing you can do. They’re not strictly a monopoly, but large enough that they can set their own rules and ignore complaints.

        I’ve reluctantly switched to sending outgoing mail via a third-party service so I can send mail to friends on gmail. Let’s see how long it works.

        1. Interesting. I never have problems sending to gmail accounts, but Yahoo sends almost anything from my server to the spam folder. Which is arguably better than some of the big ISP’s email “service” which just eats it and never gives their customers a chance to decide it’s ham or spam.

          1. I’ve been running email servers since the .UUCP days. I have a personal domain registered in the early 90’s. I do SPF, DKIM, DMARC, TLS, rDNS, etc etc. I also run a few private mailing lists for a few clubs I’m in, using ‘mailman’. Recently, google started rejecting my list mail because about 10 recipients were gmail addresses. Google’s postmaster tools doesn’t indicate a problem and my DMARC reports come in clean. Good luck getting anyone at Google to help debug the issue.

            It would seem email is dead unless you pay Google or Megamailservers or one of the others to transport your mail for you.

        2. I’m pretty sure I qualify as a “small sender,” and my mail gets through to Google just fine. I did a test just now to my Gmail address, and found the following lines in the header:

          “`
          ARC-Authentication-Results: i=1; mx.google.com;
          spf=pass (google.com: domain of scott@alfter.us designates 45.79.80.216 as permitted sender) smtp.mailfrom=scott@alfter.us
          Received-SPF: pass (google.com: domain of scott@alfter.us designates 45.79.80.216 as permitted sender) client-ip=45.79.80.216;
          Authentication-Results: mx.google.com;
          spf=pass (google.com: domain of scott@alfter.us designates 45.79.80.216 as permitted sender) smtp.mailfrom=scott@alfter.us
          “`

          That would indicate that an SPF record is sufficient to stay on their good side, as the only other measure I’ve implemented is Hashcash. SPF is probably the easiest to set up, as you only have to add a TXT DNS record. Mine is “`v=spf1 a mx -all“`.

      3. I have all those in place, and Gmail still regularly throws my mail in the spam. One of my clients once ran a checker after Gmail had thrown my email in his spam folder, and he told me he had never seen a score so good. There was absolutely no reason it should have gone into the spam folder. It’s 100% a scam and abuse of power to make sure independent email doesn’t work so you give up and host with them.

        1. Google has abused power in many ways already. Not the least forcing you to adapt their websites to match their design rules to rank well.

          Same way they don’t send referrals anymore for “privacy” reasons. You still can get that information if you use their Webmaster tools, so that’s obviously bullshit.
          They aggregate all that information, but only give you selective access to it, with paid plans to access all…

          The amount of influence Google has is sickening.

          1. When this happens I just tell my users “Ask your recepient to find your e-mail in the spam folder and click “not spam”. After 1 or 2 days the problem is fixed for both Microsoft and Gmail recepients. For sure it sucks but it works and happens maybe once a year.

        2. I agree with the above comment as I run a small mailserver and experienced it myself. (Both for Google and Microsoft.)

          I’ve understood that besides the obvious SPF, DKIM, etc. these large mail providers also keep reputation scores of other mailservers / domains.
          For small mailservers there is no reputation score kept at all and such mails are therefore considered spam easily.
          You need to send out a significant amount of email before these large email proividers will keep a reputation score of your server/domain, otherwise they can’t be bothered to keep a score.

        3. Two of mine domains (as mentioned above) are hosted with google and one of them still suffers where as the other one is fine.
          So if you can’t beat them join them sadly doesn’t work either.

      1. It’s not my domain I have problems with, but my IP address. I host my server with an address from one of the Linux hosting services. Since someone will inevitably set up a spam server using one of their IP addresses, some blacklists (Spamhaus) will just go in with a big boot and mark ALL of their IP addresses as spam senders.

        1. It’s a bit reductive, but true, that you need to move to a provider that takes seriously spam sending from their addresses. If you’re with a company that allows such behavior, then don’t be surprised when the whole range gets blocklisted.

          In an ideal world this would punish bad networks and encourage them to boot spammers ASAP.

          I’m with a provider that judiciously goes after spammers, I have SPF, DKIM, DMARC, and my IPs have correct rDNS records. I don’t have any issue sending to any of the big players.

        1. A simpler architecture is a switch to a sender pays model where a sender holds the mail locally and sends a notice of massage availability. Alternatively, extend smtp to use hashcash in the protocol. See spec for ehlo.

          1. I am having trouble picturing how this fixes the issue. Now I just get notifications from the spammers, and visit their website to see what it was. Spam is rampant enough over SMS to demonstrate this model is insufficient.

    1. ikr ridiculous. You can’t even make a fake account using alt details, as their bots block it almost instantly. THere’s websites that let you use twitter bypassing that crap tho :)

  3. My ISP blocks port 25, so I can’t run an email-server on my devices without resorting to external services to forward SMTP to non-standard ports. I would like to run one, but…

  4. I run my own mail server and never had problems. Sure, I use a network with solid reputation and that’s not easy to come by.
    But it isn’t meant to. Anyone can blast out from port 25 on a VPS. And I overblock VPS ranges, whole hosters (OVH) an countries (BR) zealously.
    If you want to run a reputable service then put in the hard work to host or bounce on a reputable ASN.

      1. And that illustrates my point. No-one answers their cell phone anymore because of all the robocalls. They may look to see if they recognize the name/number, but people are getting to where they don’t even bother to look. I kept a tally for a while, 5.8 robocalls to every legitimate call. Hackaday readers have the skillz to set different ringtones for numbers in the contact list, most users don’t know how to do that.

  5. To be fair, Big Email didn’t start this.

    I ran an independent email server back around 2001. An automatically-applied upgrade mistakenly turned it into an open relay and it was quickly blacklisted by Spamhaus. I fixed the issue within 24 hours and then contacted the Spamhaus mailing list to ask if I could get my blacklisting removed. The answer was, simply, no. I should have been more careful about my email configuration. It wasn’t their fault I’d set up an open relay. Why should they trust what I had to say anyway? And so on.

    Big Email’s current methods are simply a continuation of this pattern.

    1. I wanted to comment this on the main post itself, but I’ll just extend yours by agreeing.

      The article’s author really has a very narrow view of the email ecosystem and his experience really isn’t sufficient for such generalisations.

      Email is older than the internet, some problems stem from that, but the rest is a result of natural evolution of the ecosystem. It’s not a racket, it’s that distributed trust is just incredibly difficult. Most proposals to “solve spam” unfortunately fall short. (There’s a fun short template for testing anyone’s proposal hosted by craphound named spamsolutions.txt)

      Self-hosting email has never been easier, but also at the same time never required one to be this careful at establishing and maintaining trust. The big providers give you tools and information about what they check, it really is the small players that are the most finicky, unpredictable and opaque. There are many small, obsolete and low-quality blacklists (like UCEProtect or Backscatterer) that only small players use.

      We shouldn’t ring the death knell without reason, like this now.

      This has also been discussed in-depth on Hacker News 32720234, 32715437, 32722240

      1. My email exchange is hosted on one of the largest email providers and it still being abused by spammers, i.e. email addresses from my domain are being spoofed. The small guy vs. large guy is a strawman argument.

        1. Eh? Everyone receives spam from pretty much everywhere, that includes forgeries. Those forgeries are also the reason we have SPF+DKIM+DMARC. I’m talking about getting blocked, in that aspect the small players are empirically worse.

        2. People spoofing your address isn’t what gets you put on a spam list. Hell, even putting in SPF doesn’t solve that problem, though. Someone sends “from” bogus addresses on my domain to recipients in Japan all the time. I get bounce messages back about this or that address doesn’t exist (with everything in Japanese). All they have to do is check my SPF record to know the email is being sent from an unauthorized IP address and they could quietly drop the email.

      2. The big providers do not provide useful tools for monitoring your reputation if you only send 10 or even 100 emails per day. Tools like the Microsoft SNDS or Google Postmaster Tools really only work for domains sending more (probably 1000’s of) emails per day.

        If you are trying to establish domain reputation you will likely need to send mail at low volumes for months before you have enough reputation to send higher volumes, and if you try to send more you’ll certainly be in the spam folder or silently disappear. This makes doing things like starting a self-hosted newsletter virtually impossible without paying some sort of troll tax to the system, probably by giving up and paying an external SMTP service to send your emails for you, or resorting to black hat “warming” techniques.

        1. At that scale you don’t need those tools either. You follow the rules and you’re deliverable.

          > If you are trying to establish domain reputation you will likely need to send mail at low volumes for months before you have enough reputation to send higher volumes, and if you try to send more you’ll certainly be in the spam folder or silently disappear.

          Certainly.

          > This makes doing things like starting a self-hosted newsletter virtually impossible

          Absolutely not. Grow your newsletter (or the outgoing rate) gradually, you’ll have no issues.

          1. As long as what you do is legitimate and people agreed to receive your mails, noone else should interfer.
            This authoritarian thinking and people justifying it are way off base.

            It’s exactly the problem monopolies and too much power create.

          2. > As long as what you do is legitimate and people agreed to receive your mails, noone else should interfer.

            What has been described *is* what a legitimate sender should look like because people agreeing receive your mails have also agreed on these criteria.

            Far from authoritarian or monopolistic, just face the fact that abuse is a massive problem. You have to prove yourself before you can start spewing to hundreds. It’s not that hard to grasp really.

          3. You forget that others dont follow the rules even if you do.
            Somewhere there is an errant post officer worker that just chucks all his mail into the nearest trashcan and is waiting to get called out on it. Some manage to get away with it for years.

            it’s even easier in cyberspace since it’s a person making a machine do it and that could be anyone at Google.

  6. I run my own mailserver since 2015 and going on. I never had this problems that the author mentions. Who know if he just had sometimes misconfigured his mailserver and is now on some block list like spamhaus etc.
    Currently I have no dkim or any other thing like this running.

    In my opinion the author overstates that. Maybee, it could happen, but currently it don’t happen. Who would professionally use mailing from i.e. Microsoft, when he is not sure that mails from other enterprises will reach him?

  7. There are so many assumptions or straight up false claims in this article.

    First he talks about having to change his MX records and can no longer receive email on his personal server. This has nothing to do with sending email. He can receive email wherever he wants without any intervention.

    Then he talks about emails going directly to spam unless they are from some big provider. This is false. Anyone can send emails that won’t directly go to spam if the domain and originating IP address are properly setup with SPF, DKIM and DMARC records.

    Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.

    I’ve never had a problem with a properly configured email server. By this I mean making sure the IP isn’t currently on any ban lists. If it is, fix that first by submitting removal requests. That is pretty simple to do. Ensure you have reverse DNS configured on the IP. Setup DKIM, SPF and DMARC on your domain. Done.

    You’ll only encounter problems after this if you email server starts sending out spam or if your configuration become invalid from some change.

    Lets say all the issues he is having are true. That doesn’t mean you can’t still run your own email server. You just need to relay outgoing email through a 3rd party. For any personal server, relaying outgoing email through something like AWS SES is essentially free. There are 10’s if not 100’s of other providers out there that also have free plans that would support 1000’s of outgoing emails a month.

    1. “Anyone can send emails that won’t directly go to spam if the domain and originating IP address are properly setup with SPF, DKIM and DMARC records.”

      This is a bold claim, it takes only one counterexample to prove it false. I know of several servers that were properly configured and “just stopped” delivering to gmail at some point.

      “I’ve never had a problem with a properly configured email server. […]”
      You’re a lucky guy then. I followed the same steps that you have described, to no effect.

      “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

      I have no information on how the blacklists work, but I wouldn’t be surprised if ranges assigned to cheap VPS hosting were permanently held on a blacklist. Google certainly has resources to follow the IP range reassignments and update the blacklist accordingly.

      1. “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

        I have no information on how the blacklists work, but I wouldn’t be surprised if ranges assigned to cheap VPS hosting were permanently held on a blacklist. Google certainly has resources to follow the IP range reassignments and update the blacklist accordingly.
        </snip

        I can verify that IP ranges of some fairly large VPS hosts get blocked by one bad actor.

      2. I worked for a major email list company. We were very involved with DKIM from the beginning. However IPs did get blocked. And those IPs also built positive reputations. We had a 4 person team dedicated to delivery. There were so many factors required to quarantine 95% delivery rate. Such as rate limiting for google. Or only using these servers for yahoo, etc. We also have duly automated anti-spam mechanism. New customer emails were delivered via fresh IPs. We used honey traps with feedback loops to catch SPAM lists and would immediate block delivery for customers would had 4+ complaints within 24 hours. Mail delivery is difficult even for a professional team of dedicated pros.

    2. “Anyone can send emails that won’t directly go to spam if the domain and originating IP address are properly setup with SPF, DKIM and DMARC records.”

      This is simply not true, as many small e-mail server admins will attest to. Ever heard of IP reputation? It’s a business, you know.

      “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

      So they *do* blacklist certain IP blocks? Do you realize you’re claiming that something is happening and is not happening in one sentence? Is this Schrödingers blacklist?

      “I’ve never had a problem with a properly configured email server. By this I mean making sure the IP isn’t currently on any ban lists. If it is, fix that first by submitting removal requests. That is pretty simple to do. Ensure you have reverse DNS configured on the IP. Setup DKIM, SPF and DMARC on your domain. Done.”

      No, you’re not done. Ever tried lodging removal requests with Microsoft? They’ll claim they aren’t blocking anything *even when you show them their own “550 … is on our block list (S3150)” reply*. Incidentally, there are a lot of people on outlook/hotmail addresses.

      “Lets say all the issues he is having are true. That doesn’t mean you can’t still run your own email server. You just need to relay outgoing email through a 3rd party. For any personal server, relaying outgoing email through something like AWS SES is essentially free. There are 10’s if not 100’s of other providers out there that also have free plans that would support 1000’s of outgoing emails a month.”

      This is, essentially, what he is saying. You’re forced to use Big Tech or some business to have your e-mail delivered.

    3. “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

      When I changed ISPs – I discovered that researchgate.net was blocking the entire netblock from downloading papers that are otherwise freely available. Their proposed solution – ‘create an account to login’ – did not work because my account creation request was rejected (no published papers for me to put into their network, therefore no reason for them to grant an account), and their support was unresponsive.

      An entire netblock blocked, with no recourse. Several internet proxies work fine as a workaround.

      I recently discovered, while trying to renew a club membership on a site hosted by wildapricot.com, that their (wildapricot’s) payment system was also blacklisting my home IP block. Not even the courtesy of kicking back an error message – connection attempts were just blackholed.

      It appears that I may have inadvertently signed up with a service provider who has a netblock of ill repute – but, as you say, IPs change hands all the time and it makes no sense to punish them.
      This is a great theory, but laziness (or, to be more kind, lack of resources) prevails.

      While not email examples – my personal experience of blocking entire netblocks for an indeterminate amount of time (possibly permanent) and without any further information or recourse does, indeed, happen. And the problem is even more insidious than email servers.

    4. “Anyone can send emails that won’t directly go to spam if the domain and originating IP address are properly setup with SPF, DKIM and DMARC records.”

      As a former member of the anti-spam team of a very well-known company, I can assure you that this claim is not true. It is good advice, but it is not a guarantee. If your ISP leases a dozen nearby IP addresses to a spammer, you’re doomed. If they leased them to a spammer a month ago and kicked the spammer out after a week, it could be a few more months before the block-list rule expires.

      The reason SPF / DKIM / DMARC doesn’t solve the problem: Spammers are perfectly capable of setting up SPF, DKIM, and DMARC. And they do, and they do it much more consistently than legitimate organizations do.

      That’s a common problem with most of the proposed fixes. Whatever measures you propose, you have to ask, “what stops spammers from doing exactly the same thing?”

      The only technically feasible solution would be something like a bond, where the owner of an IP address or range puts real money into escrow, and that money is forfeited if spam emanates from the IP address or range. To mitigate conflicts of interest, the money goes to a charity of some sort. The amount of money doesn’t need to be large, it just needs to be larger than the profit from however much spam is tolerated before the money gets forfeited.

      But that’s still a nonstarter for other reasons. Just imagine the outrage if Google / Apple / Microsoft / etc said you had to submit money to send email. And imagine how hard it would be to agree on a value. One could reasonably argue that less wealthy countries should pay less, but that just means that spammers would lease IP ranges from ISPs in those countries.

      1. Spammers can set up SPF/DKIM/DMARC, but then they’re restricted to sending from domains they control, and can’t fake that they’re sending from @microsoft.com or whatever, and that’s the point of those.

    5. “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

      Permanent blocks don’t happen, but long term blocks (a year or more) do happen. It takes persistent spam activity to get there, but if your ISP has a habit of leasing nearby addresses to spammers, you’re doomed.

  8. Running a proper outbound e-mail server hasn’t been realistic for me for well over ten years. I route what little e-mail I send through my ISP’s outbound gateway. It got more difficult when my large ISP decided to fob all that off on Yahoo, and even more difficult now that it requires you to authorize via HTTPS to get a time-limited cookie of some sort. I still run OS X 10.13 for various reasons including compatibility with old apps, and I have to do a dance to re-authorize it every month or so.

    Inbound e-mail is still fine as long as you have a static IP and your own domain. Spam is very much a problem, but became much less so a few years ago when most spammers started using alternative TLDs (have you even heard of .top or .stream?) in their mail configuration. No sane person should be using them for e-mail, so filtering them cuts most of the spam instantly. Beyond that I choose to filter most third-world country TLDs, and I even have .us blocked except for whitelisted domains of a few people I’ve met in person. I only add blocks when I encounter them in spam, so every one is a decision that nothing useful will ever come from that domain or IP range.

    1. The .space TLD has been available since 2015, and I still have trouble using my email address on a lot of online forms (please enter a _valid_ email address).

      Please don’t block whole TLDs for no real reason.

  9. I used to use a remote FIOS Business IP address for sending but now I receive in my basement and smarthost out to mailgun for free. I haven’t had any problems AFAIK but it’s only me sending and receiving.

  10. I am running my own mail server, with SPF, DKIM, etc. set up properly. The IP has a very good reputation.

    I never had any problem, with one exception: Microsoft mailservers (Outlook, Hotmail, etc.) some months ago stopped accepting my e-mails, with the exact reason mentioned in this article. Another server in the IP range sent spam, so they blocked the whole range – including my mail server, which was at no time the source of any spam.

    It took some effort to make clear for Microsoft that they did some kind of overblocking. Finally I was successfull and they unblocked my IP.

    So, I agree with this article, this is a real-world-problem. But truth to be told: This just happens once with mail servers operated by Microsoft. All the other “big ones” never complained.

  11. Got to say, I’m equally worried about the future of email, but I don’t find “big email” are the problem here.

    Gmail and hotmail have in my experience been fine receiving from a properly set up email server. I’m not convinced they use IP blocks, as IP block ranges vary so much, and spam problems appear and disappear fast.

    It’s small players who rely on 3rd party blocklists which have been the issue. Usually ISP email services. They use an outdated 3rd party blocklist, and when you raise a request they Fob you off to the blocklist maintainer… who tells you to get lost as you’re not their customer.

    Also, a large amount of spam – especially that targeting businesses – originates from throwaway gmail addresses. So they’ve clearly not cracked it themselves.

    1. >who tells you to get lost as you’re not their customer.
      How nice people have become…

      They seem to forget what they are doing is abusive and harms your endevour (or business) and goes against the free market (and freedom in general).

      But that mindset is so established, it ruins the Internet and collaboration in general.

      1. Um, I don’t of any laws that say other businesses are required to play nice with your business. I don’t think freedom means “be nice.” Often, it means the exact opposite! I’m not advioating for businesses to act like jerks, but “freedom” also means “freedom to act like a jerk.” And that’s usually more profitable than being nice.

  12. My VPS provider charges *per month* for a ptr record. Sending mail without working reverse DNS is going to be super unreliable. Rather than pay the nickle and dime VPS provider, I forward outbound through a service. It is lame, but it also gets around IP reputation issues by recipient sites that block/mark as spam based on entire netblocks or ASNs.

    Special shout out to how terrible Microsoft is. They silently delete mail they accept, so neither recipient nor sender is aware it was not delivered. If you use MS office 365, you may have mail silently disappearing because you use an incompetent service provider.

    1. “Special shout out to how terrible Microsoft is. They silently delete mail they accept, so neither recipient nor sender is aware it was not delivered”
      This! Even when a small company uses a ISP/mail host company you are not assured messages wont be dropped by MS servers the reciepient uses since MS blocks large swaths of IP addresses even though everything seems correct and the IP do not appear on any other block list (outside MS)

  13. I am totally OK with any measures to get rid of spam on email and cell phones.

    I ran my own email server for a while. It was a huge waste of time and really not a lot of fun at all. I can’t think back and put my finger on ANY benefit for all the trouble. I now use gmail and never see spam and have time for interesting and worthwhile things. Works for me.

    1. Benefit? Privacy? Not giving your data to big companies for minining it? Perserving trade secrets….

      That you can’t see a benefit is because big companies own so much of email traffic that it makes almost no difference. Furthering them more makes it worse.

  14. @Avamander
    (hackaday comments don’t reliably end up under the comment they are replies to)

    Backscatter RBLs were great (when used to bias spam scores, not as a ban hammer). There is zero reason to be sending any backscatter. Pretty much the only people who did were running Microsoft Exchange, which (brokenly) accepted all mail, then sent NDRs (backscatter) to the (possibly forged) sender addresses. Spammers could relay spam through any Exchange server by putting the intended recipient as the sender address, then sending to a non-existent address on the MS Exchange server. Backscatter RBLs got this garbage off the Internet.

    The correct way to handle a non-existent recipient, which literally everything except Microsoft Exchange did/does, is to reject the mail during the SMTP conversation. The *actual* sender (even if a forged sender address) gets notified, and the forged sender address never gets spam.

    1. In theory I absolutely agree, but then there are mailops who use those list for instant rejects. Plus those specific ones both contain many false positives, yet are provided by the most arrogant set of people who admit no fault that I’ve seen in a long time.

  15. I haven’t touched email configuration since the dark ages. Is it still the case that misconfigured SPF/DKIM/DMARC settings fail without notification – that is, if you make an error in setup, your only indication is eventually noticing that no one responds to your email?

    1. No, that’s not standard. It would cause far more trouble than benefit.

      You’d be amazed how many legitimate companies can’t get SPF+DKIM+DMARC configured properly. Heck you’d be amazed how many legitimate companies can’t even get one of the three done right.

      There are free online tools to validate your configuration though, so while it’s certainly easy to get them wrong, there’s also no excuse for leaving them wrong.

  16. this article touches on a real problem, but i don’t think it really gets at the facts about it very well.

    i’ve run a little mail server for years. i’ve had a lot of problems with it. the time one of my users forwarded all their email to his gmail, eventually leading gmail to classify my whole host as spam. or when AOL decided that mailing list forwarders should rewrite the From field — or else! and i’ve had to set up SPF and so on. it is a pain. but each problem has been surmountable and i’ve been able to carry on.

    and my boss runs a mail server, about the same story. sometimes we find a customer that can’t receive our emails, and that’s their problem not ours. it’s not that we don’t care about them it’s just that we can’t do anything about big corporations with ridiculous email policies. the people we correspond with are *well* aware that their IT department has made their whole life miserable. we’re not the only ones they have trouble talking to. they’ve got a list of work-arounds handy. and so do we — if we don’t hear from someone, we’ll ping them on the phone or from someone’s private gmail account.

    the thing this article is failing to take into account is that the long tail is essential. no matter how huge gmail is, these little mail servers are still a big part of everyone’s life. some businesses do outsource to gmail but a lot still run their own mail servers internally. even a big corporation is small compared to gmail. so you’ll have an IT department representing 1,000 or 10,000 users, and they will be in the exact same boat that i am in. and those users are valuable — people want to hear from them.

    in fact, i think generally people coming from small email servers have more worthwhile communications than people who use gmail. gmail users are like the old AOL users. it’s not universal by any stretch, but most of the people who don’t know they have an email address, don’t know the difference between email and SMS, the people who click on phishing emails, they are all on gmail.

    and there’s another detail, which i don’t know if it gives me hope or not. that is, email spam is dying. i’m not sure why, but on my work email (where spamassassin tags instead of deleting), i am getting 10% of the spam i got 5 years ago. it is really dramatic. i think the casual email user who is most susceptible to scams simply isn’t looking at email anymore, or maybe gmail is just doing such a good job of filtering it out. maybe it just shows that email is dying overall.

    1. > the thing this article is failing to take into account is that the long tail is essential. no matter how huge gmail is, these little mail servers are still a big part of everyone’s life.

      Absolutely. The author has seen the sending side, but not enough of the receiving side.

      > i am getting 10% of the spam i got 5 years ago. it is really dramatic. i think the casual email user who is most susceptible to scams simply isn’t looking at email anymore, or maybe gmail is just doing such a good job of filtering it out. maybe it just shows that email is dying overall.

      I doubt it’s people using email less, but I do think that the bar for sending spam has risen. Can’t forge every domain out there, can’t use any IP instantly, can’t deliver tens of thousands of emails one instant and so on. The same methods the author blames for killing email really. But these have made it much more difficult to spew out spam.

      My suspicion is that the reason behind the article is just his expectations not matching reality. Nobody told him that it’s a hurdle race and the hurdles hurt to hit. But being prepared, it’s absolutely doable.

    2. The reduction in spam that you’re seeing is due to improved filtering. I assure you the spammers are still trying just as hard as ever.

      The amount of mail that gets rejected by large email providers is truly staggering. I used to work for one of them.

  17. Its not an accident that big tech is crushing private email, its a feature. They are intentionally closing their ecosystem to stifle competiton. The answer is as simple as not using big tech email. My business email has been hosted on my vps for years and I never have issues with it. My clients also are not using big tech email that I’m aware of. I don’t have spam problems either. Granted I only use that email for person to person correspondance, and use a seperate email for forms (a practice I highly recommend), but still.

    The whole thing is a scam, just like when Microsoft tries to get you to log into windows using a Microsoft account “for increased security”.

  18. I’ve been running my email and website server at home for about a decade on a Raspberry Pi. I followed most Google’s spam guidelines to avoid being flagged as spam.
    I don’t see a single reason to stop. There’s close to 0 maintenance and the domain name is only 1€ / month.

  19. “Imagine the Internet had begun its life as a proprietary network from a major software vendor rather than evolved as a distributed network shared by researchers. It’s a future that almost came to pass for consumers in the 1990s when walled gardens such as AOL or the original incarnation of MSN were all the rage, but thankfully the world took the Internet course. ”

    *Ahem* CompuServe was one of the driving forces for international e-mail communications!

    Long before the internet was made open for thr public.

    If you look at readme files of shareware and public domain software from the 1980s and early 90s, you will see that CompuServe e-mail adresses in number form were very popular!

      1. CompuServe started out as an online service, a database.

        Originally, it was no Internet Service Provider (ISP).
        That role did CS take in the 90s.

        The original CompuServe network was all about Forums (GO something) and e-mail.

        CompuServe had its own computers/hosts that were accessible via X.25 networks – wordwide.

        On PC, you could access CompuServe via terminal program OS/2-CIM, WinCIM, or CompuServe Informationen Manager (DOS).

        You could access CompuServe via other X.25 PADs from all over the world. Like, Datex-P network (Germany), Alaskanet, Datapac or how they were called.

        The e-mail transport worked cross-platform, to/from non-CompuServe users. You could send e-mails to, say, AOL or T-Online users, too.

      2. Please excuse if I misunderstood your comment.

        What I meant to say: The problem described in the article didn’t exist. At least not in this form.

        a) E-Mail predated the internet

        b) E-Mail itself didn’t need the internet, it was routable across boarders. Just like real mail (aka snail mail).

        It was also possible to send real, international e-mail via, say, Fidonet or Packet Radio. Gateways between the different technologies existed.

        c) AOL was just another online service, quite comparable to CompuServe.
        It had its own forums/database and protocols, that’s separate from the internet gateway it provided.
        AOL wasn’t doing business as an ISP originally, it was its own thing.

        E-Mail transfer was not limited to the internal AOL eco system, however. It was international.
        So even if AOL had been the dominant “walled garden”, it wouldn’t have had been prohibiting e-mail exchange with other providers.

        The whole assumption is wrong, from my point of view. It makes the internet more important than it really is. Even before the internet, gateways among the individual networks existed. The underlying concept wasn’t new, at all. Internet just means “Interconnected Network” – a mess of interlinked systems/network that used to exist as individuals.

    1. You’re garbling. Email existed in the sixties, but with no networking, it was all unique to the mainframe.

      Email came to Arpanet in 1971, mailing lists soon followed. That’s really email as we know it.

      Compuserve and the like had their own email, it made sense since no networking hence it was between users. But at the time, they each had a large userbase, so it was useful.

      But they wanted people to join up.

      Fidonet was probably influenced by Usenet,which came along in 1979. Both used the phone system to exchange mail and messages.

      The internet arrived at some point, and took a lot from Arpanet, including the email concept.

      At some point Compuserve and the like needed to exchange email, isolation was failing. That’s when you could send email out of the system.

      And the internet won, at some point few wanted the isolated and proprietay systems.

  20. Its not just small email hosts that can get caught up in overzealous blocklisting. Recently AT&T/Yahoo/Oath/Mindspring/whatever their name is this week managed to block all Office 365 email servers from sending email to addresses under their control. And vice-versa (apparently their blocklist put in firewall rules that disallowed connections out too?) So I had many clients complaining that all their emails were bouncing (across multiple domains, on both sides of the divide), including my boss, who has a legacy address from yahoo that forwards to his office 365 email.

  21. “Then he talks about emails going directly to spam unless they are from some big provider. This is false. Anyone can send emails that won’t directly go to spam if the domain and originating IP address are properly setup with SPF, DKIM and DMARC records.”

    I have run my own mail server for well over 20 years andas years go by it has been more and more difficult. First my IP address was rejected because it was residential, so I changed it to a commercial connection instead, which worked for a while until the ISP-provided router started spewing spam, getting the address added to several blocklists. I reported the problem to the ISP that responded fairly quickly … by blocking port 25 outgoing. WHAM! As it took a while to move to another ISP, I set up a virtual server on Linode for outgoing mail. Despite having SPF records, DKIM signatures, a DMARC record and a reverse DNS entry, I still had (and have) issues which Linode has been very helpful to try to alleviate, although there are some E-mail services that are not receptive.

    “Then he says how big email providers permanently blacklist whole IP address blocks. Outside of residential/dynamic IPs and maybe IP blocks from certain countries, this just doesn’t happen. It makes no sense to do this since IPs change hands all the time.”

    This is not true. My Linode server’s IP address is still on a couple of blacklists despite no spam has ever been sent by me in the years I have had it.

    “I’ve never had a problem with a properly configured email server. By this I mean making sure the IP isn’t currently on any ban lists. If it is, fix that first by submitting removal requests.”

    Unfortunately, many blacklist providers do not have any way to request removal, so good luck with that.

  22. I’ve had my own domain since the ’90s. I have my domains at a reputable registrar, and I get hosting and email service under my domains at a reputable midsized hosting place. As far as I can tell (including http://www.mail-tester.com) my emails almost always get through. Seems my hosting provider is on the ball. Yay. I would rather gouge my eye out with a grapefruit spoon than have to maintain my own email servers.

  23. If a charge of 2 cents is imposed on each email sent and each megabyte sent, and 1 cent is credited to each email received and each megabyte received, there probably won’t be any more spam emails.

  24. Fairly ridiculous claims in the article when you have vast systems requiring self hosted email servers to work on-premise for certain software integration, an average aggregate downtime annually for cloud systems now bordering on 5 weeks, and self hosting being easier than ever to deploy and maintain.

    But if one had little knowledge about the state of email systems, and was a fan boy of large corporate cloud email systems (which have had a blatant artificial PR push as well as manipulation to cause specific trouble for on-premise email systems in the recent year) then I guess I could see why the writer would make such a claim

    Tell us, exactly how much does Microsoft or Google pay for shadow promotion articles to declare their opposing solutions as “dead”?

  25. Implausible generalization and false assumptions.
    Just because you can not deliver Mail from dialup-IP homelab does not mean, you can not selfhost email.
    You still can selfhost email.
    1) DUL RBLs are blocking at most providers since 2 decades now. (how did that work for carlos the last 10 years?)
    2) you can selfhost email on VPS, on bare metal, on colocation, or even on a homelab AS. (and many more). just hone your tech skill a little bit! there are multiple tech stacks available which make it really easy. plenty of choice. (or maybe hotmail is the better proposition for the not so tech savy)

    1. “Implausible generalization and false assumptions.”

      Are you hosting your own email right now? Have you ever hosted an email service for other people? Have you ever had to deal with the FBI over someone using your email service in an illegal way? Have you dealt with spammers? I’ve dealt with those problems at least as far back as 1999.

      It’s easy to set up a mail server. It’s hard actually running a productive mail server and making sure the emails you send get eye’s on them reliably. That’s why most are willing to pay for hosting or at least pay for delivery these days. You can’t completely control what other people do. You get a spammer, a warrant, a lawsuit, blacklisted, or a data breach and you have to clean that up yourself.

      1. Simple: don’t let people you don’t trust use your mail server, and the FBI and spammers won’t be a problem in the first place.

        I host my own family server and have done so for over a decade, less than 20 users peak. All those problems you’re railing on about go away (except data breaches, but if you maintain competent security and a relatively low profile it’s largely not an issue). If you’re hosting for untrusted folks… well that’s what they’re paying you to handle for them, isn’t it?

        1. You have no idea how lucky you are that your ISP never leased a bunch of adjacent IP addresses to a spammer.

          That’s all it takes to find yourself on a block list for a month or three.

          1. Oh it’s happened. I end up on the UCEProtect L2/L3 lists periodically, but most major mail carriers don’t give a **** about those lists anyways precisely because of how useless they are these days.

      2. @jim:
        To follow your questions:
        1) In fact, I am hosting my own email right now.
        2) Yes, i do host email for a few dozen people noncommercial (ngo stuff), since about 2 decades
        3) Yes, authorities did contact, via the legally required procedures, which have been followed.
        4) “Dealing with spammers”: You mean as my users? No. I selfhost for people and organizations i know. What relevance has this question?
        5) just for the record: if your argument is “i do this longer than you”: I started with sendmail in 1995, switched to qmail about 1998, switching to postfix made life easier with RBLs and SpamAssasin, about 4 years ago i switched to the mailcow-stack (even before they became dockerized).

        my conclusion:
        You say, that you have trouble operating your mailserver. That’s a pitty to hear.
        Since here running a mailserver selfhosted: Piece of cake! It is a side job which requires just a few minutes maintenance per week. Perhaps i was just lucky, that best practice procedures were implemented good enough, never lost data, never had any breaches.

        Running your own infrastructure comes with responsibilities.
        I would suggest teaming up! reliable, trustworthy people to share the load and to accumulate knowledge.
        If you do not like that, then those $$$-services are perhaps the better solution.

  26. Why isn’t strong, end-to-end encryption used BY DEFAULT in email apps and provider email services? Because both governments and private firms want what’s in those emails for different reasons, both claiming that one has no “expectation of privacy” because those emails travel unencrypted through multiple servers? And when one uses some external app to do that, all they do is set themselves up for suspicion.

    1. It’s not really that. E2E is just really really damn difficult of a problem. We do have S/MIME and GPG but both have major usability problems before they could be mass-deployed. If what you described were actually true, both governments and private firms would be blocking S/MIME and GPG email.

      The good news is that CA/B Forum is drafting a new S/MIME agreement, making S/MIME ACME possible. This might provide the critical mass required to properly make use of S/MIME and do E2E encryption with email. We’ll see though, maybe the GPG zealots will sabotage that effort with FUD, and it won’t gain adoption.

    2. It’s a catch-22 problem. Nobody is going to start sending encrypted mail until they can be certain that their recipients will be able to read it with no extra steps. And nobody is going to create zero-effort decryption software until encryption is widely used.

      Server-to-server encryption is a much easier problem because the two servers can agree on an encryption scheme at the start of the connection. But user-to-user encryption requires the humans at each end to agree on an encryption scheme, which requires extra steps, so it doesn’t happen.

      It’s an “easy” problem to solve for person-to-person mail – you just need to find out which of your friends are willing to use S/MIME. But no business is going to start sending order confirmation email with S/MIME until 100% of their customers can decrypt those emails with zero effort.

  27. Linode used to be pretty good. I’ve been a customer for 10+ years.

    Unfortunately, they have recently started hosting ‘security researchers’, which are really no more than fronts for criminal gangs and also host spam pestholes.

    I guess Akami buying them has caused them to stray from their original mission of providing reliable VPS to being completely revenue driven. They are not (yet) evil like Google, but well on their way. I hope they figure this out and get back to their roots soon.

    1. I know of someone using Linode for almost all of his commercial hosting needs but he uses Amazon for email delivery. Linode just wasn’t cutting it for that aspect of his business and he is a huge fan of Linode.

  28. my biggest gripe with email isnt blacklists,
    or even the fact that the “sender” field is a farce (almost optional aka open to fraud)

    my biggest gripe with email is the unpredictable transfer delays.
    how many hours does one need to wait around the classroom waiting
    for the school’s IN-HOUSE(in-school lol) server to relay a message
    TO ITSELF so the teacher can note that the assignment WAS indeed submitted
    once every 10 minutes until all 20 copies FINALLY make it through… itself

    and its not just a school (likely a target of dropout-students),
    its also hotmail, livemail, and sometimes gmail,
    telephone calls only take a minute,
    printing and handing-in only takes 2 minutes if your already in the room.

    email is faster… but; faster then what?

  29. This problem has been brewing for years. The last time in my IT career that I was a corporate sysadmin and ran our own email servers, I spent a fair amount of time hassling with the a*holes that run blacklist services. Frequently having to submit my IP ranges and server domains to their list removal request process and hoping they approved and applied my removal request. These processes became steadily more automated with less and less opportunities for appeal or Human intervention and more and more of them obviously not giving a sh!t about small mall domains. I grew to loathe these blacklist companies.
    These days I would bet they are mostly owned by our colluding with the big email providers to intentionally drive independent server operators off the Internet and over to the big providers. Another case of “you’ll own nothing (including your own email server! ) and you’ll rent that service and be happy”. This also makes snooping you email traffic “for your own good” easier,
    making censorship and social credit scores centralized and easier to do.

  30. I get the impression robocalls are a USA thing? I’ve had the same cellphone number since 1999 and can count the robocalls I’ve received on my fingers. They do happen, but the callers seem to get caught and shut down pretty quickly. Maybe this will change after the recent Optus data breach:-(

Leave a Reply to DavidCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.