Wreck Your Mail Before You Check Your Mail

Every five years or so, I think it’s time to review my e-mail flow. (Oh no!) I run my own mail server, and you should too, but this means that I get to figure out managing and searching and archiving and indexing it all by myself. (Yippee!)

And I’ll be honest — sometimes I’m a bit of a luddite. I actually, literally have been using Mutt, or its derivative NeoMutt for maybe fifteen years, after a decade or so of mouse-intensive graphical mail readers. If e-mail is about typing words, and maybe attaching the occasional image, nothing beats a straight-up text interface. But what a lot of these simple mail clients lack is good search. So I decided to take that seriously.

Notmuch is essentially an e-mail database. It’s an e-mail searcher, tagger, and indexer, but it’s not much else. The nice thing is that it’s brutally fast. Searches and extraction of tagged subsets are faster than sending the same data back and forth to the Big G, and I have a ton more flexibility. It’s awesome. Of course good ol’ Mutt can work with Notmuch. Everything can. It’s Linux/UNIX.

But I wanted an e-mail client that would take the tags-instead-of-folders flow seriously, and make searching a first-class navigation strategy. Mutt is from the 1990s, when e-mail was in its teenage years. I ended up with Astroid, and am currently in the honeymoon phase, still configuring things so that they work just right, but all in all enjoying the change. Of course some of the keymapping is different, so if you get an e-mail from me that’s clearly intended for someone else, well, you know what happened.

So here I am, with auto-tagging scripts that fire off MQTT messages to my home automation system when certain mail comes in, and with a tag system that distinguishes between importance and urgency as well as along defined topics. It doesn’t spy me out, track links I click on, or record every online purchase I make without asking my permission either. It basically fits me like a glove, and I’m pretty happy now.

And all it took? A whole freaking afternoon of tedious work, trying out different software packages, and tweaking configuration scripts. All in all, it’s a mind-numbing effort that I’ll be happy to not repeat until we’re all composing e-mails directly with our neural implants. But how many of you out there are actually happy with your e-mail setup?

28 thoughts on “Wreck Your Mail Before You Check Your Mail

  1. Wonderful article Elliot, especially the encouragement of others to run their own mailserver. I’ve been doing the same for the last 20 years and controlling your own E-Mail destiny is panacea. I’m a fan of pine/alpine/mutt and the other wonderful ncurses style MUAs. As of late I’ve settled on FreeBSD, MATE, and CLAWs for the MUA. For automation and/or mail steering I’m using Procmail.

    The beauty of FOSS is the ability to find a solution that works for you, and if none are there, create your own. Barring that the freedom to be able to modify something that is close to what you want to a full solution is empowering.

  2. thanks, you investigated so we don’t have to.
    Noob question: when running a mail server and not wanting to run a PC/Laptop because of its continuous current usage, would this for instance run on RPI?
    And be easy enough to set it all up for a tech savvy person, but a person who’s never used and RPI before?

  3. Running a mail server has become a hassle in a time of DKIM, DMARC, large anti spam lists and blacklists. Even if you do everything correct chances are you’ll be considered untrustworthy enough to be blocked or filtered, as the system isn’t well set up to accommodate the little guys.

    1. TOTALLY. After all my sent mail to gmail always landed in recipients spam folder, bounced off or in outlooks case simply vanished I nearly gave up and considered using google instead while drunk. Came back to my senses, fiddled around with certificates, DNS TXT entries DKIM,DMARK, acme-challenge etc for two days and now everything send through my server, even newsletters and other machine sent mail, arrives at the recipients inbox. Took me a while though to understand that since i don´t want all my domains to be easily visible on the same server through reverse dns lookup i´d have to register one extra domain for the mail server.
      i´m using a VPS with own IP though so I can control myself who is using that IP to potentially getting it blacklisted.

  4. Old mutter here. User of elm since before the beginning of time. Switched to mutt many years ago. Also, vi to vim to emacs.
    Currently 403 email “folders” in ~/Mail. Nearly 7 GB of email on hand back to 2000 or earlier. Will check out Astroid. Thank you.

  5. There are issues when trying to get DKIM and DMARC working, critically you need to be aware of SPF (Sender Policy Framework) as well. I have had to deal with this at my office for some time. It works pretty well once its tuned. Our office uses O365 and so I have added some custom rules to look at the headers and do more aggressive filter/reject. The reason is that people like to setup SPF/DMARC with policies that have “none” for what to do with a message that fails the check. I refuse to accept that if your configuring protection for your domain that you want me to ignore the protection. Either get it right or don’t set it up at all. I treat “softfail” and “hardfail” as just “fail” and reject. Again protect your entire infrastructure or don’t send messages my way. We are small so we can have these more aggressive rules. I do like how SPF/DKIM and DMARC have helped cut down on a lot of spoof related spam. Finally I reject any TLD in the from & reply lines that don’t meet our needs.

    In the end I have had a small number of our vendors caught up in configuration issues. In fact one issue was they used the right rule however critically the RFC says not to have anything at the end of the line not even a ” “. That blank caused the rule to break! This is mind-numbingly nuanced and was a pain in the rump for a few weeks as we went back and forth about what was the root issue. In the end I used some of the public tools to evaluate the records involved. One said the record was fine, the other said the record was invalid. Upon closer inspection it was in fact invalid due to the trailing whitespace that made it into the vendors published rule because they copy and pasted. Its this kind of thing I can understand makes people want to avoid it but the payoff is worth it.

    For my home server I use cloudflair’s free certificates to sign my domain for DKIM. Its free and does the job for those looking.

    just my 2 cents

    1. Some Internet Service Providers use external anti-spam services that filter all outgoing email. They use it to prevent their mail server IP addresses getting blocked on anti spam lists.

      For one of my ISPs uses an external anti spam service that adds its X-Spam-Report to the mail header. Because of that some recipients bounce my legit emails with the error:

      INVALID HEADER

      Improper folded header field made up entirely of whitespace (char 20 hex):
      X-Spam-Report: …that system for details. Content previ[…]

      I cannot change anything in that header myself.

      Mhh, indeed another reason I should start running my own mail server :-)

  6. “Run your own mailserver” is a poor suggestion. DKIM, DMARC, checking RBLs to keep yourself off (or remove yourself from) spam lists, mitigating attacks from bad actors… it’s just not worth it anymore, and will add substantial overhead to your daily routine.

    Unless you’re dead set on committing the time to administer your own mailserver, it’s best to just leave it up to a turnkey solution, especially a paid one… so you can have someone to yell at when something invariably breaks.

    1. Concur; things are hard. Better to serve an upstream vendor, so that, when things break, you can recite “Thanks for reporting this we’ve placed this on our project roadmap and will report back immediately when implemented.”

      1. That’s not at all what I said, was it? I covered specific issues in the second sentence. If you’re willing to take on those specific issues, by all means… which is what I covered in the last sentence. My specific contention was as follows: “run your own mailserver” is bad advice for normal people who underestimate just how involved running a mailserver actually is, especially if you want your mail to flow without interruptions… which I addressed in the third sentence.

        Reading is hard.

        1. One possible issue: you’re focusing on normal people. HaD is somwhat tailored for the needs of non-normal people. So, good point, but possibly irrelevant to the audience you’re reaching :)

  7. Big companies aren’t interested in fixing problems, but in raising the entry bar. Otherwise they would fix the classic “local account mails external user, and the reply is sent to spam” yet spam reaches inbox, or sending spam with all the DKIM/DMARC/etc crap instead of suspending the account. At first I thought it was only me, but I have seen multiple cases reporting things like those. And I think we have seen enough “incompetence” that coincidentaly benefits the incompetent to not believe their is a bit of bad faith either at the start or by not mending the issue once known.

    As for using notmuch with programs that use directories, there is notmuchfs. Or special purpose tools like notmuch-mutt. BTW, why no links to mutt.org?

  8. I ran my own mail server for a decade. The combination of spamcop and clamav meant my mail queue was TWO DAYS in processing time, and ever growing. I decided it wasn’t worth my cpu burning 100% just processing mail, when it was supposed to be my web server first, and email second. Now I let Google burn their cpu and I’m fine with the data mining (good luck with the 1:30,000 SNR!) and low cost, as compared to the high cost of data center power.

  9. Outlook sux. All my links are converted to some gibberish (so they can see if I click them or something) and I cannot tell what I am clicking on. Plus its super slow in linux (probably intended) and only works well on windows. It eats up a couple of hundred megabytes of ram (how rude), and pretty terrible for any practical use. An old workplace used it, and was near impossible to decide what emails were important, and what was just junk. Terrible, dont use if if you dont need to..

  10. Not practical in 2021. You must at LEAST have a static IP address if you want to send email to the world yourself. Having a smarthost/relay for the outgoing email just negates most of the benefits anyway. It is just not possible to run your mail server at home with your simple mortal internet connection, you’ll be flagged as spammer in less than you can say “apt install postfix”.
    .
    Get a reasonable email provider that used FOSS, like soverin.net or similar ones. Or if you are really into it for the fun, prepare to spend money on a VPS in the cloud.

  11. Everyone taking about how hard it is to run your own mail server these days hasn’t tried mailinabox. It’s seriously amazing. Throw it up on a cloud instance and it uses Let’s Encrypt to get you an ssl cert, then configures DKIM and DMARC and everything else, it’s such a beautiful project, please check it out!

  12. Thanks for the “I concluded this is what I want to try” from all of the options out there. I don’t share my similar “conclusion” (as temporary/permanent as is it) as I get bogged down in making sure I am treating it scientifically and being “fair.” Inevitably, I just don’t share the hours of “work” I have done and leave others to replicate it for themselves.

    On that note, I am on Fastmail for email hosting. I recommend it if you don’t want to host, but if you don’t want to be on Google.

    I reviewed lots of options for self hosting and I actually enjoy (and understand after hours of research) setting up all the DMARC, SPF, Domain Keys, etc. but concluded that “deliverablilty” was not something I wanted to deal with. I ended up at Fastmail because they are a smaller player, but actively contribute to standards bodies and the open source community.

Leave a Reply to DennisCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.