Against The Cloud

One of our writers is working on an article about hosting your own (project) website on your own iron, instead of doing it the modern, cloudy-servicey way. Already, this has caused quite a bit of hubbub in the Hackaday Headquarters. Who would run their own server in 2022, and why?

The arguments against DIY are all strong. If you just want to spin up a static website, you can do it for free in a bazillion different places. GitHub’s Pages is super convenient, and your content is version controlled as a side benefit. If you want an IoT-type data-logging and presentation service, there are tons of those as well — I don’t have a favorite. If you want e-mail, well, I don’t have to tell you that a large American search monopoly offers free accounts, for the low price of slurping up all of your behavioral data. Whatever your need, chances are very good that there’s a service for you out there somewhere in the cloud.

And that’s awesome if you only want the service provided. But what if you want to play around? Or learn how it all works under the hood? This is Hackaday!

For instance, you could run your own mail server just for your friends and family. The aforementioned search monopolist will probably flag all of your e-mail as spam, partly because they don’t trust small e-mail providers, and partly because that’s the “m” in monopoly. But if you can get folks to whitelist the addresses, you’ll be in business. And then you open up a world of fun and foolery. You can write hooks to automatically handle mail, or you can create an infinite number of mail accounts, even on the fly as per Spamgourmet, the most awesome anti-spam tool of the last 30 years. Or you can invent your own. Run a mailing list for your relatives. Or do something stupid.

I used to run a service where, when a particular account received an e-mail, the attached photo was pushed up to a website with the subject line as the caption. Instant photo-blog, of the strangest and least secure sort. Getting it running was a few lines of Bash scripting, and an afternoon of fun. Is there a service that does this, already existing in the cloud? Probably. One that allows you a little privacy and doesn’t track your every move? Maybe. But even if there is, would I have learned about sendmail by using this service? Nope!

I hear you saying “security” under your breath, and you’re right. This system was secured by lock made of purest obscurity. But still, in seven years of running the service, nobody guessed the magic e-mail address, not once. Knowledge of the e-mail address was essentially a password, but if I needed extra security I probably could have implemented it in a few lines of Bash anyway. The webpage itself was static HTML, so good luck with that, Hackerman! (The site’s been down for a while now, so you missed your chance.)

If you just want a service, you can be served. But if you want to be a server, a first-class Internet citizen, with your own cloud in the sky, nothing’s stopping you either. And in contrast to using someone else’s computers, running your own is an invitation to play. It’s a big, Internet-connected sandbox. There are an infinity of funny ideas out there that you can implement on your own box, and a lot to learn. If you hack on someone else’s box, it’s a crime. If you hack on your own, it’s a pleasure.

I know it’s anachronistic, but give it a try. (PDF, obscenity, uncorrected typos.) Be your own cloud.

58 thoughts on “Against The Cloud

  1. I run my own server to host a website, but let me tell you ….

    It was like the sunrise on a bright new day when I stopped having anything to do with mail and let a large US search monopoly do that for me. I wouldn’t run a mail server again for love or money.

    1. Unfortunately got suckered into google for domains (now G suite) many years ago when giving up on running my own services from friends and families and soon to be ending the free version. Which is great with several domains being managed. Yeahs it’s had a lot of issues / technicalities but it’s been reliable and did I mention free.
      Goggle are taking their time in deciding if they are going to continue to offer a free version.
      sadly these days we are the product and that means recurring revenue.

  2. ‘d start by having fun with something else than mail, its practically impossible to ‘convince some folks’ as the article mentioned (well you could do a LAN mail server but whats the fun in that?). Id play around with some webservers, hosting your own website (or just current working directory ;) ) from a single already built-in comand on the commandline (python, php), maybe using some dynamic dns service, try to link it to a domain name, play with user input and uploads (hehe). Never gets old. Also fun to do in an classroom setting, each students laptop being its own server, trying to acces other students in different campus locations, instant fun! Maybe even just netcat actually!

  3. It was a significant PITA, but we ran a web server on our own hardware because the site had huge data traffic (a camera test & review website, lots of downloads of the original camera files including RAW files and original video files). The bandwidth charges from any cloud service were waaay higher than what we paid for the server leases, which included a lot of bandwidth for free. It’s a somewhat unique use case, as very few websites would have that kind of traffic (terabytes/month), but we checked the costs every time we needed to migrate servers, and the cost structure always heavily favored leasing our own hardware.

    1. As someone who runs a large furry art website, trust me, I know what you mean. I think we’re over 50TB/month now (average 200Mbps+), albeit over several servers.

      Part of that is transfers between dedicated machines and VPS we have around the world to act as a CDN.

      Of course there are sites out there using way more, but maybe not many running on $10/day as well.

  4. Somewhere around 15-20 years ago I set up my own mail server just for fun and even back then it was difficult to get “official” servers to accept my mail…
    But It was just some free (maybe opensource) software running on windows (2k I think).
    And a school friend of mine tested if it was usable as a anonymous public relay – could find a way to fix that so it became a very short test.

    Still, learning how all of that worked was still fun and so on.
    Same reason I once setup a Linux terminal server for https://en.wikipedia.org/wiki/Sun_Ray thin clients (somewhere around 2011)

    -> I do recommend setting up your own server – even just for all the networking stuff you need to learn.

  5. Cloud has limitations as stated but it also has it’s place. However, there is a disturbing trend of websites that are 99.9% static content but they serve you some bloated JavaScript that composes the site instead of doing it server-side. It saves big sites a few pennies so they aren’t going to stop so long as people have JavaScript enabled.

    It seems to me that the next iteration of HTML should have a templating ability so these static-esq sites could do away with composition JavaScript while retaining a dynamic appearance. Frankly, JavaScript is a bane that needs to be killed.

    1. JS itself is quite useful, but bad webdesigners will just find a way to abuse anything. Why put a paragraph of text into a page with when they can do it with a system of fuctions from some “only 1 MB!” JS library?

  6. “For instance, you could run your own mail server just for your friends and family. The aforementioned search monopolist will probably flag all of your e-mail as spam, partly because they don’t trust small e-mail providers, and partly because that’s the “m” in monopoly.”

    And partially because “others” have ruined E-mail as a communications medium. It’s like getting snail mail, and more times than not you don’t know if it’s a letter bomb waiting to kill you. Complain about monopolies all one wants, there’s a lot of spam and *ware they deal with, so you don’t have to. That’s worth something in a life is too short to waste way.

  7. Running git locally is next to trivial. I have a separate computer for backups (that also plays videos on the TV), and got git working in about an hour. Local git is great for project backups, especially if you have a lot of projects, and a spare SSD drive is cheap.

    I also have a “git clone” of every public project I have over on GitHub.

    Also I keep copies of the images, data, and files for the projects on .IO, but not the written text. Would be nice to have a one-click “clone” command to make a local project copy, but…

    I keep copies of all the software I write for my clients. Once or twice in my career a client backup has failed and they reached out (in a panic) for copies – usually years after the contract. Click, hit “send”, and get a ton of good will in return.

    Given the current state of politics in the world, it just makes sense to have backup copies of your work, just in case. The goalposts and definitions of what is acceptable keep moving – things that were OK to say and do 10 years ago can today trigger a tsunami backlash and get you banned.

    And right now we’re seeing people banned from credit card processing, banks, crypto accounts, and all of social media. Legal business is being banned for the political views of the owner. I can cite specific examples for each of these in the past month.

    It just makes sense to keep local copies of your hard work.

    Also: SpamGourmet is definitely an awesome service, but is blocked by some carriers. I had to raise the issue (of blocking) with my carrier to a very high level in order to get them to unlock it for me. I don’t know if it’s now unblocked for every customer in my ISP, or just me. Probably the latter.

    SpamGourmet is not a temporary E-mail service, but is listed as one (erroneously) by some javascript libraries. One such library is: “ivolo/disposable-email-domains” on GitHub. The owners of that library will absolutely not listen to complaints or arguments to the contrary. Their library is popular, and lots of sites and carriers blindly use it to block E-mail.

    Because of this, SpamGourmet is no longer useful as a tool and I don’t recommend it to people. Without being able to get 3rd party library maintainers to listen to reason, your SpamGourmet account will be blocked, possibly at the carrier level (as mine was), and you’ll never see your E-mail.

    1. As a followup about SpamGourmet: When your carrier upgrades and starts to use the aforementioned library, your SpamGourmet E-mails will stop working. You get no notice, and just have to realize what’s happening.

      When that happens, you’re basically stuck. You can’t change your online accounts to a new E-mail, because it requires a confirmation message sent to the old E-mail address… which you can’t receive.

      (I had to contact the owner of the ISP directly, and then I had to convince them that these were E-mails that I wanted to receive, and that the ISP should not be blocking incoming E-mails anyway.)

      The SpamGourmet community is angry as hell over this state of affairs, but nothing can be done about it.

      If anyone has a suggestion for how to help, or can somehow convince the library maintainers to see reason, you can get on the SpamGourmet discussion board from the SpamGourmet.com main site.

  8. One thing you left out was striping off the exif info from the photos before adding to a webpage (most media sites, take a permanent copy of the “exchangeable image file format” metadata and then remove it before placing photos online). Or better yet just replacing the geotags with incorrect data, to “help” data collected by crawlers and scrapers to permanently store useless information.

  9. > And that’s awesome if you only want the service provided.

    No, it’s not, because the service WILL go away, or change in a way that makes it unusable to you, at some random time. It is foolish to become dependent on the cloud for anything really important.

    It might be OK for a hobby project that you expect to delete in a year anyway, but beyond that it’s crazy.

    1. Depends on what you are doing – there are cloud services that do little more than give you an internet facing computer and IP address you can do whatever you like with, and cloud services that are doing the basic and required things that the worst that happens is the bill for using them goes up or you have to switch when x buggers up to the y version (like a backup, you pay for the data to be accessable in the cloud, worst that happens is you have to move provider).

      I do personally hate the cloud and wouldn’t want to do anything with it really, but some cloud based things you can be sure will last for a very very long time, just by the nature of what they are, others like propriety smart home junk on the other hand…

  10. I learned datacenter-ing and web development a few years before cloud took off … And it’s the same thing. You rent a VPS, it runs Linux + some software, and you run a website on it. Maybe the database will be redundant so you don’t have to configure mirrors. It all feels the same to me.

  11. I ran a mail server, and web server, from home for a while a couple of years ago. I found a free, but trusted, mail forward service that signed my outgoing nails as trusted enough for the biggies to trust them. If you self host a web server, though, you get penalised on SEO ranking. It can be done, but not if you want broad discovery.
    My .02

  12. The article alludes to it at the beginning but then really doesn’t make a distinction between running a server at your house and using a vps. It’s clearly possible to run your own service(s) in the cloud. Running a mail server out of your house seems pointless as any downtime will cause messages to bounce.

    To me it only makes sense to host something locally when it’s going to be accessed locally, or accessed by only me remotely. It’s a lot more hassle free to just spend 20 bucks a month to host a server on a cloud provider.

    1. Yeah, totally. I didn’t make the distinction because that distinction doesn’t matter all that much. A box in your basement is just like a slice of a VPS as far as they both have Internet, and you have root.

      I guess I was thinking more of cloud services vs DIY. Because the DIY way opens up so many more doors than the simple provision of service.

      My take is that a remote computer isn’t the spirit of the cloud — it’s the abstraction of the computing resources away from the tasks that’s the whole “cloud” point. IMO. Discuss! What is “the cloud”?

  13. IMHO any data storage can only be considered trusted if the data owner can physically destroy it whenever he so desires. No cloud provider can offer this feature now and in the forseeable future.

  14. I can’t understand why anyone technical WOULDN’T run their ‘own’ email server on a hosting plan ie not on bare metal. There are so many advantages compared to a gmonopoly.com account – one of the biggest being that you make a forwarder for every site that you use an email for a login, so that a) they are all different, and b) you can tell where the spam is coming from (and delete it automatically).

    Sure, have one of those monoploy accounts as well…

    And I also think many tech people would run a server at home – I certainly do – for file sharing, mariadb, and a few other interesting things. .

    1. A good reason to not use a hosting company / VPS is privacy. You have zero control and zero oversight over VPS / cloud / hosting offerings.

      Granted, most people don’t care much about privacy, and most people don’t know enough about security to make running their own services better than VPS / cloud / hosted. But for some of us, it’s how it has to be.

    2. I have/had that with google for domains. but they are closing it down very soon.
      running your own mail server is another dedicated task. I dont want to deal witn the tech for fun, I want to be doing other things.

  15. I actually have a rack of 9. I put them all together with as much hand-me-downs and, throwaways as I could muster. Even the enclosed server rack case was saved from the landfill. I did it over a period of a couple of years. I add and upgrade here and there. I think the total amount of cold paper I put into it was “Maybe” $75 USD. I learned a ton not only about Software but, the hardware and networking as well. I can host my own stuff, run databases, email. What ever I want and, I only have to pay for the Electron Flow. Biggest bonus though? It’s MINE. Nobody has access to anything unless I give it to them. I retain full control over anything I have written or designed.

  16. It’s not just that big search monopoly that won’t accept email from your home server. It’s ALL of the big players now. 95% of the email addresses you try to send mail to will flag your messages as spam. And because home addresses are a common source of spam, some of the usual blacklists include all the address blocks known to be associated with home service, so even fellow small email servers may not accept your mail.

    The other problem is that if you have typical home internet service, your ISP will stop you from running a server. It’s against their terms of service, and if they detect you running a public-facing server they’ll block it if they haven’t already done that. You’ll have to upgrade to a much more expensive business internet service to be allowed to run your own server.

    One sort-of-way around the latter problem is to set up shop on a VPS server somewhere rather than actually running your own iron. It’s not quite the same experience, but you still get to install your own OS, configure your packages, and so forth. And since it won’t be at a home IP address there is at least some hope of getting the major email providers to accept mail from you.

  17. Much as I consider it an important tool for both learning & democratisation of the internet (it was a big part of my journey too), the concern for security isn’t just about your security. It’s also the security of everyone else on the internet. If you’re on the internet, as with any environment you’re in, you have a moral obligation to not endanger other people. And unfortunately running your own server on the internet is really hard to ensure that these days, while being a much greater potential danger. I’m not saying I don’t think people should do it (no gatekeeping), but I do think it’s important to be aware of this & accept the responsibility.

  18. I did previously run a server at home and I still would if I had broadband but right now I just have a mobile connection and an online server.

    I have had a (shared) online server for decades. It’s a reseller account that costs very little. I don’t resell, I just use it personally. Some times I play with a website. I was previously a developer: php, *SQL, Apache, HTML, JavaScript, CSS etc. But it is mostly just private email now. I used to manage many backbone servers so backup systems, zone management, NS/DNS etc

    I saw a comment about email from a private sever being rejected. Unless you have an open relay it will be the extra TXT data on your DNS (Zone) files. They have to be done well now to be excepted, you can read up on that.

    I would definitely run an IoT server at home. I will be doing this shortly to steal back provider dependence for my video doorbell. This is where us hacker really need to make ourselves known.

    For services fro a home sever to the net you need to get either a dynamic DNS service or a paid for service. I would opt for the paid for service once you finished playing with the free ones especially if you want to have email sever.

    For locals services like IoT read up on mDNS. It exists on most platforms even on the LUA based ESP32 platform. This will get your local namespace working without a fully fledged DNS service.

    Lastly, this is all easy for me, nothing is guesswork, everything is specified in the RFC and WWW3 helps.

    The reason it’s easy for me is that I already know it’s easy. It’s easy for you to, you just don’t know that yet.

    It’s a good old case of RTFM.

    1. I have two mail servers, hosted by the same VPS provider. One is based on a domain name that I’ve hosted mail on for the last 20 years, as a box in an apartment, through two different VPSs. That one gets and sends mail just fine.

      As a lark, I spun up a mailinabox server with another domain name. Same VPS host (Digital Ocean, FWIW) as the other. I spent a few hours trying to dot all of the DKIM i’s and SPF t’s. It passes all of the online “check out your server” sites. And still mail from this domain ends up in the spam folder on Google and GMX, while it works just fine from the other. WTF.

      Email is a headache if you involve the big providers who take very aggressive anti-spam measures — because their practices are opaque. That’s why I was suggesting it for personal / friends use — I had a bad time recently.

      (Still willing to look into a run-your-own-mail guide, if anyone has any good resources.)

      But mail isn’t really the point. It’s having a box that you control, like you say. Lets you do the learning / playing.

      1. I’m using DigitalOcean as sort of a “router” for my domain. I don’t have any trouble with Google, but I do with Yahoo. It seems some anti-spam lists just put ALL of Digital Ocean’s IP addresses on their list because of a few bad apples. As for Google, I registered with some sort of Google service to make my domain a “trusted” email source. Or something like that. I think I put a magic token in a DNS record and Google doesn’t block anything anymore.

        As for a “run-your-own-mail guide”, take a look at iredmail.com. I ran his script years ago (after looking it over for malware) which installs and configures a bunch of open-source applications (postfix, dovecot, RoundCube, etc.) to make a dedicated email server. He also publishes instructions a couple of times a year to upgrade from one “version” to the next. It’s always been very reliable.

        For some domains, notably Yahoo mail and anything from Microsoft (Hotmail, Outlook.com, live.com), as well as various ISPs, I have to route through free or cheap SMTP relay services to keep my mail out of the spam folders or to even get their mail servers to talk to me at all (currently using MailJet and DNS Exit). Warning: MailJet seems to add a little image turd to the bottom of the emails so they can track reception and reading of the mail. I plan to try AWS to route outbound mail through. It’s fairly cheap, but I doubt AWS will get blocked by many servers and I don’t think they’ll add tracking cookies to the outbound mail. I do hate to give Amazon the opportunity to read my outbound mail, though…

  19. Another element that makes running an email server at home impossible most of the time: all major ISPs block port 25. Plus, you’ll never be able to set the correct reverse DNS. Additionally, every major RBL and DNSBL has all the residential IPs banned.

    $5/mo at a reputable VPS provider is well spent.

    1. Running a mail server at home is not problematic.
      You simply need to use your ISP’s mail server as a smart rely.
      If port 25 is block there are services that fix that as well.

      Why not use the Monopoly player that;s free — Well for me I am a HUGE fan/user of IMAP and the folder structure it allows. The search giant monopoly is a HORRID imap implementation IMHO.

      The other problem with “cloud” offerings that are free is they are free till the company decides their not. You are at the mercy of someone else who can change the rules at will. Look at the recent change from the search monopoly with the free storage provided to education market — was unlimited now 100TB per institution. For an institution that has 5000 student and 1000 or so employees that’s nothing per person (20GB) Might as well just us a flash drive.

      Yes I do have a monopoly account that I use as well — but that’s to seperate out my mail.

      1. It’s true that you can use your ISPs email service as an outbound relay but then your SMTP authenticated username and your email address don’t match which lowers the amount of trust given by other mail servers. And now your email is going to spam. Also where you going to do for inbound mail? Again, port 25 is blocked.

        I mean really if you want to simply run the mail service at your house and rely on other services to give you access to port 25 on the internet then you’re not really hosting your own email on your home connection. It would be trivial to VPN or even just SSH tunnel from my home server to my VPS and map port 25 that way. But that defeats the purpose doesn’t it? Unless you just want the hardware at home. Then I suppose that’s fine. But if your goal of self-hosting is to be self-reliant then it kind of defeats the purpose.

        1. You ISP assigned authentication username (in this case your email address) is not forwarded past your ISP.

          When ISP’s block outgoing SMTP port 25 they usually have an alternative port (usually 26) because they just want you to ring them first because a spammer isn’t going to do that. Sometime they need to manually open a port but most ISP’s are happy to do that. Most ISP’s will have their SMTP setting on their website even.

          Incoming is different in some ways. POP, POPs, IMAP, IMAPs (or any other servers / services) are on different port numbers to start with (not 25 or 26). I think the days have past when most ISP’s will expose your connection directly to the internet (DMZ). You will have to read their policy and if they allow it then you will probably have to ring them. Some will even sell you a public IP.

  20. as a developer, I am still using on premise systems to develop for the company I am working for. But for my colleagues that run their own company and design b2c products , it transformed from programming the “metal” to programming the “data center” , a new concept is data center as computer. There is a YouTube video in Turkish prepared by a friend whom I respect about his engineering skills : https://youtu.be/YfVBym-RRAI, to scale your solutions cloud is an option

  21. Hosting your own email is only a little more complicated than it used to be. Due to spam issues, you should use an SMTP relay service to send mail. Your SMTP server will then send through this service rather than trying to send mail directly. There are some free ones available if your message volume is low, they just verify that you are the domain owner in various ways.

    I also suggest a spam filtering service as a relay for incoming mail. I use MX guard dog. They are super cheap, the spam filtering is effective, and it helps if your IP changes frequently or your ISP blocks port 25. Basically your configuration there will specify your home email server IP or domain name and the port to use to send mail. It will also queue all messages until it can reach your server, in case your IP changes suddenly or your server goes down for some reason.

  22. I do this on an ODROID HC1 with a USB-to-ethernet adapter to provide a second ethernet port.

    I host email (Dovecot and Postfix), a static website and webmail (Cypht) using php-fpm and lighttpd.

    I also run a XMPP server (Prosody) with voice and video call support via a STUN/TURN server.

    Email delivery can be problematic (I run an upstream SMTP relay with reverse DNS on a VPS provider) but despite implementing DMARC, DKIM and SPF, Hotmail and Outlook accounts tend to deliver my email to the Spam folder. GMail seems to accept all my email though.

    Just recently I’ve started receiving DMARC aggregate reports from Microsoft so maybe the situation with Hotmail and Outlook will improve.

    The responsibility of running my own services securely hasn’t escaped me but, touch wood, I’ve been doing this since 2001 without incident. My ISP has even stopped probing my receiving SMTP server to see if it’s an open relay.

    Is all this effort worthwhile? For me it is because I like to manage all my own communication without putting it in the hands (or servers) of other parties.

    1. “Is all this effort worthwhile? For me it is because I like to manage all my own communication without putting it in the hands (or servers) of other parties.”

      Noble, but in this whole debate let’s not forget the nature of communications means it’s partially in the hands of others. The internet wouldn’t exist otherwise. For email to be useful it has to be to someone else with varying degrees of trust, and competency (secure on your end, leaks on the other).

  23. The issue with “free” web hosts is they might not be free or even in business tomorrow. You’ll spend more time moving your backup to a new “free” web host than just running your own web host.

  24. Been doing this for over 20 years now. Spamassassin, fail2ban and cron jobs are my friends.
    I MIGHT hesitate were I starting today, but now that it’s running and has been for a long time, I hesitate ti giver it up.

  25. This is why I started CollectiveFS.com a few days ago. I want o create a public file system where we can store personal files. Basically you create a folder and anything you put into that folder gets ran through reedsolomon erasure coding which allows ~35% loss (percentage is user configurable). The data+parity chunks are encrypted using gpg then distributed to other peers via WebRTC. Directory tree can be stored in json.

    The system is “balanced” because the amount of space you use on the network is how much space you provide for other users to store chunks on your drives. I have been studying other architecture’s like Hadoop (HDFS) and IPFS but this is fundamentally different. Hopefully it will be as simple as creating a git repo inside a folder.

  26. I don’t do email but an old PC stuffed with cheap HD’s and sat in the (detached) garage gives me a remote-ish backup server, Gitea instance for projects, Dokuwiki for notes, Transmission (torrent) client, and media server for my OSMC Pi stuck behind the TV.

    Our home PC’s rsync to it and periodically it does a bulk FTP download of my (hosted) websites as a very basic backup.

    Total cost is a little bit of electricity and a couple of hard drives. PC came from the work scrap bin.

    Setting up OwnCloud or Syncthing is on my To-Do list, one of these days…

  27. I run my own email server and almost never have to touch it. I originally set it up through the script offered by iredmail.com and I keep it up-to-date with his updates. I block swaths of IP addresses from certain counties that I never expect to get actual email from (India, Vietnam, etc.) and the amount of spam I get is tiny, especially after setting up SPF. The only problem I have with my email being flagged as spam is with Yahoo, anything Microsoft, and some ISPs (why do people still use their ISP for email???). Apparently some spam block lists just include all of DigitalOcean’s IP addresses (the frontend for my domain). So I get around those by using a cheap relay service when mail is going to those problematic domains.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.