If you’ve followed along with our series so far, you know we’ve set up a network of Raspberry Pis that PXE boot off a central server, and then used Zoneminder to run a network of IP cameras. Now that some useful services are running in our smart house, how do we access those services when away from home, and how do we keep the rest of the world from spying on our cameras?
Before we get to VPNs and port forwarding, there is a more fundamental issue: Do you trust your devices? What exactly is the firmware on those cheap cameras really doing? You could use Wireshark and a smart switch with port mirroring to audit the camera’s traffic. How much traffic would you need to inspect to feel confident the camera never sends your data off somewhere else?
Thankfully, there’s a better way. One of the major features of surveillance software like Zoneminder is that it aggregates the feeds from the cameras. This process also has the effect of proxying the video feeds: We don’t connect directly to the cameras in order to view them, we connect to the surveillance software. If you don’t completely trust those cameras, then don’t give them internet access. You can make the cameras a physically separate network, only connected to the surveillance machine, or just set their IP addresses manually, and don’t fill in the default route or DNS. Whichever way you set it up, the goal is the same: let your surveillance software talk to the cameras, but don’t let the cameras talk to the outside world.
Edit: As has been pointed out in the comments, leaving off a default route is significantly less effective than separate networks. A truly malicious peice of hardware could easily probe for the gateway.
This idea applies to more than cameras. Any device that doesn’t need internet access to function, can be isolated in this way. While this could be considered paranoia, I consider it simple good practice. Join me after the break to discuss port forwarding vs. VPNs.
For the Lazy: Port Forwarding
There are two broad categories of solutions to the problem of remote access, the first being port forwarding. Simply forwarding the assigned port from your router to the server is certainly easiest, but it’s also the least secure. The logs of a port 80 (HTTP) or 22 (SSH) exposed to the internet are frightening to sift through. And before you assume you’ll never be found in the vast sea of open ports, I present Robert Graham and masscan. Able to scan every IP address for a given port in just a few minutes, you’re not hiding that web server for long.
Stories like the libssh vulnerability are constant reminders of the dangers of leaving services open to the internet. If you opt to go this route, you must stay on top of security updates, use very strong passwords, and hope for the best. Fail2ban, virtualization, and good backups are all invaluable tools for standing up to the onslaught that is the public internet. Suffice it to say, I don’t recommend this approach for a home user.
One of the alternatives is to simply use a different external port number. Port 8080 for an HTTP server is a popular choice, which makes it a bad choice when trying to hide the service. Using a random port between 10,000 and 65,000 is a strategy known as security by obscurity. Someone with a grudge who knew your IP address will find it. But random scans through every possible port at every possible IP address make finding this more of a needle in a haystack problem.
Another technique that falls under the port forwarding category is port knocking. A series of TCP SYN packets (initial connection attempts) sent to a predetermined list of ports signals to a listening daemon which then allows your connection through the firewall. While clever, port knocking is subject to replay attacks and a few other problems. If the cleverness of port knocking appeals to you, there is a more modern take on the idea in the form of Fwknop (Full disclosure, I do some of the development on this project).
For Peace of Mind: VPN
Another approach to secure remote access is a VPN. A Virtual Private Network has been described as stretching a ridiculously long Ethernet cable through the internet. Traffic is encrypted between the endpoints, and because a VPN creates a virtual network adapter, a remote user can access the network as if they were physically connected to it.
OpenVPN is the time-tested contender in the open source VPN space, and has a good track record. It’s based on TLS, can run over TCP or UDP, and is widely used. There is good support for Android, iPhone, Windows, MacOS, and Linux. The configuration can be a challenge to work through, but once it’s up and running, OpenVPN is a great solution.
Wireguard is a much newer project, and aims to be simpler than OpenVPN. I’ve opted to use Wireguard, which has good Android and Linux support. In fact, Wireguard has been making steady progress towards being officially included in the Linux kernel. There is already macOS support through MacPorts and Homebrew, and their iOS app just entered beta. Wireguard uses more modern cryptography, aiming to be faster and more secure than the alternatives.
Both VPNs are supported by OpenWrt, and there are advantages to hosting the VPN on the router. When hosting a VPN on another machine on the local network, I have two pieces of advice, as these two problems trip me up repeatedly. The first is to enable IPv4 forwarding on the VPN machine. The second is that other devices on the network must have routes configured to get to the VPN network, in order for traffic to flow in either direction. Without that static route, packets addressed to the VPN are sent to the default router instead of the VPN server.
Running Silent
One final consideration is whether you want your IP address to be completely silent. Called a “black hole”, or “stealth mode”, this is when your IP address doesn’t respond to pings, and instead of responding to connection attempts with ICMP error packets, those incoming packets are dropped silently. The goal here is that to a network scan, your IP address appears to be unassigned with nothing listening for traffic.
OpenVPN and Wireguard both support this style of quiet operation when in UDP mode. In the case of OpenVPN, a Hash-based Message Authentication Code (HMAC) allows incoming UDP packets to be quickly dropped when they aren’t signed with a valid HMAC. Wireguard was written with this sort of radio silence as a design goal, and doesn’t require any extra configuration to enable it.
Why It’s Worth Thinking About
This has obviously been an overview, and there are many clever ideas we haven’t touched on. I often turn to Fwknop combined with SSH tunneling when I don’t want to set up a full VPN. IPSec via the Strongswan project is still a valid solution, and there are many others.
Most important, don’t thoughtlessly let the entire world try logging in to your house. Part of the appeal of hacking a house is to be able to monitor things while on vacation, but you don’t want to always feel like somebody’s watching you!
Up next, we’ll look at a networked garage door opener, and talk about the ways to work around a vendor’s proprietary protocol, and maybe a primer on how to reverse engineer it.
If I were writing a malicious firmware for an IP camera I would totally make it problem for a default gateway at the usual addresses even if it wasn’t given one… You really ought to either put them on a truly isolated network or put in exlicit firewall rules prohibiting the gateway from NATing outbound connections from the cameras in case the DO try and sneakily discover and use the default gateway they’ve been intentionally not told about.
You’re absolutely right. I’ll add a quick note to the article pointing that out.
By looking at arp traffic the default gateway even at a ‘nonstandard’ .1 or .254 can be found in seconds. The host that gets the most whohasip packets is generallly the gateway.
Heh, yeah, I had to do that a few years ago, just before Christmas, when when my flight had been cancelled and my home internet connection wasn’t working. I went into a cafe that said ‘free internet’ on the door, bought a coffee and asked for the wifi password. They gave me the password, but there was no DHCP and no one in there seemed to know anything about it, so I looked through the arp traffic. Managed to get my flight re-booked for the next morning :-)
hacker!!!
telnet to spirit-plumber.ddns.net if you dare :)
I’m curious that you wrote “You can make the cameras a physically separate network, only connected to the surveillance machine, or just set their IP addresses manually, and don’t fill in the default route or DNS” you don’t really have to physically separate the networks in order to have separate networks that solve this problem. As another commenter has already mentioned a malicious device could already circumvent a simple lack of being told about a gateway.
At some point, I’m hoping to talk about Vlans as a way to accomplish this. And yes, simply not setting the default route isn’t a deterrent for a truly malicious device. I think it’s better than not doing anything, though.
They do a good job explaining. Problem is the slightly non-standard way some makers do things. Makes VLANS harder than they need to be.
https://www.smallnetbuilder.com/tags/VLAN
The theory behind both standards us the same. However most switches implement .1q as tagging.
Configuring them on cheap netgears is a nightmare :)
VLANs alone are not necessarily the answer, as I’m sure you know. If the networks you have on your VLANs are routed at L3, you can end up right back in the same boat. Couple of ways around it:
a) non-routable VLAN. Zoneminder (or whatever) has ports into both your main network and your camera VLAN (either physical or 802.1q trunk). There is no router present on the secure network, therefore no routing (and therefore no access to the internet).
b) routable VLANs + firewall. You VLAN can be routed to, but must pass through a firewall (or at least an ACL) that filters traffic such that only your Zoneminder host (or whatever) can talk to it. Internet is firewalled off.
For a lot of folks that aren’t network savvy, it’s simpler, less error prone, and all around safer to use two different physical switches, and ensure they are not connected to each other. The “secure” switch has no uplink to the internet. Then the host that needs to talk to secure devices has two links, one to each switch. The concept is the same, but you’re not dependent on ensuring correct VLAN configuration to maintain security.
Disclosure: I say this as a guy with an extensive home network with multiple routed subnets on separate VLANs, one of which is ACLed off (I’m too cheap to buy a gigabit-capable firewall) from all but one host on the main network to act as my secure IoT network (mostly unused as yet, but that’s neither here nor there). VLANs are extremely useful if you know how to safely use them, but they’re one of those things that can make a beginner feel secure when they really aren’t, so caution is recommended.
Yes, isolated VLANs are appropriate for any devices you do not want accessible from the outside. You need to ensure your switch will not layer 3 route between the VLANs. Obscuring the gateways and lack of default route is NOT sufficient. If proxy ARP is running, that would be completely useless and in any case it would not be hard to get around. Another piece of advice that I think will cause headaches is setting your devices to use non-standard HTTP/HTTPS ports. The issue is that if you are at work or on one of many guest networks, they may not allow you to contact your home device because their firewalls may not allow an outbound connection on non-standard ports like that. Just something to be aware of.
After your last article, I tried ZM with 2 new 4k cameras. *Each* zmc process used 1.2GB of RAM and 100% of a single i7-4790K CPU thread. ZM is definitely not the solution I’m looking for…
4K cameras! Yeah, doing the mjpeg translation alone would be murder. I’d probably set those cameras to 1080p at 20 or 30 fps. Should get the ram and processor usage under control.
Zoneminder isn’t ready for 4K yet. They’re working on it. One of the keys will be when you can do live views and recording with a pure h.264 workflow. Right now, they use mjpeg to display the live view in the browser.
Might want to check out BlueIris for Windows, although I’m not sure how well it does with 4k. One option is to leave the cameras encoding alone and just store it directly on the drive. Free to try for 30days.
I would like to switch to bluecherry on a Linux machine, but I find their lack of mobile notifications disturbing…
Kudos for Red October reference.
I loved that, too! Props to Mike for adding that during editing.
Yeah, that’s not a thing though. As much as GRC and other FUD peddlers want you to believe it, there is no such thing as “stealth”. It’s easy to tell the difference whether an IP is not connected (i.e., the upstream router returns ICMP “destination unreachable”) or just not responding (i.e., packets time out). Dropping packets instead of rejecting them does exactly the same — it tells the attacker that there is a firewall — and will not increase your security in any way.
If you’re not on the same network as the IP you’re probing, then it absolutely is a thing. Not all ISP routers return the destination unreachable ICMP response for unassigned IPs.
Except that this has the exact opposite effect. Temporary measures to mitigate excessive ICMP load notwithstanding, an upstream router should return “destination unreachable” if an IP is not connected. Anything else will tell the attacker that there is some filtering in place, and may encourage a more thorough probing. Dropping connections really is only the thinnest veil of “security by obscurity.”
About 12 years ago I was chatting with the I.T. guy at my dad’s office, and the topic of hiding IPs came up. And apparently its kind of useless. He was telling me about a test/fun server he had blackholed on the work network, outside of the firewall, and it was nice and hidden for less than a week before it started getting pinged by a few addresses in China. Interestingly, from what I remember, this wasn’t a broad search style ping that goes after everything, but only directed to the single machine and ignoring the rest of the network behind the firewall. That conversation really cemented in me the idea that obscurity does not exist on a computer. Even while staying completely silent several people/organizations in China managed to find this blackholed box sitting outside the firewall of a mid-sized engineering firm.
Interesting. Was it a single address blackholed, while the surrounding addresses were responding with deny icmp responses?
Unfortunately thats all I remember of the conversation. But it definitely left a strong impression on me that active security is the only way to go. If you have anything worth securing, that means somebody might want it and therefore you’re not obscure.
You really only have the option of isolating the cameras to their own switch, and then trying to find some “lesser-evil” compromise by using a middleware layer like zoneminder/blueiris that can offer a buffer between the unsecure and secure networks.
Something like 2fa for connections to the middleware may give additional security in the weird event that the cameras are actively trying to compromise your network, but that just makes entire other levels of paranoia where you want to start monitoring traffic on your unsecure network.
All of this talk of compromised cameras though, makes me want to look at things like the rpi cameras and build my own POE cameras.
At least then if it is trying to hack my network it is my own fault…
pfsense makes an OpenVPN setup dead simple in addition to allowing you to do any kind of interception of traffic you want. It’s FreeBSD with a nice webif and the pf firewall from OpenBSD. You can run it on super cheap or low powered hardware too.
“You could use Wireshark and a smart switch with port mirroring to audit the camera’s traffic”
Or use a dumb hub, but they don’t make those anymore…
For a short cable run you can connect two nics in parallel. Not ideal but it works, usually. In case of gigabit, make sure to not transmit with the card as it used all pairs.
When can we expect the networked garage door opener post?
It’s coming. Had some real life stuff happen, and just now diving back in to writing.