The Internet has spoiled us. You assume network packets either show up pretty quickly or they are never going to show up. Even if you are using WiFi in a crowded sports stadium or LTE on the side of a deserted highway, you probably either have no connection or a fairly robust, although perhaps intermittent, network. But it hasn’t always been that way. Radio networks, especially, used to be very hit or miss and, in some cases, still are.
Perhaps the least reliable network today is one connecting things in deep space. That’s why NASA has a keen interest in Delay Tolerant Networking (DTN). Note that this is the name of a protocol, not just a wish for a certain quality in your network. DTN has been around a while, seen real use, and is available for you to use, too.
Think about it. On Earth, a long ping time might be 400 ms, and most of that is in equipment, not physical distance. Add a geostationary orbital relay, and you get 600 ms to 800 ms. The moon? The delay is 1.3 sec. Mars? Somewhere between 3 min and 22 min, depending on how far away it is at the moment. Voyager 1? Nearly a two-day round trip. That’s latency!
So how do you network at these scales? NASA’s answer is DTN. It assumes the network will not be present, and when it is, it will be intermittent and slow to respond.
This is a big change from TCP. TCP assumes that if packets don’t show up, they are lost and does special algorithms to account for the usual cause of lost TCP packets: congestion. That means, typically, they wait longer and longer to retry. But if your packets are not going through because the receiver is behind a planet, this isn’t the right approach.
Upside Down
DTN nodes operate like a mesh. If you hear something, you may have to act as a relay point even if the message isn’t for you. Unlike most store-and-forward networks, though, a DTN node may store a message for hours or even days. Unlike most Earthbound network nodes, a DTN node may be moving. In fact, all of them might be moving. So you can’t depend on any given node being able to hear another node, even if they have heard each other in the past.
Is this new? Hardly. Email is store-and-forward, even if it doesn’t seem much like it these days. UUCP and Fidonet had the same basic ideas. If you are a ham radio operator with packet (AX.25) experience, you may see some similarities there, too. But DTN forms a modern and robust network for general purposes and not just a way to send particular types of messages or files.
The Bundle Protocol
While the underlying transport layer might use small packets — think TCP — DTN uses bundles, which are large self-contained messages with a good bit of metadata attached. Bundles don’t care if they move over TCP, UDP, or some wacky RF protocol. The metadata explains where the data is going, how urgent it is, and at what point you can just give up and discard it. The bundle’s header has other data, too, such as the length and whether the current bundle is just a fragment of a larger bundle. There are also flags forbidding the fragmentation of a bundle.
In Practice
DTN isn’t just a theory. It has been used on the International Space Station and is likely to show up in future missions aimed at the moon and beyond.
But even better, DTN implementations exist and are available for anyone to use. NASA’s reference implementation is ION (Interplanetary Overlay Network), and it is made for NASA-level safety. It will, though, run on a Raspberry Pi. You can see a training video about ION and DTN in the video below.
There are some more community-minded implementations like DTN2 and DTN7. If you want to experiment, we’d suggest starting with DTN7. The video below can help you get started.
Why?
We hear you. As much as you might like to, you aren’t sending anything to Mars this week. But DTN is useful anywhere you have unreliable crummy networking. Disaster recovery? Low-power tracking transmitters that die until the sun hits their solar cells? Weak signal links in hostile terrain? All of these use cases could benefit from DTN.
We are always surprised that we don’t see more DTN in regular applications. It isn’t magic, and it doesn’t make radios defy the laws of physics. What it does is prevent your network from suffering fatally from those laws when the going gets tough.
Sure. You can do this all on your own. No NASA pun intended, but it isn’t rocket science. For specialized cases, you might even be able to do better. After all, UUCP dates back to the late 1970s and shares many of the same features. Remember UUCP schedules that determined when one machine would call another? DTN has contact plans that serve a similar purpose, except that instead of waiting for low long-distance rates, the contact plan is probably waiting for a predicted acquisition of signal time.
UUCP Redux
But otherwise? You knew UUCP wasn’t immediate. Routing decisions were often due to expectations of the future. Indefinite storage was all part of the system. Usenet, of course, rode on top of UUCP. So you could think of Usenet as almost a planetary-scale DTN network with messages instead of bundles.
A Usenet post might take days to show up at a remote site. It might arrive out of order, or twice. DTN has all of these same features. So while some would say DTN is the way of the future, at least in deep space networking, we would submit that DTN is a rediscovery of some very old techniques when networking on Earth was as tenuous as today’s space networks.
Security
We’re sure that by modern standards, UUCP had some security flaws. DTN can suffer from some security issues, too. A rogue node can accept bundles and silently kill them, for example. Or flood the network with garbage bundles.
Then again, TCP DoS or man-in-the-middle attacks are possible, too. You simply have to be careful and think through what you are doing, if it is possible someone will attack your network.
Your Turn
So next time your project needs a rough-and-tumble network that survives even when you aren’t connected to the gigabit LAN, maybe try DTN. It has come a long way, literally and figuratively, since 2008. Well, actually, since 1997, as you can see in the video below. Whatever you come up with, be sure to send us a tip.
DTN wikipedia link has an extra “]” at the end
Oops. Fixed.
In some places it’s currently not uncommon to have jitter in the second range even now, and TCP handles it just fine most of the time. In fact it’s recent services that tend to be inflexible about delay, expecting high bandwidth, low latency, and low jitter or failing in irritating ways.
DTN addresses these problems well, but the software communicating through it also behaves flexibly, I would say that’s a more important takeaway than the basic aspects of the DSN’s protocols.
Still, fun stuff.
From this article, it sounds like DTN might use some “message queueing” technology, like MQTT or IBM’s MQSeries.
Looks like I’ in for an afternoon of reading up on DTN.
Thanks
I think you may have forgotten to double those numbers for lunar and martian (!) round-trip ping.
I meant hearing a ping from one side to another, not a round trip, but in retrospect that’s not very clear, I admit.
Weak signal links in hostile terrain. -> Weak signal links in hostile terrain?
laws -> flaws
It’s also worth noting that while humans won’t be travelling to another planetary body anytime soon, there’s plenty of hardware out there for which extreme delay is already a problem.
You have to read the sentence before. “It isn’t magic, and it doesn’t make radios defy the laws of physics. What it does is prevent your network from suffering fatally from those laws when…”
So laws are not defined but not fatal. So yes, laws not flaws.
Good high-level summary of DTN and ION! A few comments:
1) DTN was a low-level plot element in the science fiction series The Expanse. The delays in interplanetary communications were mitigated via space relays that stored and forwarded communications either via RF (‘wideband’) or laser (‘narrowband’). Every spacecraft had to have a transponder, so that their location was known and messages could be communicated. In the Expanse universe, changing the transponder ID (e.g., from MCRN Tachi to Rocinante) was risky business due to security protections and could result in disastrous consequences.
2) I’ve been using ION DTN off and on since 2015. Apparently, according to Scott Burleigh (ION lead engineer), I was the first to get ION running on a couple of Raspberry Pi 1s and a MIPS CI20 board. It was not especially difficult to accomplish, which is a testament of the quality of the ION codebase. I constructed a small ‘network’ of ‘flatsats’ (single board computers) that transferred data to a ‘ground station’ (laptop computer). The ‘ground station’ was also set up as an NTP server to synchronize time with the ‘flatsats’. A Contact Plan was set up to enable scheduled routing of data. Later, I implemented multiple satellite and ground station nodes in a VM environment using NASA DTN Kit, which used the NRL (Naval Research Laboratory) CORE (Common Open Research Emulator).
3) Creating a Contact Plan is one of the challenges in making ION DTN work. It is simplified somewhat, for satellites in predictable orbits and ground stations. A knowledge of celestial mechanics, a common positional reference system, and a shared time base make Contact Plan generation relatively straightforward. The resulting Contact Plan is general in that all nodes are equally capable for routing of bundles. Practically, this can be problematic if some nodes cannot handle the bandwidth for storing and forwarding the bundles or should not receive some bundles due to security concerns. Tailoring of Contact Plans can be performed to restrict nodes so that only those bundles destined for them are transferred. I have two patents on Contact Plan and Tailored Contact Plan generation.
Very cool. Thanks for sharing that.
TCP was never designed for a scenario like this. Except for first-and-last link failure, in which case there is nothing you can do, the TCP congestion assumption is always the correct one. The assumption is always that there will be redundant pathing for TCP and UDP (routing protocols underneath TCP). This was the whole idea behind the TCP/IP stack…creating a protocol that can deal with reduntant paths (military use in the initial implementations).
DTN will be interesting to follow, as delay will only escalate as we start leaving the solar system…up to years or even centuaries of store-and-forward capabilities. Personally I believe that we will never communicate at those distances with our current protocols and technology. We will have to figure out faster-than-light communications…way faster than light, if we are ever to leave the solar system.
I thought TCP/IP and its radio packet equivalent took care of this problem a while ago… one of the “v.” protocols.
Nothing to do with TCP/IP…those are media access protocols…OSI layer 1.
Yes, Vint Serf designed and built such a thing in the 1990s, if I remember right, later to be used by NASA to talk to Mars rovers among other things.
Seek his youtube presentation on such. I believe it was called “Interplanetary Internet”.
Can we use that for Bitchat and transfer messages and files over bluetooth links?
“On Earth, a long ping time might be 400 ms, and most of that is in equipment, not physical”
If you’re getting 300ms+ (you didn’t specify what ‘most’ is) latency due to equipment you’re seriously doing something wrong! Like downloading something with multiple independent streams maxing out your connection then wondering why your latency has gone through the roof. That’s rather an edge case though.
They could use Packet Radio/AX.25, too, I guess?