SSH Enters the Mosh Pit

With so many systems depending on Linux, the secure shell SSH has become a staple for many developers. If you are connected to your Raspberry Pi via a cable or a wireless router a few feet away, SSH can provide you with an encrypted connection straight to the box. However, if you have a system out in a swamp somewhere with intermittent slow network access, SSH can be a real pain. When your IP address can change (for example, roaming on a cellular network), SSH has problems, too.

To combat these and other problems, you might consider an open source program called Mosh (mobile shell). There’s two parts to Mosh. One part works as a server while the other is the client application. Neither of these require root access. You can see a video about Mosh below.

Using Mosh is pretty similar to using SSH (and, in fact, you can use SSH to initially launch Mosh). The Mosh server has a system that synchronizes the screen, so it doesn’t waste time sending characters that won’t really appear to the user. In addition, the client keeps its own notion of the screen and it attempts to handle some operations locally. For example, in SSH every character you type echos through the server. While this is useful to know the server is there, it isn’t good for responsiveness. Mosh handles echo locally and also predicts certain things like insertion and deletion of characters. Unlike SSH, Mosh uses UDP datagrams so packet loss isn’t a big deal.

In fact, if you switch networks or if your computer sleeps and wakes up, Mosh will stay “connected.” Because Mosh doesn’t send data you won’t see, you can easily interrupt a long scrolling output (Mosh selects a frame rate for scrolling based on the network connection’s quality).

Seeing the Future

When you type something in Mosh, it might make a prediction. If the network isn’t robust, Mosh will presume that most characters will appear at the current cursor location. It also expects certain behavior of keys like backspace, right arrow, and left arrow. Of course, there’s a chance it is wrong, so it underlines guesses until they are proven right. The software gathers batches of predicted input (epochs) that it validates (or refutes) as a group. Certain characters (like the up and down arrow) heuristically start new epochs.

For example, if you start typing inside emacs (or vi), you will start building an epoch that does not display on the screen. When data comes back from the host computer that validates the guess as correct, Mosh will display the epoch and continue to display its guesses until something (like an up or down arrow) changes the epoch. Then it will go back to not performing the local echo until it is once again correctly predicting. All this works well, but you can turn predictions off if you wish.

If you want to see the behavior firsthand, open up a Mosh session and start typing on a shell prompt. For example, consider you are going to enter this command:

find / -name 'mosh*' -print

Type up to the first slash. Now disconnect your network (pull the cable or shut off your wireless). Type some more of the command and you’ll see the speculative characters show up underlined (see below; the red cursor indicates the connection is currently not up). Now either reconnect the network or turn on your phone’s hot spot or otherwise connect to a different network. Finish typing the command and it should all work.


The Kitchen Sync

Mosh also claims to solve a host of other display-related problems stemming from the use of Unicode and escape characters. However, the real engine is the synchronization piece which can actually keep any abstract object in sync across two devices. Mosh uses two instances of this sync protocol (SSP); one to keep the screen buffer and one to handle the input buffer. The reality is that Mosh is just a terminal emulator built over SSP. It seems like SSP could find use in other applications where two machines need to stay in sync over a low-quality network.

Mosh is available on a surprising number of platforms. In addition to the usual suspects, you can get Mosh as part of the JuiceSSH client on Android. Windows users can use Cygwin or a version that runs within Chrome. It also runs on a Mac or any of several Linux and BSD distributions. Of course, you can also build it from source. Installing it on Raspian is as easy as issuing the usual command:

sudo apt-get install mosh

If you are just a desktop or server Linux user, you might never care about Mosh. But if you deploy systems to places where network connectivity is spotty or changes, it is worth a look. Even more importantly, if you have a need to keep an arbitrary object in sync between two different computers despite a poor network, you should really check out SSP to see if you can bend it to your will.

8 thoughts on “SSH Enters the Mosh Pit

  1. I’ve been using mosh for a few years now and generally find it wonderful. A few times while roaming around in cafés and such I’ve run into problems with it though. Many of those systems won’t allow the arbitrary UDP packets back and forth that mosh uses, so I end up falling back to regular SSH (which goes over TCP.)

    The other issue has been getting a decent iOS client with mosh support. Mosh uses an algorithm called OCB, which is patented, but the author has granted use to GPL licensed projects, which is great if you’re into GPL, but not so great for a lot of iOS app developers. Although in looking into this again, I just discovered, which has mosh support, so maybe I’ll give that a spin.

  2. Mosh is interesting, but I feel like the combination of available tools don’t fit well together. mosh does too little to be a complete solution (you still need screen or tmux), and has too much overlap with screen and tmux for them to layer neatly. I believe this may be more the fault of screen/tmux than mosh*, but mosh would probably see a lot more adoption if it either monolithically implemented the rest of screen’s functionality in whatever form, or at least had provision for a .moshrc on the server side, and came with example .moshrcs to automatically use screen and tmux.

    *Logically, it seems like screen’s fuctionality could be divided into:
    1. pty detach/reattach
    2. windowing system, which could be further subdivided:
    2.a. splitting the display into multiple virtual ttys
    2.b. swapping multiple windows in the same space
    3. i/o magic:
    3.a. copy/paste/scrollback
    3.b. character set translation
    3.c. arbitrary key bindings
    Replacing screen with a set of 3-5 separate, arbitrarily composable programs seems like the philosophically correct answer here, but it would require a lot of careful thought (on key-bindings, if nothing else), or you’d end up with a usability nightmare even worse than screen.

  3. I run irc in screen, and I found mosh and screen to be an unpleasant combination. I can ssh in and run screen -x from anywhere, but moshing in and running screen -x would always leave old mosh sessions around. I guess I rely on ssh timing out on the server side when a client is disconnected, and mosh will sit and patiently wait forever?

    1. Mosh waits for the same client to return, forever if necessary.
      If you kill your client, the related Mosh server will not get a notice about that.

      On the other hand you can put your system with the client to sleep and no matter whether minutes or months have passed or the cleient has a new address in a different network now, you can resume with that server.

      Newer Mosh versions add more options to terminate idle connections automagically… but I never tried them because I prefer to clean up myself.

    2. You probably know this, but screen -x connects to a running screen session without detaching it from any other ttys — it’s useful if you want to simultaneously access the same screen session from multiple locations.

      If you don’t want or need this function, use screen -Dr, -DR, or -DRR. Any of -r, -R, or -RR reconnect to a running session (the difference is how they handle the situation where this no/more than one matching session), but will not allow multiple connections. Without -d or -D, you’ll get an error if that session is attached elsewhere; with them, it will detach that existing connection. -d just disconnects, -D also sends a hang-up signal to the parent process. Thus if the remote connection was started from a shell with ssh or mosh, -d will drop back to a shell prompt, while -D will close the shell, thus closing the ssh/mosh connection.

  4. Back in 2007, I used to manage several servers , some of the clients were on Dial up connections. To manage all the slowness, I had kacked together my own kind of terminal management script, where I could switch between typing in real time directly, or type in a command line app in another terminal, which would send the keystrokes over to the original terminal, over KDE and Kterm’s dbus. The two terminals were tiled horizontally on the screen. the top terminal was a simple terminal. The terminal below would run a simple script to which the handle of the terminal above was provoded. The script would then continuously run, reading one line at a time, and sending the whole command to the active tab of the other terminal. All there was to do to get between continuous mode and line by line mode was an alt-tab switch.

    It was so nice to use, that I kept updating the script to do other magic. if I started the line with an escape sequence , and thn give it a single lettrr command, it would do a lot of things for me, like list out the clients I had, list out the servers the client had, their contact details, open a tab in the terminal, smartly even – it would switch to the tab if it was already open . Then came automating ticket generyion to our level 1 support team – if the issues were the standard ones , Like stuck mail queue all I had to do was
    #= tick “mail queue stuck”
    and it would create a ticket with the customer details, the server details, etc depending on the tab in the ither window.

    That was just a little hint .. giving the whole list of features here would be too offtopic.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s