If you aren’t a Linux user and you watch someone who knows what they are doing use Bash — the popular command line interpreter — you might get the impression they type much faster than they actually do. That’s because experienced Linux users know that pressing the tab key will tend to complete what they are typing, so you can type just a few characters and get a much longer line of text. The feature is very smart so you may not have realized it, but it knows a good bit about what you could type. For example, if you try to unzip a file, it knows the expected file name probably has a .zip extension.
How does that happen? At first, you might think, “who cares how it happens?” The problem is when you write a shell script or a program that runs on Linux, the completion gets dumb. Someone has to make Bash smart about each command line program and if you are the author then that someone is you.
Anatomy of Command Completion
Turns out completion depends on a particular GNU library known as readline. It reads text for lots of different programs, including Bash and you can configure it using the .inputrc file in your home directory. For example, here’s my .inputrc:
"\e[A": history-search-backward "\e[B": history-search-forward $if Bash Space: magic-space $endif set match-hidden-files off set completion-ignore-case on set visible-stats on set show-all-if-ambiguous on
That doesn’t look like much, but there is a system-wide configuration at /etc/inputrc which is much more substantial. You can also issue certain commands that will modify the configuration on the fly. For example, the Bash built-in “bind” can set things up and also show you a long list of what’s already set up. If you do this command:
bind -p
You’ll see a long list of output. Of interest here are the following few lines:
"\C-i": complete
"\e\e": complete
"\e!": complete-command
"\e/": complete-filename
"\e@": complete-hostname
"\e{": complete-into-braces
"\e~": complete-username
"\e$": complete-variable
Yours may vary, but essentially this says that when the tab key or a double escape shows up, to do a completion operation. There are also a few specialized completions that start with escape.
But How Does it Work?
There are three built-in shell commands that cooperate to manage completion. They are:
- complete – Configure completion
- compgen – Generate possible completions
- compopt – Modify completion options
I won’t get into all the possible options, although you can read the Bash manual if you are curious. For our purposes, you’ll only need a few, and all three commands mostly take the same arguments anyway.
Let’s try a few things with compgen, since that’s what bash eventually calls to get completions. Suppose you are typing ls at the command line and you press tab. Bash will make the following call:
compgen -c ls
You’ll see a bunch of output (depending on what you have installed) of all the commands that start with ls, such as ls itself, lsusb, lspci, etc. You can also use the more modern “-A command” syntax to get the same result.
Try this one:
compgen -d /e
Again, you could use -A directory if you prefer. Clearly, this outputs directories. This is fine, but how does Bash know what options to pass to compgen? That’s where the complete command comes in. The complete command can tell Bash to call a shell function or a command in response to a completion for a certain command. It can also set one for an empty command or a command that otherwise has no matching rule. There are also nice shortcuts for common cases.
Consider a script called burnhex. If you type burnhex, a space, and a tab, Bash (via readline) will offer you all the files in the current directory for completion. However, you can guess you only want files that end in .hex. You can easily do that with complete:
complete -G "*.hex" burnhex
Now only files that end in .hex will show up.
To look at even more complex completions, let’s start with a simple shell function. By convention, these start with underscores, although that isn’t a technical requirement:
function _hackaday {
COMPREPLY=( "hackaday.com" "hackaday.io" )
return 0
}
This tells the system we have two completions. Let’s pretend we have a script or program called “gohackaday” and we want this to be the completion for it (the command doesn’t have to exist for this to work):
complete -F _hackaday gohackaday
Now if you type:
gohackaday<SPACE><TAB>
You’ll see your completion in action. The system is smart enough to know that it can fill in the “hackaday.” part, since both choices start with that. If you type a “c” or an “i” and hit tab again, it will go ahead and pick the right one.
By the way, the function gets three arguments that we aren’t using. The first argument ($1) is the name of the command that’s active, the second is the word being completed, and the third is the word before the current word. For example, consider the line below and the three arguments following:
ls -l /foo $1=ls $2=/foo $3=-l
There are also several environment variables set if you want to extract the entire line or determine how the user called up the completion. For simple cases, you probably don’t need these.
Of course, this is an easy example. For things that are more than you can process in a function, you can also run a full-blown command by using -C instead of -F. Commands get the same arguments and most of the same environment variables.
Besides that, remember the -c and -d options to compgen? They work here, too. So if you had a command called foobar that needs a directory name as an argument you could say:
complete -d foobar
You can even call compgen in your function or command to generate data directly or for further filtering and augmentation in your code. This can get quite complex and if you look at some of the built-in ones, you’ll see what I mean.
In general, a lot of completion scripts live in either /etc/bash_completion.d or /usr/share/bash-completions. Yes, one of those is an underscore and one is a dash. One of the joys of Linux is how every system is a little bit different.
A More Typical Example
Consider the ftp command. You can find out what the completion is for that by typing:
complete -p | grep ftp
The result will look like:
complete -F _known_hosts ftp
That means there is a shell function called _known_hosts. If you want to see what it looks like, try:
declare -f _known_hosts
You should see something like this:
_known_hosts ()
{
local cur prev words cword;
_init_completion -n : || return;
local options;
[[ "$1" == -a || "$2" == -a ]] && options=-a;
[[ "$1" == -c || "$2" == -c ]] && options+=" -c";
_known_hosts_real $options -- "$cur"
}
Obviously, the real work is being done in _known_hosts_real and you can dump it the same way. It is complex, but you can guess it is going to dump host names the computer knows about. There are also other helper functions like _init_completion. The point isn’t how these work, but that they exist, and you can use them just like the ftp completion setup does.
This has just been a quick overview. If you want all the gory details, the Bash manual is helpful. You can also find a lot of useful documents at the Bash Completion project GitHub page. Bash isn’t the only game in town, by the way. Zsh has a very powerful (and different) completion mechanism if this one isn’t enough for you.
Once you understand how Bash reads command lines, there are a lot of interesting tricks you can play. I’ll show you my favorite next time. Until then, I hope this helps you type less and do more.
I actually turn off a lot of the command-specific bash completion because it gets in the way more than it helps (the bash-completion package in Fedora).
For example, today I was on a newly installed machine trying to extract a tarball that had a non-standard extension. The tab would not complete that filename because it didn’t end with .tar. After removing the package, it worked just fine.
Yes!
You can use Alt-/ to complete files that don’t match.
\tar x tarbalnamestart
…eeeh xf… but hit shappens.
https://xkcd.com/1168/
lol, hadn’t seen that one, so true. Randall for president!
The bomb works by unpacking a massive blob of files and directories with assorted permissions and strange filenames into your current working directory.
You do know you can force it to complete using the right keys? / will always complete a file name, for example. A lot of people like completion so much they switch to zsh which has even more completion. Besides… worst case you have to type the file yourself which is no different than turning it off, so I don’t get why that matters? If you turn off completion you lose the ability to complete things like options or specific files so you have to mentally filter through more files in many cases. For example, in the tar case if I type a it is going to show me al.jpg and aragon.txt along with abackup.tar.
Personal tradeoff, I suppose, but most of us just suffer the occasional manual typing or learn the specific keycodes.
Bash completion is a particularly annoying beast in Fedora.
I’ve recently upgraded this machine to F26 and discovered along the way that it won’t complete a filename when re-creating an existing tar file. Something it was perfectly happy to do before.
I have to wonder if Poettering and/or Sievers had something to do with this.
Well, the other thing is that like most things in Linux you can a) turn it off, or b) open up the source and modify it to suit you.
Me too. The built-in completions are more than enough. And on (really) slow machines it takes considerable time to load the completion scripts shipped with modern distributions.
Long ago I learnt to write a different command. Or change it if already typed most of it. Sometimes I remember the other completion methods, as mentioned (/ for files, @ for hosts, and so on). But I always remember this ever since:
C-a to go to start of line.
Type “a” (or whatever), so it reads “atar”.
C-e to go to end of line.
Tab for completion.
C-a.
C-d to delete the “a” and it says “tar” again.
Enter, or C-e and keep on typing.
Sounds like a lot, but can be typed fast. Someone already suggested using \t to hide from the completion, and that one requires no delete step (OTOH, \ is a complex combo in some keyboard layouts).
Removing tab completion tools are not good idea when they save so much time and memory (commands with lots and lots of parameters). Yep, you can also use Home, End and such keys. I prefer to invest the memory in C-, M-, etc combos (M-b, M-u, etc). They tend to work with old devices better than the extra keys.
Using command completion in a bash script is a ridiculous security vulnerability for obvious reasons. I cannot imagine ever exposing a system I’m managing to the nearly infinite state space that results from allowing the code to modify itself according to the file-system contents. This seems like an opportunity for polymorphism trolling, but otherwise appears like an exercise in chainsaw juggling.
I think you may have misread. The article is explaining how tab-completion works under the hood, and how to expand it to be smart about completing arguments for your own scripts/programs. It is *not* advocating for somehow using tab-completion output to modify your own script’s code.
Tab completion polymorphism to escalate permissions would be a great level in a wargame.
bash is a fine interactive shell, but for goodness sake, don’t use it to write scripts. I use ruby. I also recommend python. Even perl is a great choice. Long ago I decided to forgo learning bash, awk, and sed and just learn perl and it was one of the best things I ever did — but I have moved on to using ruby (perl done right).
Using completion in a script? Well you shouldn’t be writing bash scripts in the first place, but certainly don’t use completion in a script — what are you thinking??
All I have ever need to know about bash filename completion is to put “set -o vi” in my .bashrc and I can use vim keystrokes. Of course I am a vim user — other such modes are probably available for those with other habits.
While I agree with the gist of the message, don’t fully dismiss bash or shell. It’s useful for gluing the other scripting languages and native commands together. Just once you start to do anything complex or lengthy looping it’s not the appropriate tool. I find that to be often but I still find bash useful for quick piping and cutting (along with sed, awk, cut, uniq and sort).
As far as completion in scripts, I don’t think that’s what the above was about (I hope not). That seems suicidal.
No, although I used a non-existant script as an example if you read this is about enhancing the existing completion method to know about your programs no matter what they are written in. I’m not sure where anyone has the idea that this is completion inside a script. Not even sure what that would mean other than using globbing in a script.
It amazes me how something like this can turn into a virtue of my favorite language/evil of your favorite language discussion. I fully expected a raft of comments that if you want completion you should use zsh.
As for the merits of bash, it is actually quite useful with a few caveats. But that isn’t what this post was about at all so I’m going to stop with just that comment. Then again, a lot of things are quite useful, too. I don’t know that ruby would actually be my first choice for scripting, but if it works for you, awesome. I do a lot of awk, which isn’t a lot of people’s first choice either (for good reasons). As for programming robustness and security, I find that is almost always more a function of the programmer than the language.
While I agree that you should never use completion in a script, I am not sure what is wrong with writing scripts in bash. I have written many scripts over the years and I have had no more and perhaps even less issues with things written in bash than in something like perl. I am glad you enjoy ruby, but good luck with that in say wifi router or any small embedded system.
Indeed, tiny embedded systems that use linux may not have the full complement of tools like ruby or python. On a really small linux based embedded system I might be surprised to even have bash and would have to deal with some truly primitive shell. You do what you have to do in such cases. But you may be able to use a package manager and add some things of your choice .. maybe.
It is easy to take completion for granted. In the early beaglebone black days (circa Angstrom linux), they shipped bash without any kind of completion configured. What misery! You don’t realize how often you use it until then. You don’t know what you got til it’s gone!! Debian on the beaglebone black fixed that and a thousand other things. But that was the first time I realized that bash completion had configuration files that needed to be set up. And as Al is telling us, you can make a new hobby out of customizing those files beyond how they come shipped to you.
is worth noting oh-my-zsh a very useful set of script for zsh.
I’ve been meaning to get in to this for years but was dreading the details. An executive summary is just the ticket. Thank-you!
Thanks for this overview!
Quoting: “You’ll see your completion in action. The system is smart enough to know that it can fill in the “hackaday.” part, since both choices start with that. If you type a “c” or an “i” and hit tab again, it will go ahead and pick the right one.”
This isn’t working for me: when I type the “c”, it jumps back to the “hackaday.”, giving me the error beep. Is there something different enough on macOS that this doesn’t work the same way?