bash

55 Articles

Linux Fu: Leaning Down With Exec

December 30, 2019 by 11 Comments

Shell scripting is handy and with a shell like bash it is very capable, too. However, shell scripting isn’t always very efficient. Think about it. If you run grep or tr or sort to do some operation in a shell script, you are spawning a whole new process. That takes time and resources. But there are some answers to reducing — but not eliminating — the problem.

Have you ever written a program like this (in any language, but I’ll use C):

int foo(void)
{
  ...
  bar();

}

You hope the compiler doesn’t write assembly code like this:

_foo: 
....

      call _bar
      ret

Most optimizers should pick up on the fact that you can convert a call like this to a jump and let the ret statement in _bar return to foo’s caller. However, shell scripts are not that smart. If you have a shell script called MungeData and it calls another program or shell script called PostProcess on its last line, then you will have at one time three processes in play: your original shell, the shell running MungeData, and either the PostProcess program or a shell running the script. Not to mention, the processes to do things inside post process. So what do you do?

Continue reading “Linux Fu: Leaning Down With Exec” →

Linux Fu: Debugging Bash Scripts

December 11, 2019 by 23 Comments

A recent post about debugging constructs surprised me. There were quite a few comments about how you didn’t need a debugger, as long as you had printf. For that matter, we’ve all debugged systems where you had nothing but an LED to flash or otherwise turn on to communicate with the user. However, it is hard to deny that a debugger can help with complex code.

To say you only need printf would be like saying you only need machine language. Technically accurate — you can do anything in machine language. But it sure makes things easier to have an assembler or some language to help you work out your problem. If you write a simple bash script, you can use the equivalent to printf — maybe that’s the echo command, although there is usually a printf command on a typical system, if you want to use it. However, there are other things you can do with bash including a pretty cool debugger if you know how to find it.

I assume you already know how to use echo and printf, but let’s dig into how to use trace execution line by line without the need for echo statements on every other line. Along the way, you’ll learn how to get started with the bash debugger.

Continue reading “Linux Fu: Debugging Bash Scripts” →

Linux Fu: Shell Scripts In C, C++, And Others

September 17, 2019 by 44 Comments

At first glance, it might not seem to make sense to write shell scripts in C/C++. After all, the whole point to a shell script is to knock out something quick and dirty. However, there are cases where you might want to write a quick C program to do something that would be hard to do in a traditional scripting language, perhaps you have a library that makes the job easier, or maybe you just know C and can knock it out faster.

While it is true that C generates executables, so there’s no need for a script, usually, the setup to build an executable is not what you want to spend your time on when you are just trying to get something done. In addition, scripts are largely portable. But sending an executable to someone else is fairly risky — but your in luck because C shell scripts can be shared as… well, as scripts. One option is to use a C interpreter like Cling. This is especially common when you are using something like Jupyter notebook. However, it is another piece of software you need on the user’s system. It would be nice to not depend on anything other than the system C compiler which is most likely gcc.

Luckily, there are a few ways to do this and none of them are especially hard. Even if you don’t want to actually script in C, understanding how to get there can be illustrative.

Continue reading “Linux Fu: Shell Scripts In C, C++, And Others” →

Secret Messages Could Be Hiding In Your Server Logs

August 28, 2019 by 13 Comments

[Ryan Flowers] writes in with a clever little hack that can allow you to hide data where nobody is going to go looking for it. By exploiting the fact that a web server will generally log all HTTP requests whether or not it’s valid, he shows how you can covertly send a message by asking the server for a carefully crafted fictitious URL.

We aren’t talking about requesting “yousuck.txt” from the server that hosts your least favorite website, either. As [Ryan] demonstrates, you can compress a text file, encode it with uuencode, and then send it line by line to the destination server with curl. He shows how the process, which he calls “CurlyTP” can be done manually on the command line, but it would be a simple matter of wrapping it up in a Bash script.

To get the message back, you just do the opposite. Use grep to find the lines in the log file that contain the encoded data, and then put them through uudecode to get the original text back. Finding the appropriate lines in the log file is made easier by prepending a prearranged keyword to the beginning of the URL requests. The keyword can be changed for each message to make things easier to keep track of.

If you’re still wondering why anyone would go through the trouble to do this, [Ryan] provides an excellent example: a covert “dead drop” where people could leave messages they’d rather not send through the usual channels. As long as the sender used a service to mask their true IP address, they could anonymously deliver messages onto the server without having to use any special software or protocol they might not have access to. Even the most restrictive firewalls and security measures aren’t likely to be scanning URLs for compressed text files.

We’ve seen web-based dead drops done with Python in the past, and even purpose built “PirateBoxes” that allow people to covertly exchange files, but we like how this method doesn’t require any special configuration on the server side. You should check your server logs, somebody might be trying to tell you something.

Windows 10 Goes To Shell

June 10, 2019 by 51 Comments

Windows 10 — the operating system people love to hate or hate to love. Even if you’re a Linux die-hard, it is a fair bet that your workplace uses it and that you have friends and family members that need help forcing you to use Windows at least some times. If you prefer a command line — or even just find a place where you have to use the command line, you might find the classic Windows shell a bit anemic. Some of that’s the shell’s fault, but some of it is the Windows console which is — sort of — the terminal program that runs various Windows text-based programs. If you have the creator update channel on Windows 10, though, there have been some recent improvements to the console and the Linux system that will eventually trickle down to the mainstream users.

What’s New?

So what’s new? According to Microsoft, they’ve improved the call interface to make the following things work correctly (along with “many others”):

  • Core tools: apt, sed, grep, awk, top, tmux, ssh, scp, etc.
  • Shells: Bash, zsh, fish, etc.
  • Dev tools: vim, emacs, nano, git, gdb, etc.
  • Languages & platforms: Node.js & npm, Ruby & Gems, Java & Maven, Python & Pip, C/C++, C# &
  • .NET Core & Nuget, Go, Rust, Haskell, Elixir/Erlang, etc.
  • Systems & Services: sshd, Apache, lighttpd, nginx, MySQL, PostgreSQL

The changes to the console are mostly surrounding escape sequences, colors, and mouse support. The API changes included things like allowing certain non-administrative users to create symlinks. We’ve made X Windows work with Windows (using a third-party X server) and Microsoft acknowledges that it has been done. However, they still don’t support it officially.

Continue reading “Windows 10 Goes To Shell” →

Easy Git Repository Summaries With Web-git-sum

March 25, 2019 by 16 Comments

For those hosting their own git repositories there are a number of solutions for creating convenient web-accessible front ends, but [mitxela] wasn’t quite satisfied with any of them. After trying a number of alternatives and reflecting on his requirements, he realized that all he really needed was a summary page listing the latest commits, and a file tree with a list of branches and tags. To accomplish this, he created web-git-sum. It’s a bash script that runs on git’s post-receive hook and generates only two files: a summary page and an index of the repository. You can see a demo of the output at git.mitxela.com.

[mitxela]’s writeup goes into some detail on how git repositories work, how those repositories are served over HTTP, and covers a few of the different options for providing convenient and accessible web front ends. Not all repositories are alike, and what works well for one may not work or scale well for another.

Intrigued by the idea of a private git server? We covered exactly how to set one up (spoiler: it’s really easy.)

Bash Pipeline Puzzle: Green, Blue, Or Blue Green?

March 10, 2019 by 16 Comments

[Marek Gibney] poses an interesting puzzle. What does the following bash command line print?

( echo red ; echo green 1>&2 ) | echo blue

You’d like to think it prints three lines: red, green, and blue. But would you be surprised to find out that it can sometimes output “blue green” and sometimes just output blue. The first surprise is that it isn’t deterministic. But the second thing that is surprising is the sometimes the entire left-hand part of the line doesn’t do anything. [Chris Siebenmann] did the analysis and explains what’s going on in a recent blog post.

Before you click the link or read further, you might want to see if you can deduce what’s going on. Give up? Here’s a hint: Part of the solution hinges on the fact that echo is built into the shell.

Continue reading “Bash Pipeline Puzzle: Green, Blue, Or Blue Green?” →