Linux Fu: Mapping Files

If you use C or C++, you have probably learned how to open a file and read data from it. Usually, we read a character or a line at a time. At least, it seems that way. The reality is there are usually quite a number of buffers between you and the hard drive, so your request for a character might trigger a read for 2,048 characters and then your subsequent calls return from the buffer. There may even be layers of buffers feeding buffers.

A modern computer can do so much better than reading using things using old calls like fgetc. Given that your program has a huge virtual address space and that your computer has a perfectly good memory management unit within it, you can ask the operating system to simply map the file into your memory space. Then you can treat it like any other array of characters and let the OS do the rest.

The operating system doesn’t necessarily read the entire file in at one time, it just reserves space for you. Any time you hit a page that isn’t in memory, the operating system grabs it for you invisibly. Pages that you don’t use very often may be discarded and reloaded later. Behind the scenes, the OS does a lot so you can work on very large files with no real effort. The call that does it all is mmap.

Continue reading “Linux Fu: Mapping Files”

Linux Fu: Leaning Down With Exec

Shell scripting is handy and with a shell like bash it is very capable, too. However, shell scripting isn’t always very efficient. Think about it. If you run grep or tr or sort to do some operation in a shell script, you are spawning a whole new process. That takes time and resources. But there are some answers to reducing — but not eliminating — the problem.

Have you ever written a program like this (in any language, but I’ll use C):

int foo(void)
{
  ...
  bar();

}

You hope the compiler doesn’t write assembly code like this:

_foo: 
....

      call _bar
      ret

Most optimizers should pick up on the fact that you can convert a call like this to a jump and let the ret statement in _bar return to foo’s caller. However, shell scripts are not that smart. If you have a shell script called MungeData and it calls another program or shell script called PostProcess on its last line, then you will have at one time three processes in play: your original shell, the shell running MungeData, and either the PostProcess program or a shell running the script. Not to mention, the processes to do things inside post process. So what do you do?

Continue reading “Linux Fu: Leaning Down With Exec”

Linux Fu: WSL Tricks Blur The Windows/Linux Line

We have to admit, we have an odd fascination with WSL — the Windows subsystem for Linux. On the one hand, it gives us more options on Windows 10 for running the software we love. On the other hand, we wonder why we aren’t just running Linux. Sometimes it is because our cool laptop doesn’t work well on Linux. Other times we are using someone else’s computer that we aren’t allowed to reload or dual boot. Still, as long as we have to use Windows, we are glad to have WSL. A recent blog post by [Hanselman] shows some very cool tricks for using WSL that make it even better.

Exploring WSL

Did you know you can use WSL to run Linux commands in a Windows command shell? For example, you have a long directory and you want to run grep:

dir c:\archive\* | wsl grep -i hackaday

Of course, from bash you could access the same directory:

ls /mnt/c/archive | grep -i hackaday

Continue reading “Linux Fu: WSL Tricks Blur The Windows/Linux Line”

Linux Fu: Debugging Bash Scripts

A recent post about debugging constructs surprised me. There were quite a few comments about how you didn’t need a debugger, as long as you had printf. For that matter, we’ve all debugged systems where you had nothing but an LED to flash or otherwise turn on to communicate with the user. However, it is hard to deny that a debugger can help with complex code.

To say you only need printf would be like saying you only need machine language. Technically accurate — you can do anything in machine language. But it sure makes things easier to have an assembler or some language to help you work out your problem. If you write a simple bash script, you can use the equivalent to printf — maybe that’s the echo command, although there is usually a printf command on a typical system, if you want to use it. However, there are other things you can do with bash including a pretty cool debugger if you know how to find it.

I assume you already know how to use echo and printf, but let’s dig into how to use trace execution line by line without the need for echo statements on every other line. Along the way, you’ll learn how to get started with the bash debugger.

Continue reading “Linux Fu: Debugging Bash Scripts”

Linux Fu: It’s A Trap!

It is easy to think that a Linux shell like Bash is just a way to enter commands at a terminal. But, in fact, it is also a powerful programming language as we’ve seen from projects ranging from web servers to simple utilities to make dangerous commands safer. Like most programming languages, though, there are multiple layers of complexity. You can spend a little time and get by or you can invest more time and learn about the language and, hopefully, write more robust programs.

Continue reading “Linux Fu: It’s A Trap!”

Linux Fu: Easier File Watching

In an earlier installment of Linux Fu, I mentioned how you can use inotifywait to efficiently watch for file system changes. The comments had a lot of alternative ways to do the same job, which is great. But there was one very easy-to-use tool that didn’t show up, so I wanted to talk about it. That tool is entr. It isn’t as versatile, but it is easy to use and covers a lot of common use cases where you want some action to occur when a file changes.

Continue reading “Linux Fu: Easier File Watching”

Linux Fu: The Kitchen Sync

One of the great things about Linux and similar operating systems is they are configurable. If you don’t like something, there’s a great chance you can change it easily with a few entries in a file somewhere. For example, take bash — a very popular shell by any measure. If you want a different style of command line editing, there’s an option. You want the tab key to match files regardless of case? Another option. Usually, these are set in one of your so-called profile files like .bashrc in your home directory.

As long as you are sitting in front of your single computer working, this is great. You customize your .bashrc and other files to your heart’s content and then you work in an environment that acts the way you want it to. The problem is when you have a lot of computers. Maybe you have a web server, a desktop, a firewall machine, and a few dozen Raspberry Pi computers. How do you keep all the configurations the same? Then once they are the same, how do you keep them up to date?

Continue reading “Linux Fu: The Kitchen Sync”