Linux Fu: Leaning Down With Exec

December 30, 2019 by Al Williams 11 Comments

Shell scripting is handy and with a shell like bash it is very capable, too. However, shell scripting isn’t always very efficient. Think about it. If you run grep or tr or sort to do some operation in a shell script, you are spawning a whole new process. That takes time and resources. But there are some answers to reducing — but not eliminating — the problem.

Have you ever written a program like this (in any language, but I’ll use C):

int foo(void)
{
  ...
  bar();

}

You hope the compiler doesn’t write assembly code like this:

_foo: 
....

      call _bar
      ret

Most optimizers should pick up on the fact that you can convert a call like this to a jump and let the ret statement in _bar return to foo’s caller. However, shell scripts are not that smart. If you have a shell script called MungeData and it calls another program or shell script called PostProcess on its last line, then you will have at one time three processes in play: your original shell, the shell running MungeData, and either the PostProcess program or a shell running the script. Not to mention, the processes to do things inside post process. So what do you do?

Continue reading “Linux Fu: Leaning Down With Exec” →

Linux Fu: Stupid SSH Tricks

December 17, 2019 by Al Williams 52 Comments

If you connect to remote computers over the Internet, it is a pretty good chance you use some form of SSH or secure shell. On Linux or Unix you’ll use the ssh command. Same goes for Linux-like environments on Windows like Cygwin or WSL. For native Windows, you might be using Putty. In its simplest form, ssh is just a terminal program that talks to a server using an encrypted connection. We think it is very hard to eavesdrop on anyone communicating with a remote computer via ssh.

There are several tricks for using ssh — some are pretty straightforward and some are things you might not think of as being in the domain of a terminal program. You probably know that ssh can copy files securely, and there are easy and hard ways to set up logging in with no password.

However, you can also mount a remote filesystem via ssh (actually, there are several ways to do that). You can use ssh to securely browse the web in your favorite browser, or even use it to tunnel specific traffic by port or even use it as a makeshift VPN. In fact, there’s so much ground to cover that this won’t be the last Linux Fu to talk about ssh. But enough setup, let’s get to the tricks.

Continue reading “Linux Fu: Stupid SSH Tricks” →

Linux Fu: Debugging Bash Scripts

December 11, 2019 by Al Williams 23 Comments

A recent post about debugging constructs surprised me. There were quite a few comments about how you didn’t need a debugger, as long as you had printf. For that matter, we’ve all debugged systems where you had nothing but an LED to flash or otherwise turn on to communicate with the user. However, it is hard to deny that a debugger can help with complex code.

To say you only need printf would be like saying you only need machine language. Technically accurate — you can do anything in machine language. But it sure makes things easier to have an assembler or some language to help you work out your problem. If you write a simple bash script, you can use the equivalent to printf — maybe that’s the echo command, although there is usually a printf command on a typical system, if you want to use it. However, there are other things you can do with bash including a pretty cool debugger if you know how to find it.

I assume you already know how to use echo and printf, but let’s dig into how to use trace execution line by line without the need for echo statements on every other line. Along the way, you’ll learn how to get started with the bash debugger.

Continue reading “Linux Fu: Debugging Bash Scripts” →

Tales From The Sysadmin: Dumped Into The Grub Command Line

November 25, 2019 by Jonathan Bennett 53 Comments

Today I have a tale of mystery, of horror, and of hope. The allure of a newer kernel and packages was too much to resist, so I found myself upgrading to Fedora 30. All the packages had downloaded, all that was left was to let DNF reboot the machine and install all the new packages. I started the process and meandered off to find a cup of coffee: black, and darker than the stain this line of work leaves on the soul. After enough time had elapsed, I returned, expecting the warming light of a newly upgraded desktop. Instead, all that greeted me was the harsh darkness of a grub command line. Something was amiss, and it was bad.

(An aside to the reader, I had this experience on two different machines, stemming from two different root problems. One was a wayward setting, and the other an unusual permissions problem.)

How does the fledgling Linux sysadmin recover from such a problem? The grub command line is an inscrutable mystery to the uninitiated, but once you understand the basics, it’s not terribly difficult to boot your system and try to restore the normal boot process. This depends on what has broken, of course. If the disk containing your root partition has crashed, then sorry, this article won’t help.

In order to get a system booting, what exactly needs to happen? How does booting Linux work, even? Two components need to be loaded into memory: the kernel, and the initramfs. Once these two elements are loaded into memory, grub performs a jump into the kernel code, which takes over and finishes the machine’s boot. There is one more important detail that we care about — the kernel needs to know where to find the root partition. This is typically part of the kernel parameters, specified on the kernel boot line.

When working with an unfamiliar shell, the help command is a good starting point. grub runs in a very limited environment, and running the help command scrolls most of the text off the screen. There is an environment variable that helps out here, enabling output paging:set pager=1.
Continue reading “Tales From The Sysadmin: Dumped Into The Grub Command Line” →

Training Bats In The Random Forest With The Confusion Matrix

November 21, 2019 by Pat Whetman 4 Comments

When exploring the realm of Machine Learning, it’s always nice to have some real and interesting data to work with. That’s where the bats come in – they’re fascinating animals that emit very particular ultrasonic calls that can be recorded and analysed with computer software to get a fairly good idea of what species they are. When analysed with an FFT spectogram, we can see the individual call shapes very clearly.

Nyctalus noctula (noctule bat)

Myotis nattereri (natterera’s bat)

Plecotus auritus (brown long eared)

Pipistrellus pipistrellus (common pipistrelle)

Creating an open source classifier for bats is also potentially useful for the world outside of Machine Learning as it could not only enable us to more easily monitor bats themselves, but also the knock on effects of modern farming methods on the natural environment. Bats feed on moths and other night flying insects which themselves have been decimated in numbers. Even in the depths of the countryside here in the UK these insects are a fraction of the population that they used to be 30 years ago, but nobody seems to have monitored this decline.

So getting back to our spectograms, it would be perfectly reasonable to throw these images at a convolutional neural network (CNN) and use an image feature-recognition strategy. But I wanted to explore the depths of the mysterious Random Forest. Continue reading “Training Bats In The Random Forest With The Confusion Matrix” →

DSP Spreadsheet: IQ Diagrams

November 15, 2019 by Al Williams 5 Comments

In previous installments of DSP Spreadsheet, we’ve looked at generating signals, mixing them, and filtering them. If you start trying to work with DSP, though, you’ll find a topic that always rears its head: IQ signals. It turns out, these aren’t as hard as they appear at first and, as usual, we’ll tackle them in a spreadsheet.

What does IQ stand for? The I stands for “in phase” and the Q stands for quadrature. By convention, the I signal is a cosine wave and the Q signal is a sine wave. Another way to say that is that the I and Q signals are 90 degrees out of phase. By manipulating the amplitude of I and Q, you can create complex modulation or, conversely, demodulate signals. We’ll see a spreadsheet that shows that completely next time.

Continue reading “DSP Spreadsheet: IQ Diagrams” →

DSP Spreadsheet: Frequency Mixing

November 1, 2019 by Al Williams 18 Comments

Circuit simulation and software workbooks like Matlab and Jupyter are great for being able to build things without a lot of overhead. But these all have some learning curve and often use clever tricks, abstractions, or library calls to obscure what’s really happening. Sometimes it is clearer to build math models in a spreadsheet.

You might think that spreadsheets aren’t built for doing frequency calculation and visualization but you’re wrong. That’s exactly what they’re made for — performing simple but repetative math and helping make sense of the results.

In this installment of the DSP Spreadsheet series, I’m going to talk about two simple yet fundamental things you’ll need to create mathematical models of signals: generating signals and mixing them. Since it is ubiquitous, I’ll use Google Sheets. Most of these examples will work on any spreadsheet, but at least everyone can share a Google Sheets document. Along the way, we’ll see a neat spreadsheet trick I should probably use more often.

Continue reading “DSP Spreadsheet: Frequency Mixing” →