Tales from the Sysadmin

2 Articles

Troubleshooting A Symlink — A Whodunnit For The Git Record Books

February 6, 2020 by 14 Comments

While I normally sport the well-worn fedora of a hard-boiled sysadmin, Sunday mornings I swap that neo-noir accessory for the tech-noir: a pair of pro headphones. This is the tale of the collision of those two roles. An educational caper, dear reader. You see, my weekly gig is to run a Facebook Live Stream, and Facebook just recently began enforcing a new policy: all video streams are required to use encryption. We have Fedora installed on the media machine, and use Open Broadcaster Software (OBS) to stream. It should have been easy to update the stream settings. I made the necessary changes and tested it out — no luck. The error message was less than helpful: “Failed to connect to server”. With a sigh, I took off my headphones, put my sysadmin hat on, and walked out into the digital darkness. It was time to get back to work.

Continue reading “Troubleshooting A Symlink — A Whodunnit For The Git Record Books” →

Tales From The Sysadmin: Dumped Into The Grub Command Line

November 25, 2019 by 53 Comments

Today I have a tale of mystery, of horror, and of hope. The allure of a newer kernel and packages was too much to resist, so I found myself upgrading to Fedora 30. All the packages had downloaded, all that was left was to let DNF reboot the machine and install all the new packages. I started the process and meandered off to find a cup of coffee: black, and darker than the stain this line of work leaves on the soul. After enough time had elapsed, I returned, expecting the warming light of a newly upgraded desktop. Instead, all that greeted me was the harsh darkness of a grub command line. Something was amiss, and it was bad.

(An aside to the reader, I had this experience on two different machines, stemming from two different root problems. One was a wayward setting, and the other an unusual permissions problem.)

How does the fledgling Linux sysadmin recover from such a problem? The grub command line is an inscrutable mystery to the uninitiated, but once you understand the basics, it’s not terribly difficult to boot your system and try to restore the normal boot process. This depends on what has broken, of course. If the disk containing your root partition has crashed, then sorry, this article won’t help.

In order to get a system booting, what exactly needs to happen? How does booting Linux work, even? Two components need to be loaded into memory: the kernel, and the initramfs. Once these two elements are loaded into memory, grub performs a jump into the kernel code, which takes over and finishes the machine’s boot. There is one more important detail that we care about — the kernel needs to know where to find the root partition. This is typically part of the kernel parameters, specified on the kernel boot line.

When working with an unfamiliar shell, the help command is a good starting point. grub runs in a very limited environment, and running the help command scrolls most of the text off the screen. There is an environment variable that helps out here, enabling output paging:set pager=1.
Continue reading “Tales From The Sysadmin: Dumped Into The Grub Command Line” →