A lot of embedded systems are running Linux on platforms like Raspberry Pi. Since Linux is fully functional from a command line and fully network-capable, it is possible to run servers that you’ve never had physical access to.
There are a few problems, though. Sometimes you really need to reboot the box physically. You also need to be at the console to do things like totally install a new operating system. Or do you? Over on GitHub, user [marcan] has a C program and a shell script that allows you to take over a running system without using any software on the root filesystem. It starts an ssh server and you can remotely unmount the main drive, do any maintenance you want and –presumably–reboot into a new operating system.
The key is to create a temporary file system (which lives in RAM0 and copy a system rescue CD to it. The system also has to use systemv-style init so that the command “telinit u ” will cause init to rerun itself. The init process is the program the bootloader really executes and always has PID 1.
However, with [marcan’s] script, the file system is scrambled around and a script built on the fly by the takeover script replaces init. So when init reruns itself, it really executes the script which eventually runs in the included fakeinit which just sits around and waits.
Usually, the Linux boot process just works and you might not be aware of all that goes on with it. But if you do know, you can pull tricks like this. Raspbian, for example, derives from Debian, so you can learn more about the boot process by looking at the Debian documentation.
Of course, if your distribution has switched to systemd, you’ll have to try a different tack (we did a post about the Linux kernel awhile back that led to a lot of comments about the systemd war). Understanding the Linux boot system is one pillar to Linux wizardry. Learning more about the kernel and system calls are two more.
But I thought that Debian Stable(Jessie) and current Raspbian (based on Debian Jessie) both have been running systemd now for sometime.
This is a bad idea in any exposed production machine, and at first glance I think the Author of this project understands this (hence [marcan’s] “WARNING WARNING WARNING WARNING” header in his GitHub README). So IMO there’s nothing earth-shaking here, just another chip-away at the (arguably) unnecessary obfuscation introduced by Systemd on machines with limited need to manage drivers etc. for non-persistent peripherals (like a laptop or hand-phone – but unlike a server). The hard Black/Red Security Paradigm mandates that a host must be self-healing without any external intervention. Obviously this becomes a topological problem in machines connected to more than one other machine, especially in a routable mesh network. So the likes of [marcan’s] script is mildly interesting from a development stand-point, but there are myriad other solutions to circumvent Systemd’s added complexity when not needed, Finally, I think the HaD post’s title should have reflected this fact. But I’m not ready yet to raise the “Click-Bait” flag.
Use a cheap (DDWRT-able) router that has a usb to attach an FT232 or something for serial.
Use that as some kinda almost open source management engine (make sure to have encrypted-rootfs and a STRONG password).
Shell into your “unreachable” server via this serial connection.
Booting from an already loaded OS is as simple as killing any potentially interfering applications (or programs, if you’d prefer).
KEXEC into the replacement OS with boot args telling the KEXEC’d kernel where to find it’s root-fs (also preferably encrypted if need be)
Oh, forgot:
Use read-only filesystems for the servers and only have what you need writable. Oh and do what it takes to keep the RO-FS in read-only (Even if by hardware switch and/or pull WR_EN to the disable voltage).
If need be, make any images that stay RO in ROM and loaded during boot into a RW initfs/ramdisk with configuration loaded from other writable locations.
Make the first ROM be a recovery/bootshell with some delay.
Ok you probably can’t do these things with the PI.
But these are just things I’ve seen in DAP/MP3/Phone/assorted-embedded industries through my life to give me ideas.
Does kexec work on anything beyond x86? I thought it had real/virtual mode requirements to work.
https://github.com/mjg59/kexec-tools/blob/master/kexec/arch/arm/kexec-zImage-arm.c
Looks like it works! Neat!
I used to use kexec to get Cyanogenmod going on a Droid 4.
Worked fine.
Or… attach to network connected kvm and have complete control of the machine even at bios config level.
Those are very difficult to control programmatically because of their async nature. You issue commands and hope for the best and try again when the device doesn’t appear on the network. I’ve written one of these systems and the solution to most issues was adding sleeps in various stages. It’s all kinds of wrong. Best case if it’s not just kvm but serial console. SUN had this right with LOM and intel screwed the pooch with their ipmi implementation. In any case a console lets you grab the output and parse it for success at least. It still has a sync problem.
Couldn’t read the article… cus cus… emacs in the image.. aaahhh
Wan kommt der Bus?