We’ve often heard that you don’t need to know how an engine works to drive a car, but you can bet that professional race car drivers know. By analogy, you can build lots of systems with off-the-shelf boards like Raspberry Pis and program that using Python or some other high-level abstraction. The most competent hackers, though, know what’s going on inside that Pi and what Python is doing under the hood down to some low level.
If you’ve been using Linux “under the hood” often means understanding what happens inside the kernel–the heart of the Linux OS that manages and controls everything. It can be a bit daunting; the kernel is simple in concept, but has grown over the years and is now a big chunk of software to approach.
Your first embedded system project probably shouldn’t be a real time 3D gamma ray scanner. A blinking LED is a better start. If you are approaching the kernel, you need a similar entry level project. [Stephen Brennan] has just the project for you: add your own system call to a custom Linux kernel.
Like a blinking LED, the system call isn’t all that exciting in itself. But learning how to set up the kernel source tree, configure it, and build it can be educational. Of course, you might already know how to do the steps to configure and build since it isn’t uncommon to build custom kernels. However, adding your own code and making it work is educational.
There was a time when experimenting with a kernel almost required you to have a spare machine or a strong constitution. These days, you just do your build in a virtual machine and you can mess up all you like.
You might not get excited over a little thing like a custom system call, but it is a gateway to bigger things. Using GPU acceleration for a screaming fast kernel, for example. If you don’t want to bother with a virtual machine, maybe you can work on an 8-bit AVR instead.
Photo credit: [Shmuel Csaba Otto Traian] CC BY-SA 4.0
Not much point trying to do any hacking in modern Linux distro because its probably been consumed by systemd
Please, not again.
Some like systemd, some don’t (I don’t). But the way this discussion went (from both sides) isn’t constructive. There are enough projects out there to have Linux without systemd.
Help some of those if you care. Accept that others swing differently.
Given the amount of jury-rigging I have to do with systemd compared to sysvinit or openrc to do something as menial as shutdown (example) on a generic x86 PC with a high-profile distro doesn’t help systemd’s case at all.
“keep it simple, stupid” is a concept utterly lost on the developers of that, instead I’m getting the vibe they’ll even replace or “absorb” the kernel completely if they could.
poweroff? Worked, works and will work.
I usually like to preach tolerance, but wow systemd is really unfortunate. That and Baloo are my two least favorite things forced on me lately. I’m an old Unix guy and Unix along with most Linux is all about transparency. But there is a recent trend where it is starting to look more like other OSs.
I fully get that we could use a replacement for init.d and sure, full disk indexing is great if you can make it work without killing everybody. But the distros really need to get more careful about what they force on all their users. If you force it, it better be unobtrusive, it better work just about the time, and–rarely–it should be something everyone REALLY wants. For example, even if you like, say, zsh, you probably want sh and maybe even bash installed to run stuff from other people. Same for, say, Perl and awk. But not everyone needs–say–word processing, semantic file systems, etc. etc.
Time to switch to a BSD : ) FreeBSD is the jam. And it runs on Raspberry Pi now.
FreeBSD is bloaty kenrel wise as I understand it though so.. meh.
OpenBSD is pretty ok… clean kernel supports recentish intel laptops well enough. Will still run on very low end machines.. but is probably the slowest Unix out there.
NetBSD is sort of a mix of the two I guess… although it has the oldest heritage.
Minix3 is a BSD compatible now… thats interesting at least.
LiteBSD and RetroBSD on PIC32 etc… which is very neat but not entirely useful.
In short there isn’t really a nice/fast/bloat-free kernel out there at the moment… and thats without even considering driver availability.
I dunno… Give FreeBSD a chance, you may like it. I certainly do.
ZFS is a killer feature by the way. Filesystem of the future.
I’ve recently started using OpenBSD on my Yeeloong, and the only thing I notice is particularly slow is the video, which is un-accelerated.
Main reason I’m running OpenBSD is because Debian have turned elitists and no longer support MIPS below MIPS32 or MIPS64, Gentoo runs but building packages takes a long time and some software (notably Firefox) dislikes building on n32 (and on AMD64 x32). Linux from Scratch is an awful lot of work to maintain, and this machine is no speed demon!
OpenBSD, with pkgsrc, at least has some binary packages, so I’m not building it all myself, and the stuff that isn’t pre-built, much if it is available via ports.
You should give Void Linux a shot. I’ve happily installed it on various computers the last month, and it works like a charm. No pulseaudio or systemd. And these are not just servers. They’re desktops, with video/audio, and one running Cuda stuff.
All of it has been pretty painless, and man Void is fast!
http://media.giphy.com/media/5xtDarAgrjoOrBxSVYk/giphy.gif
BTW apparently they need to add “sudo” and “dhclient” to this list (?!)
Oh hey are we commenting systemd gifs now?
https://media.giphy.com/media/xTiTnp0prNgWpSGtFK/giphy.gif
Yeah, it’s horrible trying to hack on Linux when something changed the ways things used to be. Me, I still hold on to my 2.4 kernel on my Pentium-driven machine, and the 2012-dated Raspberry Pi SD card images. It’s harder and harder each year, but it’s so comfortable never having to learn anything new and criticising decisions other people made, especially when bothering to learn the reasons is just too much work. start-stop-daemon for the win!
It’s sad that a Pentium doesn’t even really cut it to run a recent kernel and actually do much of anything with it… the bloat is strong. I know it can run a recent kernel… but its less than optimal.
Fact is the current linux kernel is almost too bloaty to build a basic kernel that will even boot on a real sparc32 machine. And by boot I mean serial console + disk driver + fs driver and thats it… not even ipv6 or anthing else. It’s literally too big… I think the limit is 3Mb or so uncompressed or about 2.6Mb compressed. I have an SS20 that I can load Slowlaris up on and Opera 8.5 or so.. and it will even browse the modern web slowly but decently enough for a machine of its vintage. I mean it’s saying something when Linux has gotten more bloated than Slowlaris… X.x I’m still hoping they’ll have a swath of bloat cutting someday… in addition to making the kernel relocatable (that way kernels larger than 3Mb or so could be loaded on sparc32)
Even Celerons are suitable for latest kernels, that’s for sure. In fact, I’ve recently set up a PC that’s acting like an IRC client, always on and providing entertainment to hackerspace members. It’s got the latest kernel, and systemd with that. Specs: 700MHz P3, 256MB RAM (an old sturdy Compaq).
I’ve also got an EEE PC 900 (701 with 9″ screen, no other changes) – 700MHz Celeron with 1GB RAM, and it serves great as a machine for occasional Linux x86 work development, pentesting shenaningans and listening to music – I now use CLI exclusively, but I used GUIs a lot and the slowness wasn’t usually dependent on CPU – storage, now that was a limiting thing until I bought a more recent flash drive (oh, and it all ran from USB flash). Again, recent kernel (Debian stock kernel) and systemd – it’s really fast, and the boot times are so much more quick with no-bullshit parallel loading.
I don’t know about sparc32 that much, but I’ve seen and used plenty of 4MB OpenWRT images. Serial’s there, remove networking drivers and utilities, add FS&disk drivers and some basic utils for your FS – where’s the space problem? Again, there might be sparc32-specific things, but I’d be very interested to hear about those.
I love the above specs… and did I read that correctly, a 700MHz P3 is CLI-worthy?
My first linux computer was a 486SX25 I dug out of a dumpster with a whopping 8MB RAM and 120MB hard disk. X-windows ran great! Full-on boot-times were comparable with a high-end machine today, if not speedy… Now I’ve a P150 laptop with 48MB RAM I was thinking about getting running as a VNC client and can’t even figure out how to find a regular-ol’ (even old) distro that’ll even boot on it, let-alone which might be the latest *usable* version (yahknow, not *slow as heck*, still has driver-support, etc.). Might just be grateful I saved those old Slackware CDs.
http://aprs.fi/info/a/VK4MSL-1
Toshiba Portégé 7010CT
Pentium II 300MHz
160MB RAM, 160GB HDD
Gentoo Linux i386 with kernel 3.11
It sits there running xastir all day long, listening to 145.175MHz via the kernel AX.25 stack and kissattach via PL2303 USB serial to a Kantronics KPC-3 TNC.
When you said Pentium I assumed a original Pentium… The slowest thing I’ve used recently is my P1120 Fujtisu Lifebook which is has a Crusoe so similar to a 3-400Mhz Pentium. Unfortunately It got damaged during some repairs to its touchscreen I was attempting and I haven’t used it since (it still runs just headless)
RE Sparc32: its not a space problem, I have a 147Gb Scsi drive attached to it. It is that the kernel has literally gotten so bloated that it wont load… sparc32 has a size limit on the maximum kernel size that can be loaded. It can be worked around… but it was never an issue for Solaris which fully supported these machines and was considered very bloaty in it’s own right by the Linux developers at the time.
You may want to patch against the DirtyCOW vulnerability: https://dirtycow.ninja/
Not pixies? …. I have been misinformed.
I thought it was like a box of worms:
A very big box of worms like out of a nightmare, all inter-tangling and sliding in asynchronous synchronization with a feeling it wants to tangle us into it’s code.
I always thought the kernel’s developers were its slaves caught in the tangle of code trying to develop a way out.
Well this article clears things up.
“I always thought the kernel’s developers were its slaves caught in the tangle of code trying to develop a way out”
That’s actually a pretty good analogy.
An excellent example of this is printk(). The design is ancient and it’s become an unholy mess over the years which is almost impossible to maintain. Everyone agrees it needs to be redesigned. The problem is doing that without breaking everything which makes use of it and everything else which relies on the things which make use of it.
Even worse, the situation isn’t helped by the likes of Kay Sievers and Lennart Poettering who steadfastly refuse to fix problems in udev, pulse audio and – you’ve guessed it – systemd, forcing the kernel devs to work round them.
This problem isn’t limited to the Linux kernel. Even Microsoft had a similar problem with Windows 2000 and Office. The Office dev team refused to fix a fairly fundamental flaw in their code as this would stop it working on older Windows variants, forcing W2k devs to work round it.
Was joking, though based on some truth.
However don’t get me started on Fulse Audio and Grr-streamer.
Always had problems with those, thus I bypassed all of that and hacked up boot-scripts for my multimedia laptop
I use the same SSD multiple PCs and two laptops.
That is on an old Debian Wheezy install with swiss cheese security (by todays standards) and non critical (non sensitive) data.
Newer distros seem to fall apart after some setup.
udev? That doesn’t seem so broken as far as I’ve used it, though had some trouble manually hot swapping SATA HDDs during something critical at work.
That’s because they don’t have exclusive control over udev, unlike systemd.
The following will give you some idea of the sort of shit the kernel devs have to deal with on a regular basis from these two twats.
https://bugs.freedesktop.org/show_bug.cgi?id=76935
http://lkml.iu.edu//hypermail/linux/kernel/1404.0/01331.html
https://igurublog.wordpress.com/2014/04/03/tso-and-linus-and-the-impotent-rage-against-systemd/
http://www.phoronix.com/scan.php?page=news_item&px=MTY1MzA
No Wonder things like the Baytrail support, init, udev, DKMS (for Nvidia/ATI builds) and the audio subsystem are broken for years without solutions, the kernel devs are way too busy with those idiots you pointed out.
Thanks for pointing me to the root cause.
Not sure what is more broken: Windows 10 spyware or Linux+spaghetti-layer-userland.
I’m sure there are some decent Linux+alternative_userspaces out there despite partial systemd integration into Xorg.
“No Wonder things like the Baytrail support, init, udev, DKMS (for Nvidia/ATI builds) and the audio subsystem are broken for years without solutions, the kernel devs are way too busy with those idiots you pointed out.”
Oh yes indeed.
Fortunately Greg Kroah-Hartman has been able to mitigate a lot of their nonsense to some degree otherwise it would be an awful lot worse. I shudder to think what would happen if he suddenly decided he’d waded through the sewer long enough…
And no, the latter is not a reference to Linus’ sometimes rather intemperate language :)
KGPU sounds nice, but I’ve given up waiting for Linux to treat us PC users right. If you’re not a headless server with a gazillion cores and more ram than a Walsh farm. The kernel just doesn’t favour you.
That said it’s been a long time since I’ve used anything other than Debian. Not because of Linux but because of the GNU (the OS in “Linux OS”).
Tho I don’t want to start the GNU v Linux argument here.
Not because of Linux but because of the GNU (the OS in “Linux OS”).
Tho I don’t want to start the GNU v Linux argument here.
Yes, you do, otherwise you would have omitted this part.
That said, I prefer Linux with a truly free POSIX implementation, not that GNU garbage.
Define “truly free.” ;)
And I was thinking along similar lines when I saw the “Linux OS” part – not exactly the same lines, as I didn’t jump to the “hey, no, GNU is the OS!” conclusion, but I did immediately think “um, no, the kernel is not the heart of the Linux OS, the kernel is Linux, period.” The OS is something else which includes the Linux kernel, be it GNU or your mysterious freer-than-GNU POSIX implementation or even Android.
To abuse the race car driver analogy, I think the kernel could be considered analogous to the engine, or perhaps even the entire chassis and drivetrain. When a carmaker uses the same engine in several different cars, or even bases both a truck and an SUV on the same chassis and drivetrain, they still get different names and aren’t considered anything more than close siblings, not versions of the same car. Each car or truck has its own name, as does each OS that uses Linux as its kernel.
Being in the Linux family, or the GNU family for that matter, doesn’t make something Linux (or GNU/Linux!), it just makes it Linux-based. But the real name of the OS is Debian, or Fedora, or Slackware, or Trisquel, or Ubuntu, or OpenWRT, or Android, or…
Truly free = free from ideology = embedded with my ideology which doesn’t count as an ideology because those are things that only other people have
Linux hacking can be mighty frustrating. The way C is used by the linux developers can make it like its own dialect. And drivers are an endlessly moving target as kernel API’s change. Not much hacking these days going on with desktop x86 machines (great development hosts though, I am using one right now). But for embedded hacking, consider something like xinu, which currently runs on the beaglebone black and the intel galileo. A clean simple code base that is much easier to understand.
The Linux kernel is definitely just kept up as a fuck you from Linus to C++.
And I am totally on board with that.
Have you tried C++11? It is pretty swanky.
Almost every modern language has linked C objects if you dig deep enough.
However, ask yourself how many platforms does Linux actually support?
Linux with the GNU tools built most tech companies, and still runs most of the internet.
If you don’t understand why, than you are not even qualified to give an opinion on the smell of my farts.
Your rambling incoherent post seems to be directed at the wrong person, but I would guess that your farts smell like ass.
Right, up until I was startlingly disillusioned a couple of years ago, I thought once linux got a driver for something, you were set for life… nuh uh.
+1
The unfortunate question becomes what’s happened to the contributors… ?
Whereas once it was a matter of learning the api and then developing with it as-needed, and those contributions could be merged into the betterment of the opensource-community at-large, now those apis are changing so fast that folks who aren’t doing this stuff as full-time jobs are developing stuff that will be incompatible before they’re even released.
It’s great to hear sentiment like “woe is me, I have to learn something new,” at least it inspires people to keep learning. But remember that many of the developments that makes open-source as ubiquitous as it is wasn’t due to people working for corporations or even institutions, but due to people developing things where the OpenSource software being improved was merely a tangential *tool* toward their otherwise completely-unrelated end-goals.
Dude was building a house, improved the hammer.
Now the hammer’s only being improved by those trying to make money, by people who’ve never pounded a nail. We end up with hammers made of steel which wasn’t hardened, with screwdrivers on the handles jabbing us in the palm, and somehow they completely forgot about the claw for removing nails, apparently having no concept that nails sometimes bend-over, and weight-distribution went out the window *long* ago.
Please give an example about a kernel API that has changed significantly.
Porting an old driver to a new kernel is usually a matter of minutes.
And when there is an API change, it is usually acompanied with dozend of examples of how to convert drivers.
100% agreed Tucson. I’ve been a coder for over 20 years and when ever I see Linux code I think WTF, what monkey wrote that. Yer it works but if one of my junior coders wrote code like that, they would not be in my good books.
Most of the code in the kernel is bloody brilliant.
Tucson isn’t saying that the kernel is poorly coded, just the code only makes sense to a kernel programmer.
Remember the kernel coders write code needs to be clear and make sense to hundreds of people while being as fast as possible and not necessarily targeting any single platform.
What, So my first embedded system project shouldn’t be a real time 3D gamma ray scanner!?
Nice to see kernel related projects!
Though, I would personally say that the Linux kernel is a bit boring, and that is the reason I rather build a kernel from scratch myself, and yes, that sounds like building a real time 3D gamma ray scanner as a first project, but a kernel can be rather simple, as it’s mostly a task scheduler, memory manager and some system functions that are shared between all applications (Like putting text to screen, or handling network). Though, I do get the point of; why reinvent the wheel.
“Don’t tell me how to live my life!”
why is it that projects appear on hackaday when i work on them ??
no joke this is the third one!
but i guess it just meens that there are alot of people out there thinking the same things as i am :)
There’s almost an inane fear of messing with the kernel that I’ve never understood. Ultimately, it’s another piece of software… and no different in a lot of ways to programming environments like Arduino: it runs on the bare metal, and your application makes calls to that software to get the job done.
I can heartily recommend people have a look at the Industrial I/O interface, the LEDs frameworks and sysfs as a means for adding kernel-level smarts to devices connected via GPIO. Doing stuff in the kernel means you can handle the interrupts with far less latency than you otherwise would doing things from userspace.
Best of all, doing so is not rocket science. If you’ve programmed various microcontrollers in C, then moving to a SoC running Linux really isn’t much different. Once you have a kernel tree that builds and runs, you can start hacking it to bend it to your will.
Sorta been there before. Now proudly fluent in five obsolete programming languages.
And how many different obsolete assembly languages? But that is different really. Old processor architectures don’t become obsolete, they just become “cool and retro”. In somebodies eyes anyway.
Just Motorola 68000… Never a fan of 8080…
So what’s all this nonsense about systemd? New to me.
It seems irresponsible of the article to not mention that this is not something you would want to do… ever.
Giving this to a user as their first introduction to kernel-space work will surely have the result that this is the first thing to come to mind next time they need to implement an interface between user-space and kernel-space.
The article itself seems well written, it’s the premise of it that alarms me.
Support for thousands of devices will have been added to the kernel in 2016, how many of these involved adding a new system call? My money is firmly on 0 (although I could be wrong).
From [CRImier]:
“…I’ve also got an EEE PC 900 (701 with 9″ screen, no other changes) – 700MHz Celeron with 1GB RAM, and it serves great as a machine for occasional Linux x86 work development, pentesting shenaningans and listening to music – I now use CLI exclusively, but I used GUIs a lot and the slowness wasn’t usually dependent on CPU – storage, now that was a limiting thing until I bought a more recent flash drive (oh, and it all ran from USB flash). Again, recent kernel (Debian stock kernel) and systemd – it’s really fast, and the boot times are so much more quick with no-bullshit parallel loading.”…
I’ve often wondered why the EeePC 701 and 900 were not favorites of the hackercrowd: bright screen, runs Linux, smallest PC you’ll ever get, built like a tank, and on and on. The only personal experience I can offer is that my 701, bought immediately after the EeePC was announced in 2007 by Asus (I NEVER do that)(it never occurred to me to check the serial number) is still running strong.
I think I’ll buy another one.
They were, for quite a long time. There are hundreds of mods, people using it for things like wardriving, coding and any other portable tasks, forums entirely for EEEs and even 1 or 2 xkcd’s involving a EEE. Shame Linux compatibility sucked in many later models of EEEs. It’s very hackable and the parts are hella cheap now. Definitely do buy another one – I have 2 working and 2 more for parts, it’s a very reliable setup.
Another great way to “look under the hood” is to write a simple kernel module (i.e. device driver). For instance a character device or a module with sysfs interface. It doesn’t even need to do anything real or communicate with real HW. Compiling and loading a kernel module is a good way to start.
And there’s a great book about it, and its free: https://lwn.net/Kernel/LDD3/