Linux Fu: Getting Started With Systemd

I will confess. I started writing this post about some stupid systemd tricks. However, I wanted to explain a little about systemd first, and that wound up being longer than the tricks. So this Linux Fu will be some very fundamental systemd information. The next one will have some examples, including how to automount a Raspberry Pi Pico. Of course, by the end of this post, you’ll have only scratched the surface of systemd, but I did want to give you some context for reading through the rest of it.

Like many long-time Unix users, I’m not a big fan of systemd. Then again, I’m also waiting for the whole “windows, icon, mouse, pointer” fad to die down. Like it or not, systemd is here and probably here to stay for the foreseeable future. I don’t want to get into a flame war over systemd. Love it or hate it, it is a fact of life. I will say that it does have some interesting features. I will also say that the documentation has gotten better over time. But I will also say that it made many changes that perhaps didn’t need to be made and made some simple things more complicated than they needed to be.

In the old days, we used “init scripts,” and you can still do so if you are really motivated. They weren’t well documented either, but it was pretty easy to puzzle out the shell scripts that would run, and we all know how to write shell scripts. The systemd way is to use services that are not defined by shell scripts. However, systemd tries to do lots of other things, too. It can replace cron and run things periodically. It can replace inetd, syslog, and many other traditional services. This is a benefit or a drawback, depending on your point of view.

(Editor’s note: And this logging functionality was exactly what was abused in last week’s insane liblzma / ssh backdoor.)

Configuring systemd requires you to create files in one of several locations. In systemd lingo, they are “units.” For the purpose of this Linux Fu, we’ll look at only a few kinds of units: services, mounts, and timers. Services let you run programs in response to something like system start-up. You can require that certain other services are already running or are not running and many other options. If the service dies, you can ask systemd to automatically restart it, or not. Timers can trigger a service at a particular time, much like cron does. Another unit you’ll run into are sockets that represent — you guessed it — a network socket.


If you haven’t used systemd, the main interface to it is systemctl (not sysctl, which is something different). With it, you can enable and disable “units,” which are usually “services.” So, for example, my ssh server is a service unit. By defining a unit, someone could say, “Well, if you are starting in multiuser mode, wait for networking to be available and run the ssh server.” Or, sort of like inetd, run the ssh server when someone opens a connection on port 22. Not all units are services. Some are network sockets, and some are timers. There are other kinds, too. A timer is like a cron job. You can set them up to run every so often.

Systemd maintains two sets of units. One set is for the system, and you need to be root to work with those. But it also manages user-level things. For example, your sound server, probably pulseaudio or pipewire, has a systemd service unit. Not only does it launch the service, it restarts it if it dies. If you change a unit file, you have to tell systemd to reread all the files with systemctl daemon-reload. You can enable, disable, start, and stop services and other units with the systemctl command, too. There’s also a way to mask and unmask units, which is like a super disable.

While the documentation has become better over the years, the format for the units is still pretty hairy because there are so many options available. Most of the time, you only need a small subset of the many available options.

For Example

Here’s a fairly simple service unit:

Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5) auditd.service

ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID


You can probably puzzle most of that out. Here’s a very complicated service unit:

Description=Entropy Daemon based on the HAVEGE algorithm
After=apparmor.service systemd-tmpfiles-setup.service systemd-tmpfiles-setup-dev.service
# RNDADDENTROPY ioctl requires host-level CAP_SYS_ADMIN, fails in unprivileged container

ExecStart=/usr/sbin/haveged --Foreground --verbose=1 $DAEMON_ARGS
SuccessExitStatus=137 143

SystemCallFilter=@basic-io @file-system @io-event @network-io @signal
SystemCallFilter=arch_prctl brk ioctl mprotect sysinfo


This online form takes the tedium out of writing this purely fictional service (link in text).

Whoa! There’s a lot to parse there. You can find all the documentation, of course. But many people copy some starter service and modify it. Another option is to use a wizard to help you create your unit files. For example, you can find some online. Fill out the forms on the left and observe the file on the right. Installation instructions are up top. If you don’t know what a field asks for, there’s a little info button to give you some help text. You can often pick things from a drop-down. Sure, you can write it all yourself, but this does make it a little easier.

If you want to make a timer, you need a service file and a timer file. There’s an online wizard for that, too. It isn’t quite as helpful, and you need to understand how systemd represents time intervals. This is a bit more complicated than cron because you can set times relative to the timer’s activation, the time of system boot, or relative to the timer’s last activation. You can also set specific dates and times, add a random delay, or make timers persist.

That last option is like using anacron. Suppose you have a timer set to run at 1 AM daily on a laptop. The problem is that by 1 AM, the laptop is probably off. A persistent timer will realize that when you turn the laptop on at 8 AM that it should have run the 1 AM trigger, and then do so.

There’s More…

A similar web page can make your timers, too.

There is so much more to know about units. Some of that is related to features. Some of it is related to quirks. For example, comments can’t go at the end of a line — they have to be on their own line. For another example, special characters in the file names get hex escapes (this will be important later).  To further complicate things, there are ways to make templates and add in pieces. All of that is beyond the scope of this post, but be aware, I’m painting the simplest possible picture of the systemd universe.

The other problem is exactly where to put your unit files. They are scattered everywhere, and while there is a good reason for where they live, it is still annoying.

The short version is you should probably put system units in /etc/systemd/system unless you have a good reason to put it somewhere else. Units for all users can go in /etc/systemd/user. If you want a unit just for one user, try ~/.config/systemd/user. If you are looking for the location of an existing unit, usually “systemctl status xxx.service” will tell you. Otherwise, try “systemctl show -P FragmentPath xxx.service.”

If you want to read the actual documentation:

System unit directories

The systemd system manager reads unit configuration from various directories. Packages that want to install unit files shall place them in the directory returned by pkg-config systemd –variable=systemdsystemunitdir. Other directories checked are /usr/local/lib/systemd/system and /usr/lib/systemd/system. User configuration always takes precedence. pkg-config systemd –variable=systemdsystemconfdir returns the path of the system configuration directory. Packages should alter the content of these directories only with the enable and disable commands of the systemctl(1) tool. Full list of directories is provided in systemd.unit(5).

User unit directories

Similar rules apply for the user unit directories. However, here the XDG Base Directory specification is followed to find units. Applications should place their unit files in the directory returned by pkg-config systemd –variable=systemduserunitdir. Global configuration is done in the directory reported by pkg-config systemd –variable=systemduserconfdir. The enable and disable commands of the systemctl(1) tool can handle both global (i.e. for all users) and private (for one user) enabling/disabling of units. Full list of directories is provided in systemd.unit(5).

Stupid Trick #0: Run Some Program on Startup

This isn’t really a trick, but I wanted at least one example this time to show you how simple a homebrew service can be.

In an earlier post, I used a cheap “smart display” to show Hackaday’s RSS feed. If you want that to run on startup — or you just want the normal Python code to get the system status display — you need to run it somewhere. Sure, you could execute it by hand or put it in your startup files. But the systemd way to do it is with a unit. Here’s mine:

Description=[Operate Smart Display] 



That’s easy enough. It defines a simple service that outputs to the journal. It starts by default. Piece of cake. Next time, we’ll see a few more simple examples and maybe that will spur you to write your own unit files. Watch for it!

45 thoughts on “Linux Fu: Getting Started With Systemd

  1. Misguided goals of all kinds, systemd being just one of them, may be “here to stay,” but that doesn’t mean everyone has to join (or is joining) the lemmings running towards the edge of the cliff. Systemd’s authors appear to want it to do everything. That is a fundamentally misguided design goal. As a long-time Unix user, you surely do know better and it would have been a service to readers to at least point out 1) Linux distros exist that omit systemd, and 2) the objective fact that systemd’s attack surface is vast. Fortunately, the editor did…sort of, and comments can do the rest.

    More generally, complexity is anathema to determinism, and I still expect my IT to be deterministic. Given that more and more lemmings seem to be OK letting go of that expectation and they’re now even learning to trust “stochastic parrots,” opportunities for bad actors just keep increasing.

    1. I really don’t want to give the impression that I’m a systemd apologist. But it is sort of like the dvorak keyboard. QWERTY is objectively worse, but unless you never touch anyone else’s computer you should probably learn how to type on QWERTY. I may elect not to use systemd — you can remove it, roll your own distro, or find a distro that doesn’t use it. But for most of the people I have to interact with, it is there and — sorry — it isn’t going away soon no matter how much you and I might wish for that.

      We are seeing more and more software that isn’t “unixy” and we are suffering for it. But I don’t see any indication that the tide is changing anytime in the near future, as sorry as I am to have to say that.

      1. eh, i don’t think we need to learn how to use systemd. other people use it, just like people use android zygote or whatever replaced “Kicker” et al on macos, or whatever the heck nightmare manages sessions on Windows NT 10. they use it but they don’t know how it works, and if i sit down at their PC for a few minutes, i won’t have to know either. the one thing about these systems is that mostly they “just work”. as an outsider, you don’t need to get to the bottom of it. and on my own pc where i do need to get to the bottom of it, OH MY GOD, it is so easy to just uninstall systemd and bugs i didn’t even know about fix themselves immediately!

        the thing that really frustrates me about systemd is that when you really do need to learn to use it…generally because you’re trying to deploy a novel application of some sort in an environment you’re forbidden to uninstall systemd from…it’s very hard to make productive use of boilerplate examples. you need to understand what all the keywords mean, After vs. Requires vs. Wants kind of questions become of paramount importance. so you read the Unit / Service reference documentation and you still will find you need to unravel some chain of dependencies in already-existing services. and no sooner do you embark on that journey than you learn that there are abundant like … i just imagine it must look like literally if (!strcmp(service->name, “getty@.service”)) kind of bs inside systemd itself. you can’t unravel it. the existing OS-provided services don’t actually obey the abstraction you learned in the tutorial.

        so inevitably you wind up either shooting in the dark or on a soul-crushing deep dive that no tutorial can prepare you for. some people *do* need to understand systemd but they’re not going to have a gentle introduction to it. i really did try exactly what this article would suggest — do it the systemd way by learning one thing after another about systemd — and i really did hit the wall that *systemd does not work in the way it’s documented because the vast majority of it is below the surface of the water ripping giant holes in the side of the titanic*.

    2. By the way, it was systemd integration that enabled the recent XZ backdoor. Distros were patching sshd which dragged in libsystemd and thus liblzma, enabling compromise of the SSH daemon. So it’s been proven that systemd vastly increased the attack surface and enabled what would have been the worst backdoor ever in the history of open source software.

      Also, the use of systemd is not a “fact of life”, as mentioned in the article. People have choice over what software to run on their computers or not.

  2. Just went through this to set up a daily reboot service with a timer as a quick-fix for a memory leak on a kiosk system.

    I was having problems with a reboot loop, until I learned that the reboot service should not have the [Install] section included, unless you want the service to be started on every boot.

    Works like a charm now.

    1. A daemon is a specific program (e.g. diskmand) that runs in the background performing one or more tasks.
      A service is the name for the type of task to be done (e.g. disk management).

      So, a service (disk management) is implemented by a daemon (diskmand).

  3. ugh so much to disagree with. systemd is *not* here to stay — they’ve demonstrated that their fundamental goal is to constantly break everything. sure i hate it but if i loved it, that’s what i’d love about it too. there’s nothing else there to love. they break everything. and that is lovable, for a certain kind of moss-free rolling stone. so systemd isn’t here to stay, because it will be broken again and again. you won’t be able to recognize it in a decade. it’ll be something totally different.

    so it sounds like there’s nothing worthwhile to learn about systemd but i found that not to be the case. you can learn that they’re for breaking everything constantly and you’re against that, and once you learn it, you know you don’t want to use it, and don’t have to. and that’s the other thing wrong with this article. you just simply don’t have to. you can opt out.

    i spent a couple days learning systemd because i believed the errant premise of this document and i eventually learned, *i could not do what i want with systemd without hacking its C source code*. because that’s all it is — systemd’s developers fundamentally do not believe in the abstractions that they’ve built up (unit / service / Before= / After= / …) and everything it accomplishes in practice uses one-off hacks and magic features that somehow got shipped as production. the actual work isn’t done in shell scripts and configuration files, but in obscure undocumented hacks implementing complicated (and, imo, wrong) heuristics in C. so i followed the “debian without systemd” tutorial, and now i use devuan.

    of course, if you want effortless integrated gnome environment and don’t mind being completely unable to get to the bottom of any problem that comes up…then by all means, windows 95^H^H^H i mean systemd is right there for you. it certainly has its place. i’m not against other people using it, or ‘desktop distributions’ embracing it. but the important thing is: if systemd doesn’t have a place in your heart then you don’t have to use it.

    1. I always have to chuckle when I see people ranting about systemd. It reminds me of System D as practiced by the marines in “Falkenberg’s Legion.”

      > “What are they doing?” Falkenberg asked.

      “You name it. They’ve taken over a couple of taverns and won’t let anybody in without their permission, for one thing. And they have fights with street gangs every night.

      “We could live with all that, but they go to other parts of town, too. Lots of them. They go into taverns and drink all night, then say they can’t pay. If the owner gets sticky, they wreck the place. …”

      “And they’re gone before your patrols get there,” Falkenberg finished for him. “It’s an old tradition. They call it System D, and more planning effort goes into that operation than I can ever get them to put out in combat. I’ll try to put a stop to System D, anyway.”

    2. Thanks for hitting all the point I missed aiming for brevity, especially the Win95 comparison. Spot on! They’ve already added a BSOD. Now all the systemd devs need to add is the Registry and they will have achieved Win95 “perfection”…on top of the Linux kernel, no less! Oh, the ironies…

  4. Back when systemd was just starting to take off, I remember someone also came out with a makefile based init system that didn’t go anywhere. /sbin/init was mostly just a wrapper for make -C /etc/init.d -j CPUCOUNT and through makefile magic dependancies would be resolved and should something fail the wrapper script would catch the fallout and start an emergency shell if needed.
    It’s interesting the contrast between systemd and BSD startup scripts of old, or even stranger init systems like OpenVMS (it’s all DCL, all the way down, with a very linear startup progression) or AS/400 (you get a subsystem, and you get a subsystem.. we all get subsystems!)

        1. Devuan is not installed with systemd, so no need for `apt remove systemd`.

          My question, ‘What “apt install” was done before that “apt remove”?’, in other words:

          When systemd is removed, there needs to be another /sbin/init, which package does provide it?

  5. Not to pile on about on a systemd article that is at least as good as systemd deserves, but I’d also like to see a write up on systemd alternatives. If anything, the xz incident should push more people to think about getting away from systemd, and more guides about that will help.

    I’m still using crontab and @reboot for starting services, and I’m sure there’s plenty of tips and best practices around that we could be sharing instead of talking about systemd unit files.

  6. I’ve been using Linux as my desktop for 26 years. For the majority of them it has been Gentoo although there was Debian before and various RPM based distros before that. (Adjusting expectations for the time period… nothing will ever touch Mandrake for Desktop use)

    Anyway… I resisted SystemD. Until I really really wanted to try Anbox. And so I had to switch. Turned out I didn’t like Anbox all that much. But SystemD was already working.

    And that is what it has done. It has worked. No, I don’t notice any advantage over OpenRC. But I haven’t noticed much of a disadvantage either. Certainly not enough to make the effort to switch back.

    And honestly.. every version of every distro had such different helper functions and workflow to learn for writing an init script… The very few times I’ve had to write a unit… have been much more straightforward.

    I do hate the wordy names though. Even though it’s rarely necessary every example wants one to type the .service or .timer in every command. Every initscript I have ever started or stopped has been much shorter named!

    And the order of arguments to systemctl is just plain STUPID. “systemctl xable blah.service”. But you are likely to type two commands, one to start it and one to enable it. Make the action, start/enable/disable go last. That way one can just hit the up error, backspace over the action (or Alt-bkspc) and type the second action. Don’t put the action in the middle! What was wrong with those people?!!?

    But no, since installing SystemD my system has not failed, been hacked or really had anything bad happen to it that was a result of the switch.

    So not in love with SystemD. But don’t see a need to switch back either.

      1. Also, not all actions require a unit file and may require other flags, so `systemctl [action] [actions options]` makes more sense semantically.

        This is different from `service [service] [services action]`, where the action was more service-specific

    1. yeah i can really relate to this narrative, and i wish it was mine. i am definitely past the need to tear up everything i use — if it works i don’t have to touch it. i used macosx for about 5 years and i used it just like you are using systemd. it simply worked. i had a brief battle with Kicker but i was able to work out the one undocumented thing i was hung up on and then it worked exactly how i wanted it to, even though i didn’t understand how it was really all tied together.

      and of course i use android every day and i simply accept that a lot of the OS-provided APIs *don’t work and i will never get to the bottom of it*. isn’t that a funny way to tolerate software that “just works” lol

      i definitely agree it’s good to let a sleeping dog lie and i wish systemd hadn’t both failed to meet my needs and assertively interfered with all my workarounds. i would definitely still use systemd if i had an experience like yours.

    2. reversing the commands would be easy with a custom script. i do that for `mv` in case i want to move a bunch of files to one directory (or just easily move a file back that i recently moved, that has a long path). something like `function vm() { destination=$1; shift; mv $@ $destination; }`. you could do `function sysctl() { service=$1; shift; for action in “$@”; do systemctl $action $service; done; }` and run it as `sysctl blah.service start enable` in one go.

  7. An important detail for everyone’s sanity:
    /lib/systemd/ is the default service files and config with no customizations or enablement by the operator, like turning on services that aren’t on by default.
    /run/systemd/ is where the auto-created units go for things like mount points and the wrappers for sysvinit compatibility. It can and happily *will* override things in /lib/systemd/ as it’s specific to your system state.
    /etc/systemd/ is where most of the symlinks that define what’s enabled or disables go, as well as where any per-system customization goes. It overrides both /lib/systemd/ and /var/systemd/ and is essentially the last word.
    Everything in the systemd graph gets a file or symlink in one of these.

  8. I hated systemd until i wrote my first unit. Then, I wondered why things weren’t this simple earlier. It may have excessive scope, but denying its utility, especially for things that have no viable alternatives, is to limit oneself to inferior tools.

    I have some remarks for a follow-up post:

    – You missed the most important part of timers: unlike cron, they prevent duplicate jobs. I’ve seen many systems brought down by long-running crons piling up, and that’s just not a problem with timers.
    – The excellent cgroup-based isolation for all units (much of which is optional) is seemingly unavailable with other tools unless you’re actually running things in a full container system.
    – The resource controls offered by systemd.exec allow for far more effective limitations than other tools. The ability to easily set a memory limit for a large process and set its CPU and I/O shares low is fantastic. If you want to see how much better it is, run a heavy process with nice -n 19, then run it again with a single share via systemd, and see how much more responsive your system is.
    – Systemd is so far the only available tool for enabling KSM on processes that aren’t VMs.
    – You can use the fabulous systemd-run command to run processes in transient units, and can make use of every systemd control for said units. Very nice for resource limits on one-offs, or backgrounding a long-running process with full journald log capture and no need for tmux. This saved my bacon at work once, as I needed to run multiple long-running one-off processes across the fleet, but one cannot do so with Ansible and tmux. You can with Ansible and systemd-run, however.

    1. Thanks for going in and countering all that cheap negativity.

      I sign everything you’ve written. As a sysadmin, I consider those features you have mentioned absolutely critical. I’ve never looked back and thoroughly enjoyed the day we’ve finally migrated away from OS versions that did not have systemd.

      1. it doesn’t counter my negativity, anyways. people who need the features badly enough to justify the expense of the deep dive to get started obviously find it useful. i would never say that’s not the case. and so far as i can tell, it doesn’t look like Sum Gai would criticize my point that if you want to leverage the power of tools that do one thing well, then systemd isn’t going to help you.

        i mean, if you live in a world where long-running cron jobs piling up has been a problem for you then there is absolutely no denying that cron is the wrong tool for that job!

        my point isn’t that systemd is awful at everything, but that it is awful specifically for my needs, and the attitude that i have to learn to get along with it is fundamentally wrong. i don’t have to misuse systemd anymore than you all have to misuse cron!

    2. Also timers unlike cron can run as users and with dropped privs because they’re units again. I just need the systemd documentation to be written in a way that doesn’t require knowing the answer (e.g. finding an option name without knowing which section it’s in). It’s a great reference *once* I can get that far.

      Smaller log files would be nice too!

  9. Al’s right, we’re stuck with this desktop-wagging-the-server thing for a while. But, we don’t have to be. The things that systemd solves only exist for desktop system with attached mouse/keyboard/monitor, running a GUI and having to deal with USB plug-ins, WiFi connections, etc.

    Servers (including most SBCs), which are still the main Linux platform, don’t need that crap. They need maintainability and security that can be managed by the people running them, one bloody /etc/init.d/item at a time.

    The day the US defense agencies find that their servers are pwned from head to tail because the systemd maintainers had too many balls in the air, it will be gone, as well as any Federal IT contractor that doesn’t make it happen forthwith.

  10. Install may be harder/confusing next time, but my PC is getting long in the tooth (15 years old?), and I plan on installing Devuan (w/o a Windows partition) on my next PC to get rid of systemd on the current one.

    1. I ditched Windows entirely, and have been on devuan since the first stable release. The only reason I kept Windows was gaming, and Steam+Proton has got that covered now. You might find a couple multiplayer shooters that won’t work because of cheat protection or copy protection, but they’re few and far between now.

      1. I kept Win7 because some apps (such as connecting a cell phone to the PC) didn’t have Linux equivalents.
        But, I never gave Wine a second chance, way back when, I’ve heard it is much better now.


    * Category:Linux distributions without systemd

    * What are the best Linux distros that don’t use systemd?

    * No systemd

    * open-source operating systems without systemd in the default installation

    * Here are 14 Systemd-Free Linux Distributions

    * The Best Linux Distributions Without systemd

  12. Out of busybox, freebsd rc, openrc, sysvinit, and runit the only init system that has a proven track record of ruining my projects since 2018 (debian jessie) is systemd.
    It boots slower on low end embedded systems (an extra 1-2 seconds counts in some applications)
    It can sometimes wait for minutes when powering down (2+ seconds for clean poweroff can be a deal breaker in most of my use cases)
    It adds extra complexity where non is needed (why do I need to have a service manager when the service can be run in a shell while loop called from rc.local?)
    It introduces dependency hell for small projects (I don’t want to pull in 20+ libraries for a minimal desktop)

    People will accuse me of not configuring things properly. To them I ask: Why should I have to constantly reconfigure everything out of the box when the old way JUST WERKd, and still JUST WERKS?
    Others will accuse me of being stuck in the past. To them I say: I think I’ll stick with the old way if the price of the new way is the opportunity to brick your motherboard.
    I think I’ll stick with things that have given me 20+ years of reliable, predictable behavior, that run extremely well on $9 computers, and obscenely well on $50+ computers.

  13. I also dislike systemd and I still prefer to run debian rather than devuan. To remove it from a recent debian system:
    # apt-get install sysvinit-core sysvinit-utils
    # cp /usr/share/sysvinit/inittab /etc/inittab
    (you may also need to edit /etc/default/grub if you use it)
    then place in /etc/apt/preferences (so that it won’t get installed again when updating)
    Package: systemd*
    Pin: origin *
    Pin-Priority: -1
    and finally remove and reboot
    # apt-get remove –purge –auto-remove systemd

    I also replace udev (which is another troublesome package by the same author) with eudev
    and fix it in /etc/apt/preferences too.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.