The best kind of Hackaday posts are the ones where there was some insurmountable problem with an elegant solution devised through deep analysis of the problem and creativity. This is not one of those posts. I’m sure you are familiar with bit rot. You know, something works for a long time and then, for no apparent reason, stops working. Well, that has been biting me, and lacking the time for the creative, elegant solution, I decided to attack it with a virtual chainsaw.
It all started with a 2022 Linux Fu about using autokey.
The Problem
I use autokey to give me emacs-style keystrokes in Web browsers and certain other programs. It intercepts keystrokes and translates them into other keystrokes. The problem is, the current Linux community hates autokey. Well, that’s not strictly true. They just love Wayland more. One reason I won’t switch from X11 is that I haven’t found a way to do something like I do with autokey. But since most of the powers-that-be have decided that X11 is bad and Wayland is good, X11 development is starting to show cracks.
In particular, autokey isn’t in the normal repositories for my distro anymore (KDE Neon). Of course, I’ve installed the latest version myself. I’m perfectly capable of doing that or even building from source. But lately, I’ve noticed my computer hangs, especially after sleeping for a long time. Also, after a long time, I notice that autokey just quits working. It is running but not working and I have to restart it. The memory consumption seems high when this happens.
You know how it is. Your system has quirks; you just live with them for a while. But eventually those paper cuts add up. I finally decided I needed to tackle the issue. But I don’t really have time to go debug autokey, especially when it takes hours for the problem to manifest.
The Chainsaw
I’ll say it upfront: Finding the memory leak would be the right thing to do. Build with debug symbols. Run the code and probe it when the problem comes up. Try to figure out what combination of X11, evdev, and whatever other hocus pocus it uses is causing this glitch.
But who’s got time for that? I decided that instead of launching autokey directly, I’d launch a wrapper script. I already had autokey removed from the KDE session so that I don’t try to start it myself and then get the system restaring it also. But now I run the wrapper instead of autokey.
So what does the wrapper do? It watches the memory consumption of autokey. Sure enough, it goes up just a little bit all the time. When the script sees it go over a threshold it kills it and restarts it. It also restarts if autokey dies, but I rarely see that.
What’s Memory Mean?
The problem is, how do you determine how much memory a process is using? Is it the amount of physical pages it has? The virtual space? What about shared libraries? In this case, I don’t really care as long as I have a number that is rising all the time that I can watch.
The /proc file system has a directory for each PID and there’s a ton of info in there. One of them is an accounting of memory. If you look at /proc/$PID/smaps for some program you’ll see something like this:
00400000-00420000 r--p 00000000 fd:0e 238814592 /usr/bin/python3.12 Size: 128 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 128 kB Pss: 25 kB Pss_Dirty: 0 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd mr mw me sd 00420000-00703000 r-xp 00020000 fd:0e 238814592 /usr/bin/python3.12 Size: 2956 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 2944 kB Pss: 595 kB Pss_Dirty: 0 kB Shared_Clean: 2944 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB . . .
Note that there is a section for each executable and shared object along with lots of information. You can get all the PSS (proportional set size) numbers for each module added together like this (among other ways):
cat /proc/$PID/smaps | grep -i pss | awk '{Total+=$2} END { print Total}'
Building the Chainsaw
So armed with that code, it is pretty easy to just run the program, see if it is eating up too much memory, and restart it if it is. I also threw in some optional debugging code.
#!/bin/bash
#- Run autokey, kill it if it gets too big
#- what's too big? $MLIMIT
MLIMIT=500000
#- how often to check (seconds)
POLL=10
#- Print debug info if you want
function pdebug {
#- comment out if you don't want debugging. Leave in if you do
#- echo $1 $2 $3 $4
}
while true # do forever
do
PID=$(pgrep autokey-qt) # find autokey
pdebug "PID",$PID
if [ ! -z "$PID" ] # if it is there
then
# get the memory size
PSS=$(cat /proc/$PID/smaps | grep -i pss | awk '{Total+=$2} END { print Total}')
pdebug "PSS", $PSS
echo $PSS >>/tmp/autokey-current.log
# too big?
if [ "$PSS" -gt "$MLIMIT" ]
then
pdebug "Kill"
echo Killed >>/tmp/autokey-current.log
# save old log before we start another
cp /tmp/autokey-current.log /tmp/autokey-$PID.log
kill $PID
PID=
sleep 2
fi
fi
if [ -z $PID ]
then
# if died, relaunch
pdebug "Launch"
autokey-qt & 2>&1 >/tmp/autokey-current.log
fi
pdebug "Sleep"
sleep $POLL
done
In practice, you’ll probably want to remove the cp command that saves the old log, but while troubleshooting, it is good to see how often the process is killed. Running this once with a big number gave me an idea that PSS was about 140,000 but rising every 10 seconds. So when it gets to 500,000, it is done. That seems to work well. Obviously, you’d adjust the numbers for whatever you are doing.
Bad Chainsaw
There are lots of ways this could have been done. A systemd timer, for example. Maybe even a cgroup. But this works, and took just a few minutes. Sure, a chainsaw is a lot to just cut a 2×4, but then again, it will go through it like a hot knife through butter.
I did consider just killing autokey periodically and restarting it. The problem is I work odd hours sometimes, and that means I’d have to do something like tie it to the screensaver. But I agree there are dozens of ways to do this, including to quit using autokey. What would your solution be? Let us know in the comments. Have you ever resorted to a trick this dirty?
I use lintalist https://lintalist.github.io/ on windows. A lot. For many apps like word and outlook, and Windows Explorer tasks. Also web browser URL and quick password entries that are repetitious. Lintalist will give different results and options depending on which Window is in-focus.
I’ve never found something for my Linux usage that has the same abilities.
Tools like this used to be available, and often underpinned accessibility functionality. Between the switch to a new windowing system and the major desktop developers going in other directions most of that tooling will not work on current systems and needs to be overhauled. This is actually happening, but it’s very slow as a factor of the smaller number of interested developers. Ideally a business with a vested interest would chip in, but big companies that often used to contribute towards this sort of goal either no longer do or are holding off from that sort of investment in the current economy.
That’s great, but it’s pretty annoying when people interject windows comments on a clearly a Linux centered post. Read the title.
Are you really trying to fan-gate Linux?
The community wants to have a word with you, outside, in 30s.
I’ve used this hack to restart Apache when the OOM-Killer kept killing it off. We knew what was wrong, but didn’t have the authority to make changes to fix the resource usage. So, instead I scripted up a solution that restarted Apache when it was killed and logged the occurrences, so I’d have proof of how often it happened.
I like it. In my case, though, the program would quit working but didn’t die. OOM might have eventually killed it but it would be “dead” long before that.
Isn’t the whole purpose of systemd services that it will restart them automatically?
Only if explicitly dictated in the service file. You’d be surprised how many of them won’t have a Restart clause
Systemd services for a gui tool?
>Blah blah blah technobabble blah blah more technobabble.Meanwhile on Windows, AutoHotkey just works™.
If you’re using OS designed for telephone exchanges, which was crudely adapted to do general-purpose GUI computing then expect things to be broken.
Don’t think that’s the OS that was designed for telephone exchanges…
Meanwhile on Windows, you don’t even have the freedom to move the taskbar to the top…
H’mmm,
Mine is at the top.
Win 10 Pro.
Look in the settings for the taskbar.
Windows is way simpler:
– don’t like the Start menu? too bad, that’s what you got.
– don’t like File Explorer? well, be used to it.
– don’t like the system tray, or the control panel, or how things work? that’s your problem.
Linux have way more ways to do things, and that is both a good and bad thing. You can have beautiful distros, others heavily customized, with security pretty locked down, or a light one that runs on early 2000’s hardware without issues. It’s yours, treats the computer as yours, so you do whatever you want with it.
Windows you have what is supported and that’s it. If your computer don’t have a TPM2, it’s not officially supported. I know you can run it (I do), but MS says you cannot and you have to work around the setup. It treats the computer as MS asset, so it does what MS allows you to do. To do what you want, you have to break the EULA sometimes.
And all that “Linux don’t have a proper GUI” is a joke as old as NT 3.5, come on… It have a proper GUI at least since WinXP was around…
Wayland is the real problem here.
achkshually, the compositor is, no? wayland is a protocol, compositor implements it?
Modern Linux seems to hate CRT monitors. 1024×768 is all you will ever get on those.
Ten years ago, the system would query the monitor’s I²C chip what modes it supports and select one that it thinks would look good on the display, but allowing you to switch resolutions (who remembers CTRL-ALT-PLUS or MINUS?).
One day, you had to start cheating because all you got was 1024×768. On a 14 inch CRT, no problem, but what if I had a 17 or even 19 inch one? Or a friggin’ overpowered CAD monitor?
Until Ubuntu 20.04, you could inject modlines into xrandr to tell it your monitor can do 1920×1440@70Hz (and is still damn sharp, but with true black and really vibrant colors, looks absolutely unreal* if you’re used to LCD panels or oldskool TVs) – until one day you couldn’t. Some update just thrashed that. Back to 1024×786 with no (re)solution whatsoever.
On 22.04 you could tell the NVidia control panel what horizontal bandwidth your monitor is capable of and you’ll get a list of resolutions the program thinks are convenient (1920×1440, albeit at a slightly flickery 60Hz).
Haven’t tried installing 24.04 on that machine yet, haven’t really used it in years.
(and well yes I know, it’s not CRT monitors, it’s VGA because it also happens with LCD monitors that only have VGA)
(and no, I’m not a Linux Guru, just an average user who knows a little bit of Bash and thinks Windows is a $h!tsh0w since Win95 – not that 3.1x was any better but I grew up on that and feel nostalgic for it and still like it a lot. And it’s got a few cool features modern Windows is missing.)
*) you could say – unparalleled… hehe… parallel… because CRTs and geometry is quite a challenge… but my monitor is pretty well adjusted
Try the CVT tool to generate mode lines (for X; if you are suffering Wayland then good luck!). With this I managed to tweak 4k @ 30ish Hz VGA over an analogue Aten KVM over RJ45 switch.
oh my friend! Solve that particular problem with QMK! That way, your keystrokes and macros go anywhere your keyboard does, sans any host-side software to set up, configure, and maintain. Works on locked-down corporate machines, cross-OS’s, and persists across updates, re-installs, and changes.
Get yourself an old terminal keyboard or maybe a Sun 5 with a bunch of extra keys on it, wire in a devboard to the matrix, and you can do all sorts of fun things with it!
I do understand the article was about memory, but to add another suggestion for alternatives to autokey and QMK: Keyd, from rvaiya on GitHub.
To quote its documentation:
Linux lacks a good key remapping solution. In order to achieve satisfactory results a medley of tools need to be employed (e.g xcape, xmodmap) with the end result often being tethered to a specified environment (X11). keyd attempts to solve this problem by providing a flexible system wide daemon which remaps keys using kernel level input primitives (evdev, uinput).
X11 hasn’t received a stable release since June 6th, 2012.
if i had to do that, it would be on my laptop, and i would tie restarting the thing into my suspend script. nothing magic, you can get in the middle of that process just by playing with /etc/acpi/lid.sh
i really relate to being too lazy to properly debug something but it really gets under my skin too.
on X11, i use XTestFakeInput() to generate synthetic key/mouse events, which i imagine is the same X protocol extension that autokeys uses. it kind of sucks. there are subtle differences between authentic input events and XTest-generated events, which i don’t think the user can fully compensate for. especially for mouse events. the result is there’s a few specific programs where the menus only work with the real hardware trackpad. most things work the same though.
if i was to really get to the bottom of it, i’d probably make a custom xorg input driver so the X server can’t tell the difference between a real and synthetic event.
i believe the myth that there was a crew of people hacking on Xorg like 10-20 years ago who cleaned up a lot of old hacks and then ran into the really baked-in problems and bailed and formed wayland. anyways, one of the things they did was decentralize the x11 build process. so now to replace the input driver, you can like download the source to “xf86-input-libinput”, which is a relatively tiny thing to build. and then “apt install xserver-xorg-dev” and it will build independently!
it used to be such a huge pain in the butt to rebuild all of xfree86, but these days you just download the module you’re interested in, hack it, and then overwrite the file in /usr/lib/xorg/modules.
so my point is i just want to believe that there’s a fairly trivial way to replace the input driver on wayland as well. but i don’t know because i’ve never used it :)
If you have the source, compile it in debug mode and run under valgrind.
Valgrind will keep track of the memory, and on exit tell you which memory blocks have not be free’d and the source and line where they were allocated. That’s probably enough info to show you where the error is.
Valgrind will also flag line and source for other errors, such as freeing memory twice.
For my personal code development, I have the following aliases:
alias pev=”valgrind –leak-check=yes $BINDIR/$TESTEXEC $TESTARGS”
alias pev1=”valgrind –leak-check=full –show-leak-kinds=all $BINDIR/$TESTEXEC $TESTARGS”
Program only needs to be compiled in debug mode for this to work.
sounds blissful.
I’ve always found the cost of bliss to be excessive, though.
Bash has ulimit builtin. Script could call it with -m or -v or -d, then do “while true; do autokey-qt; move_log_files; done.”
Personally, I use xremap on Wayland to give me those emacs style bindings in web browser and other apps. If you aren’t wedded to auto hotkey, give it a try.
Clearly you have never been to a Tux/Gown party.
I hated Wayland because I DO use remote display.
Then I heard about Waypipe.
That seemed to work well.
Then I tried to get tricky, I was hoping I could run a desktop environment over Waypipe to have something like a Wayland based VNC.
I don’t know what I did but Waypipe won’t work anymore. I just get some sort of permission errors. I think it has to do with trying to write something to a temp folder… but I can’t tell from the cryptic messages if it is talking about client side or server side. Or which side even is client and server with Wayland? And the ‘docs’… not helpful.
Back to hating Wayland. It all still just works with X. Yes, even Kde/Plasma still works, it just isn’t default anymore.
But… Cosmic looks interesting. It might have just the blend of tiling and windows… So back to Wayland. Damnit!
Talking of linux bugs, I run an old Acer laptop with 3GB of RAM and it always had this unreliability with the WiFi, eventually it would give a kernel error and only way of restarting it was a power cycle reboot. Long story short, before I got around to creating some shonky systemd (I HATE SYSTEMD!) script, another problem came up, the BIOS got corrupted. Then it wouldn’t boot at all.
So, to reset the BIOS you have to pull the WiFi network and RAM out and short a jumper on the motherboard. Put it all back together, all was good.
Then I noticed, the WiFi now behaves itself! (Mostly). All that time, it probably just needed reseating!
So… before you blame the software….
If autokey continues giving you issues, you should look at keyd (https://github.com/rvaiya/keyd) – it works at the level of the kernels input system, so it works on both X11 and Wayland. Then your choice won’t be constrained by this feature, at least.
somewhat related: i’ve recently switched to mac and the keyboard shortcuts are driving me insane. is there an autohotkey for mac that lets me remap the shortcuts?
Is there a way to set a memory cap on processes like you can on containers?
What about xwayland? Doesn’t that like run X11 apps on wayland?
A fun chainsaw! When I’ve done this trying to root cause something similar…my chainsaw was a little bigger & more complicated. This wasn’t Linux but the same method could be used (although commenter(s) noted, valgrind & similar tools are purpose built for this problem.. sadly not an option in my case.. also mine were not well localized since there was effectively a single process container.. so similar but different)
The system collected a partial memory dump, and I collected a bunch of dumps. Each was statistically analyzed, first counting the allocated number for each size of memory element. Dump collection had bias to save recently created/accessed elements which helped bias towards the offender. After identifying top sizes, each size element was analyzed for most frequent values within the element. Finally a search for a similarly constructed structure started. Which fortunately provided enough of a clue to locate root cause.
It was a very slow process, but successful in several memory leaks but far better to catch them before code release IMHO!