Linux, Now In Real Time

Although Linux runs almost every supercomputer, most of the web, the majority of smart phones, and a few writers’ ancient Macbooks, there’s one major weak point in the Linux world that will almost always have developers reaching for a different operating system. Linux is not a real-time operating system (RTOS), meaning that it can’t respond to requests in the real world within a set timeframe. This means that applications needing computer control in industry, medicine, robotics, and other real-world situations generally need a purpose-built RTOS. At least, that was true until recently when an update to the Linux kernel added real-time capabilities.

The feature, called PREEMPT_RT, forces the Linux kernel to respond to certain request within a set limit of time. This means that a system with this support built into the kernel can “preempt” any current task, stopping everything else a computer is doing in order to execute that task right away. There are some existing solutions to getting a functional equivalent system working with Linux now, but they can be clunky or inelegant, requiring the user to install patches or other software to get it to work. With the support built directly into the kernel this will become much less of a pain point for anyone who needs this functionality going forward.

This feature has been in the works for around two decades now, so with this entering general use now we would expect to start seeing it show up in various projects as well as in commercial offerings soon, especially since other RTOS solutions can be pricey. Don’t recompile the kernel in your desktop for this feature just yet, though; real-time function can cause some unintended consequences with normal use you’ll need to account for. There’s some more discussion on this in the /r/Linux subreddit and there are some other real-time operating systems available for computers not typically capable of running Linux to take a look at as well.

27 thoughts on “Linux, Now In Real Time

  1. Okay, so it is not a Real Time system, but does give priority to declared critical operations.
    At this point, I’m okay with that, if I need a Real Time system, they are available.

    1. First define realtime. linux has been realtime for decades, if realtime behavior is ‘within a day’. See, it all depeds. though to be fair, wecan now forcefully switch to a task withhin a certain time frame. Before, ‘realtime’ meant of course ‘within 1000hz’ as per scheduler setting.

      Imo sill a pretty big win :D

        1. No, it’s not. A kernel space (the system) will always run even if the user space is misbehaving (or it’s a bug). There’s no way for a process to hang the system (except for a loop in the init process). The PREEMPT_RT patch simply allow the user space to trigger a kernel’s preemption upon certain condition while maintaining kernel integrity in all its own structure. You can still hang your user space processes (for example with mutexes with wrong/no priority inversion), but the kernel shouldn’t break in that case.

          The patch lived for many year and was functional already. There’s also the Xenomai RT kernel that let run linux as a user-space “kernel process” that was very successful.

      1. Except that isn’t what “real time” means in the context of an OS.
        It means “an interrupt/operation is guaranteed to be serviced within a window of X to Y cycles”, where X and Y are predefined.

        A soft real time system is a little more lax, where it will ALMOST always meet that window. +95% X-Y but always less than Z.

        It isn’t about being fast.
        It is about repeatability.

        If a sensor updates values every 100 cycles you don’t want a random read to take 250 cycles when you expect it every 100.
        But you ALSO don’t want to have a read happen at 91 cycles.

        Wikipedia /real-time_computing

        1. You probably meant this but where you say “an interrupt/operation is guaranteed to be serviced within a window of X to Y cycles” that includes the scenario of failing to service, and returning with a timeout/error condition in that X/Y time. Just for completeness.

    1. I was about to say that. If you write a kernel driver you get access to interrupts and can respond to real-time events in real-time. Just that argument breaks down a little once you transition to user space.

  2. How does this compare to RTAI? I did some basic tests with RTAI and Scilab/SciCos around 20 years ago. Back then this and RTLinux were two available options to run real time apps. I have no further experience wit real time systems.

    1. It’s best to think of PREEMPT_RT as a set of patches to allow you to make an RTOS out of Linux itself, rather than a hybrid cokernel approach like Xenomai/RTAI, but it doesn’t turn Linux into an RTOS right away. There’s no magic you can do to make that work anyway, there’s always going to be a hardware component, period – any misbehaving hardware can cause problems, and if your process is waiting for external data through an extremely slow driver, you might be screwed period.

      Just as an example, I’ve worked with systems that have “extremely soft real-time” requirements: we built in enough buffering and a framework that could handle delays on second timescales. The problem was that the way the software has taking data, it was buffering things so large that it would occasionally have to dump a gigabyte-sized buffer to disk, and hey, that just took time, period. Obviously that was fixable by changing buffer sizes, but the basic idea there is that you need to be able to handle the throughput, period, and if your attempts to make sure you can respond in time kill that throughput, you’re toast.

      Basically PREEMPT_RT makes IRQ handlers all threaded, and gets rid of spinlocks in favor of other mechanisms that can handle priority inheritance properly in virtually all sections.

      Plus there’s also the idea of a “formally proven” RTOS, where you can say “no, this can’t possibly happen” and you have a definitive worst-case execution time (WCET) which is unlikely to ever happen with mainline Linux even with PREEMPT_RT just because it’s so big (even if the kernel’s preemptible, there’s no guarantee that the latency won’t be too large because it has too much to do).

      This doesn’t mean that this isn’t a huge deal – PREEMPT_RT was an incredibly ambitious idea when it began and it’s already led to huge amounts of changes and improvements in Linux overall. I’ve seen PREEMPT_RT described as ” promising when adopted in industrial environments only if the hard real-time requirements are not dictated by safety-critical constraints.”

  3. No it has not. Linux is a GPOS not an RTOS. Semantics aside as soon as some other task fires up, the timing of critical tasks is effected. Lose a single step in CNC and your part is toast.

  4. EMC2/LinuxCNC have been running on top of RTAI or PREEMPT-RT for 25 years, and do quite well for CNC control. “hard real-time” kinda depends on how small a time resolution you require to be deterministic, of course. But I’ve been making pretty nice milled parts on a linux system for 20 years.

    1. Yeah, especially LinuxCNC with 1 kHz control loop combined with one of the Mesa FPGA boards has worked well for me. The direct stepper driving with 100 kHz realtime loop on PC was a bit of hit-and-miss.

      So far even if the PC has been heavily swapping, it has kept up with the realtime tasks without exceeding the 0.1 ms deadline. The FPGA board has its own watchdog that will stop the machine should the PC hang up, but that hasn’t happened to me yet.

      1. Please do – a decent open-source toolchain guide for CNC would be great!

        Seems like the universal expectation for home CNC is an old version of Windows and Mach4/5 and that just feels like there should be a better way especially given how rich the offerings are for 3D printers now by comparison.

  5. Linux has had RT capability for at least 25 years. Not in an unpatched mainstream kernel, sure, but the capability has been around for literally decades. My coworker Pierre-Phillipe used to set up demos of it at Lineo.

    1. Yeah, I wish this was available from the get-go. Could have saved a lot of time back then without having to apply the patchs to use Xenomai or RTAI…. Not that we needed ‘hard’ real-time, but needed our SCADA threads to all have highest priority and not be held up by lower level activities…. Leave the ‘hard’ Real-time to the RTUs (which ran a RT kernel like VRTX) in the field where control actions and milli-second time stamping took place.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.