Realtime process management

This article provides information on prioritizing process threads in real time, as opposed to at startup only. It shows how you can control CPU, memory, and other resource utilization of individual processes, or all processes run by a particular group.

While many recent processors are powerful enough to play a dozen video or audio streams simultaneously, it is still possible that another thread hijacks the processor for half a second to complete another task. This results in short interrupts in audio or video streams. It is also possible that video/audio streams get out of sync. While this is annoying for a casual music listener; for a content producer, composer or video editor this issue is much more serious as it interrupts their workflow.

The solution is to run time-sensitive processes in realtime. In linux, this means changing the process to a realtime scheduler, like SCHED_RR or SCHED_FIFO. See sched(7) for descriptions of these schedulers.

Configuration

On Arch Linux, system, group and user-wide configuration can be achieved using PAM and systemd.

The realtime package group provides additional tools to modify the realtime scheduling policies of IRQs and processes.

Note: When running a custom kernel, ensure it is PREEMPT enabled to make use of the methods mentioned above.

Configuring PAM

The /etc/security/limits.conf file provides configuration for the pam_limits PAM module, which sets limits on system resources (see limits.conf(5)).

Tip: It is advised to externalize the configuration of pam_limits to separate files below /etc/security/limits.d as those take precedence over the main configuration file.

There are two types of resource limits that pam_limits provides: hard limits and soft limits. Hard limits are set by root and enforced by the kernel, while soft limits may be configured by the user within the range allowed by the hard limits.

Installing the package realtime-privileges and adding the user to the realtime group, provides reasonable default values (e.g. relevant for Professional audio).

Configuring systemd services

Processes spawned by systemd system services need to specifically set up equivalents to limits.conf. Further information can be found in the sections systemd.exec(5) § CREDENTIALS and systemd.exec(5) § PROCESS PROPERTIES in systemd.exec(5).

Usage

To run in realtime, a process running in the realtime group must set LIMIT_RTPRIO to any value above 0, then change to a realtime scheduler, like SCHED_FIFO or SCHED_RR. Here is an example:

#include <assert.h>
#include <sys/resource.h>
#include <sched.h>

void realtime()
{
    struct rlimit rl;
    assert(getrlimit(RLIMIT_RTPRIO, &rl) == 0);
    assert(rl.rlim_max != 0);
    rl.rlim_cur = rl.rlim_max;
    assert(setrlimit(RLIMIT_RTPRIO, &rl) == 0);

    assert(sched_setscheduler(0, SCHED_FIFO, &(struct sched_param){
            .sched_priority = rl.rlim_cur
        }) != -1);
}

Again, LIMIT_RTPRIO must be above 0 for the process to be able to change its scheduler. This is why realtime-privileges raises the rtprio limit to 98 for all processes in the realtime group. See sched(7) for more.

Using chrt

To do this automatically, or in case you are running somebody else's software, you can use chrt. Here is an example of starting ls -al in realtime under the SCHED_FIFO scheduler:

$ chrt --fifo 98 ls -al

This shouldn't normally be required. Applications that need to run in realtime, like jack2, normally set their schedulers all on their own.

Hard and soft realtime

Realtime is a synonym for a process which has the capability to run in time without being interrupted by any other process. However, cycles can occasionally be dropped despite this. Low power supply or a process with higher priority could be a potential cause. To solve this problem, there is a scaling of realtime quality. This article deals with soft realtime. Hard realtime is usually not so much desired as it is needed. An example could be made for car's ABS (anti-lock braking system). This can not be "rendered" and there is no second chance.

Tips and tricks

PAM-enabled login

Warning: Before making any changes in /etc/pam.d, read PAM to avoid breaking your system's authentication or creating security issues. Default installations should work without changes to this configuration.

PAM is installed and configured on default Arch Linux installations. Nearly all display managers are pam-enabled, too. You can check which modules use pam_limits.so with the following command:

$ grep pam_limits.so /etc/pam.d/*

On a default installation, only two files use the module directly:

/etc/pam.d/system-auth:session    required                    pam_limits.so
/etc/pam.d/system-services:session   required    pam_limits.so

/etc/pam.d/system-auth is included by other pam-aware applications. You can further search /etc/pam.d:

$ grep system-auth /etc/pam.d/*

In the default configuration this shows which applications are using system-auth and thus pam_limits.so. For example

...
/etc/pam.d/su:password        include         system-auth
/etc/pam.d/su-l:password        include         system-auth
/etc/pam.d/system-login:auth       include    system-auth
/etc/pam.d/system-login:account    include    system-auth
/etc/pam.d/system-login:password   include    system-auth
/etc/pam.d/system-login:session    include    system-auth
/etc/pam.d/system-services:account   include     system-auth

This covers both graphical and console login.

Checking Limits

In Bash, use ulimit (see bash(1) § ulimit) to check limits, for example the rtprio limit can be checked as follows:

$ ulimit -r
98

This is the value configured by realtime-privileges.