Frequently Asked Questions
This page contains a list of frequently asked questions about the real-time support in the Linux Kernel.
What is real-time?
Real-time applications have operational deadlines between some triggering event and the application's response to that event. To meet these operational deadlines, programmers use real-time operating systems (RTOS) on which the maximum response time can be calculated or measured reliably for the given application and environment.
A typical RTOS uses priorities. The highest priority task wanting the CPU always gets the CPU within a fixed amount of time after the event waking the task has taken place. On such an RTOS the latency of a task only depends on the tasks running at equal or higher priorities, all other tasks can be ignored. On a normal OS (such as normal Linux) the latencies depend on everything running on the system, which of course makes it much harder to be convinced that the deadlines will be met every time on a reasonably complicated system. This is because preemption can be switched off for an unknown amount of time. The high priority task wanting to run can thus be delayed for an unknown amount of time by low priority tasks running with preemption switched off.
Where can I read more about real-time?
Have a look to Publications page of this wiki.
What are real-time capabilities of the stock 2.6 linux kernel?
Traditionally, the Linux kernel will only allow one process to preempt another only under certain circumstances:
- When the CPU is running user-mode code
- When kernel code returns from a system call or an interrupt back to user space
- When kernel code code blocks on a mutex, or explicitly yields control to another process
If kernel code is executing when some event takes place that requires a high priority thread to start executing, the high priority thread can not preempt the running kernel code, until the kernel code explicitly yields control. In the worst case, the latency could potentially be hundreds milliseconds or more.
The Linux 2.6 configuration option CONFIG_PREEMPT_VOLUNTARY introduces checks to the most common causes of long latencies, so that the kernel can voluntarily yield control to a higher priority task waiting to execute. This can be helpful, but while it reduces the occurences of long latencies (hundreds of milliseconds to potentially seconds or more), it does not eliminate them. However unlike CONFIG_PREEMPT (discussed below), CONFIG_PREEMPT_VOLUNTARY has a much lower impact on the overall throughput of the system. (As always, there is a classical tradeoff between throughput --- the overall efficiency of the system --- and latency. With the faster CPU's of modern-day systems, it often makes sense to trade off throughput for lower latencies, but server class systems that do not need minimum latency guarantees may very well choose to use either CONFIG_PREEMPT_VOLUNTARY, or to stick with the traditional non-preemptible kernel design.)
The 2.6 Linux kernel has an additional configuration option, CONFIG_PREEMPT, which causes all kernel code outside of spinlock-protected regions and interrupt handlers to be eligible for non-voluntary preemption by higher priority kernel threads. With this option, worst case latency drops to (around) single digit milliseconds, although some device drivers can have interrupt handlers that will introduce latency much worse than that. If a real-time Linux application requires latencies smaller than single-digit milliseconds, use of the CONFIG_PREEMPT_RT patch is highly recommended.
How does the CONFIG_PREEMPT_RT patch work?
The RT-Preempt patch converts Linux into a fully preemptible kernel. The magic is done with:
- Making in-kernel locking-primitives (using spinlocks) preemptible though reimplementation with rtmutexes.
- Critical sections protected by i.e. spinlock_t and rwlock_t are now preemptible. The creation of non-preemptible sections (in kernel) is still possible with raw_spinlock_t (same APIs like spinlock_t).
- Implementing priority inheritance for in-kernel spinlocks and semaphores. For more information on priority inversion and priority inheritance please consult Introduction to Priority Inversion.
- Converting interrupt handlers into preemptible kernel threads: The RT-Preempt patch treats soft interrupt handlers in kernel thread context, which is represented by a task_struct like a common user space process. However it is also possible to register an IRQ in kernel context.
- Converting the old Linux timer API into separate infrastructures for high resolution kernel timers plus one for timeouts, leading to user space POSIX timers with high resolution.
Who are the maintainers of CONFIG_PREEMPT_RT patch?
Steven Rostedt is the maintainer of the 3.x stable RT tree.
Which architectures does the CONFIG_PREEMPT_RT patch support?
There are systems representing the x86, x86_64, ARM, MIPS, and Power architectures using the CONFIG_PREEMPT_RT patch. However, in many ways this is the wrong question. Support for real-time is not just about the instruction set architecture, but also about supporting the high resolution timer provided by the CPU and/or CPU support chipset, the device drivers for the system being well behaved, etc. So just because an ARM system from company A may work quite well with the -rt patchset, it does not guarantee that another ARM system from Company B will work as well; it might have some longer latency problems or it might not work at all. It is true that overall design and architecture of the -rt patch tends to avoid the need for device-driver specific changes, but there are always software bugs as well as hardware design bugs.
Please refer to platforms tested and in use with CONFIG_PREEMT_RT section in this wiki for a list of platforms that members of the -rt community have used successfully. The list is not exhaustive, of course, but it should give a flavor of the sorts of packages that have had enjoyed success using the -rt patch.
How can I start using the CONFIG_PREEMPT_RT patch?
Please see the RT PREEMPT HOWTO for a detailed description. The short version is to get the latest -rt patch from http://www.kernel.org/pub/linux/kernel/projects/rt/, then get the related vanilla kernel from http://kernel.org, apply the patch to the kernel, configure, and compile the kernel.
What is the "3.x stable RT tree"? What is "OSADL Latest Stable"?
3.x stable RT tree
The stable RT tree process is modeled on the mainline Linux stable release process (see Documentation/stable_kernel_rules.txt in the kernel source tree). When the head of the PREEMPT_RT development moves to a new Linux kernel version (eg from 3.2.x to 3.4.x), the old Linux kernel version will be considered for maintenance as a stable RT tree. The intent is to maintain a stable RT tree while the underlying mainline Linux kernel tree is maintained as a stable mainline tree. This means that only patches approved during the PREEMPT_RT development process will go into a stable RT tree. The same applies to patches of the mainline kernel that are unrelated to RT. The only way they can enter the stable RT tree is through the mainline stable mechanism and then be forwarded to RT. The stable RT tree process is described by the README file at the root of each RT tree. For example, the README for 3.4.x is located at http://www.kernel.org/pub/linux/kernel/projects/rt/3.4/README.
OSADL "Latest Stable"
One of the services that the Open Source Automation Development Lab (OSADL) provides to the community is a test farm where a large variety of supported architectures and systems is continuously tested for stability, performance and real-time capabilities. Whenever a real-time kernel version is considered suitable for use in industrial systems and products, it is labeled "Latest Stable". This recommendation is based on a significant amount and duration of tests. One of the reasons this service is valuable is because the PREEMPT_RT development process does not differentiate between -rc versions (as the mainline Linux kernel does) and released versions. The criteria for an OSADL "Latest Stable" real-time kernel version along with examples of regression analysis are available via links at https://www.osadl.org/Latest-Stable.
What sort of real-time performance should I expect?
The typical answer given by all performance experts when asked question like this is, "it depends". Not only does it depend on the speed of the CPU and the architecture, but also on the device drivers and of the hardware. For example, if a device grabs the PCI bus for long periods during DMA activity, that can introduce significant latencies in the system. In addition some firmwares can stop the system for housekeeping activities via Service Management Interrupts (SMIs) on x86 and x86_64 architectures; SMIs can not be trapped by the OS, so latencies introduced by SMBIOS routines can only be addressed by working with the firmware designers of the motherboard. On 2006-2007 high-end AMD and Intel CPUs systems have been deployed with 10 to 30 µs scheduler and interrupt latencies; on other platforms, your mileage will definitely vary. The platforms tested and in use with CONFIG_PREEMPT_RT section of this wiki does have some examples of latency numbers found by various -rt users and developers.
What mailing lists are available to discuss the CONFIG_PREEMPT_RT patch?
Please see the mailing lists page in the rt wiki.
How to enable/disable realtime support?
Change realtime support run time or statically?
Statically, you need to compile the realtime patched kernel with CONFIG_PREEMPT_RT=y or CONFIG_PREEMPT_RT_FULL=y to support realtime in 2.6.x or 3.x kernels, respectively.
Required parameters for configuring realtime kernel?
Optional parameters for configuring realtime kernel, but they affect realtime performance?
Real-time Application Programming
List of realtime APIs?
No special API's available in the real-time patch. Real-time makes use of the standard Posix API's.
Also: If the Hires-timer is enabled in the kernel, every timer in userspace becomes a hires timer. Real-time only has impact on the kernel; Userspace does not notice the difference except for better real time behavior.
Is any API available which allows me to switch from Task A to Task B and/or Thread A to Thread B?
The thread with the highest RT-priority, and which is in the run-queue, usually runs until it gives up the CPU by some blocking call.
If you have to switch between 2 RT-threads of type SCHED_FIFO with same priority and both are runnable, you can switch to the other by means of sched_yield(), but if you need such constructions you are having a really bad RT design.
How to write realtime applications?
Which programming languages are suitable for writing realtime applications?
It depends on your latency requirements, but usually these are advisable:
- C is preferred;
- C++ is possible if some constraints are taken into account (see HOWTO: Build an RT-application)
What are important things to keep in mind while writing realtime applications?
Taking care of the following during the initial startup phase:
- Call mlockall() as soon as possible from main().
- Create all threads at startup time of the application, and touch each page of the entire stack of each thread. Never start threads dynamically during RT show time, this will ruin RT behavior.
- Never use system calls that are known to generate page faults, such as fopen(). (Opening of files does the mmap() system call, which generates a page-fault).
- If you use 'compile time global variables' and/or 'compile time global arrays', then use mlockall() to prevent page faults when accessing them.
more information: HOWTO: Build an RT-application
Do I need to recompile my applications to get realtime performance?
Recompile is not necessary. Userland does not notice any difference if it is running on RT or not, except for a much better RT-responsiveness, only changing the kernel is enough.
But, usually, locking bugs in userland applications become much more visible due the more dynamic and fine grained scheduling of the kernel. These are not new bugs, but bugs already existing in the applications, which only become visible on RT (Notice that these bugs are also likely to occur on SMP systems).
- A bad designed application on non-RT will never behave realtime on RT. See also HOWTO: Build an RT-application.
- Never protect shared/global data in userland processes with semaphores (yes, many people do that...), but use PTHREAD_PRIO_INHERIT mutexes instead. (PTHREAD_PRIO_INHERIT mutexes are only supported in Glibc 2.5 and higher. uClibc based file systems do not support this)
Do I need root privileges to start a realtime application?
No, you can configure your system so that also a non-root user can run a realtime application. A typical realtime application uses realtime priorities and it might want to lock its address space (mlockall). For security reasons, a normal user is usually not allowed to do such things. However, /etc /security/limits.conf lets you modify the default settings for a given user or group:
someuser - memlock 51200 someuser - rtprio 80
With these sample settings someuser is now allowed to use realtime priorities up to priority 80 and to lock a maximum address space of 50MB. ulimit -a can be used to retrieve the security limits. You have to logout and login for the new limits to take effect. The limits.conf manpage contains further information.
Note: if you create a daemon that runs as an unprivileged user, started via /etc/init.d/ scripts, it may not take limits.conf into account (this is the case for a default install of Debian Lenny). To set the realtime rights, you can use the ulimit command in your /etc/init.d script before starting the daemon.
Authors of this FAQ include:
- Jaswinder Singh
- Luotao Fu (l.fu AT pengutronix DOT de), Pengutronix e.K., Kernel Development Group
- Robert Schwebel (r.schwebel AT pengutronix DOT de), Pengutronix e.K., Kernel Development Group
- Remy Bohmer (linux AT bohmer DOT net)
- Theodore Ts'o (tytso AT mit DOT edu), IBM
- Frank Rowand (frank.rowand AT am DOT sony DOT com), Sony Network Entertainment