Frequently Asked Questions
This page contains a list of frequently asked questions about the real-time support in the Linux Kernel.
What is real-time?
Real-time applications have operational deadlines between some triggering event and the application's response to that event. To meet these operational deadlines, programmers use real-time operating systems (RTOS) on which make maximum on which the maximum response time can be calculated or measured realiable for the given application and environment.
A typical RTOS use priorities. The highest priority task wanting the CPU always get the CPU within a fixed amount of time after the event waking the task has taken place. On such an RTOS the latency of a task only depends on the tasks running at equal or higher priorities, all other tasks can be ignored. On a normal OS (such as normal Linux) the latencies depends on everything running on the system, which of course makes it much harder to be convinced that the deadlines will be meet every time on just a reasonably complicated system. This is because preemption can be switched off for a unknown amount of time. The high priority task wanting to run can thus be delayed for an unknown amount of time by low priority tasks running with preemption switched off.
Where can I read more about real-time?
Have a look to Publications page of this wiki.
What are real-time capabilities of the stock 2.6 linux kernel?
Traditionally, the Linux kernel will only allow one process to preempt another only under certain circumstances:
- When the CPU is running user-mode code
- When kernel code returns from a system call or an interrupt back to user space
- When kernel code code blocks on a mutex, or explicitly yields control to another process
If kernel code is executing when some event takes place that requires a high priority thread to start executing, the high priority thread can not preempt the running kernel code, until the kernel code explicitly yields control. In the worst case, the latency could potentially be hundreds milliseconds or more.
The Linux 2.6 configuration option CONFIG_PREEMPT_VOLUNTARY introduces checks to the most common causes of long latencies, so that the kernel can voluntarily yield control to a higher priority task waiting to execute. This can be helpful, but while it reduces the occurences of long latencies (hundreds of milliseconds to potentially seconds or more), it does not eliminate them. However unlike CONFIG_PREEMPT (discussed below), CONFIG_PREEMPT_VOLUNTARY has a much lower impact on the overall throughput of the system. (As always, there is a classical tradeoff between throughput --- the overall efficiency of the system --- and latency. With the faster CPU's of modern-day systems, it often makes sense to trade off throughput for lower latencies, but server class systems that do not need minimum latency guarantees may very well chose to use either CONFIG_PREEMPT_VOLUNTARY, or to stick with the traditional non-preemptible kernel design.)
The 2.6 Linux kernel has an additional configuration option, CONFIG_PREEMPT, which causes all kernel code outside of spinlock-protected regions and interrupt handlers to be eligible for non-voluntary preemption by higher priority kernel threads. With this option, worst case latency drops to (around) single digit milliseconds, although some device drivers can have interrupt handlers that will introduce latency much worse than that. If a real-time Linux application requires latencies smaller than single-digit milliseconds, use of the CONFIG_PREEMPT_RT patch is highly recommended.
How does the CONFIG_PREEMPT_RT patch work?
The RT-Preempt patch converts Linux into a fully preemptible kernel. The magic is done with:
- Making in-kernel locking-primitives (using spinlocks) preemptible though reimplementation with rtmutexes:
- Critical sections protected by i.e. spinlock_t and rwlock_t are now preemptible. The creation of non-preemptible sections (in kernel) is still possible with raw_spinlock_t (same APIs like spinlock_t)
- Implementing priority inheritance for in-kernel spinlocks and semaphores. For more information on priority inversion and priority inheritance please consult Introduction to Priority Inversion
- Converting interrupt handlers into preemptible kernel threads: The RT-Preempt patch treats soft interrupt handlers in kernel thread context, which is represented by a task_struct like a common user space process. However it is also possible to register an IRQ in kernel context.
- Converting the old Linux timer API into separate infrastructures for high resolution kernel timers plus one for timeouts, leading to user space POSIX timers with high resolution.
How you define CONFIG_PREEMPT_RT patch in one word?
Who are the maintainers of CONFIG_PREEMPT_RT patch?
Which architectures does the CONFIG_PREEMPT_RT patch support?
There are systems representing the x86, x86_64, ARM, MIPS, and Power architectures using the CONFIG_PREEMPT_RT patch. However, in many ways this is the wrong question. Support for real-time is not just about the instruction set architecture, but also about supporting the high resolution timer provided by the CPU and/or CPU support chipset, the device drivers for the system being well behaved, etc. So just because an ARM system from company A may work quite well with the -rt patchset, it does not guarantee that another ARM system from Company B will work as well; it might have some longer latency problems or it might not work at all. It is true that overall design and architecture of the -rt patch tends to avoid the need for device-driver specific changes, but there are always software bugs as well as hardware design bugs.
Please refer to platforms tested and in use with CONFIG_PREEMT_RT section in this wiki for a list of platforms that members of the -rt community have used successfully. The list is not exhaustive, of course, but it should give a flavor of the sorts of packages that have had enjoyed success using the -rt patch.
How can I start using the CONFIG_PREEMPT_RT patch?
Please see the RT PREEMPT HOWTO for a detailed description. The short version is to get the latest -rt patch from http://www.kernel.org/pub/linux/kernel/projects/rt/, then get the related vanilla kernel from http://kernel.org, apply the patch to the kernel, configure, and compile the kernel.
What sort of real-time performance should I expect?
The typical answer given by all performance experts when asked question like this is, "it depends". Not only does it depend on the speed of the CPU and the architecture, but also on the device drivers and of the hardware. For example, if a device grabs the PCI bus for long periods during DMA activity, that can introduce significant latencies in the system. In addition some firmwares can stop the system for housekeeping activities via Service Management Interrupts (SMI's) on x86 and x86_64 architectures; SMI's can not be trapped by the OS, so latencies introduced by SMBIOS routines can only be addressed by working with the firmware designers of the motherboard. On 2006-2007 high-end AMD and Intell CPU's systems have been deployed with 10-30us scheduler and interrupt latencies; on other platforms, your mileage will definitely vary. The platforms tested and in use with CONFIG_PREEMT_RT section of this wiki does have some examples of latency numbers found by various -rt users and developers.
What mailing lists are available to discuss the CONFIG_PREEMPT_RT patch?
Please see the mailing lists page in the rt wiki.
How to enable/disable realtime support?
Enable : CONFIG_PREEMPT_RT=y
Disable : CONFIG_PREEMPT_RT=n
Change realtime support run time or statically?
Statically, you need to compile kernel with realtime patch with CONFIG_PREEMPT_RT=y to support realtime
Required parameters for configuring realtime kernel?
Optional parameters for configuring realtime kernel, but they effect realtime performance?
Real-time Application Programming
List of realtime APIs?
No special API's available in the real-time patch. Real-time makes use of the standard Posix API's.
Also: If the Hires-timer is enabled in the kernel, every timer in userspace becomes a hires timer. Real-time only has impact on the kernel; Userspace does not notice the difference except for better real time behavior.
Is any API available which allows me to switch from Task A to Task B and/or Thread A to Thread B?
The thread with the highest RT-priority, and which is in the run-queue, usually runs until it gives up the CPU by some blocking call.
If you have to switch between 2 RT-threads of type SCHED_FIFO with same priority and both are runnable, you can switch to the other by means of sched_yield(), but if you need such constructions you are having a really bad RT design.
How to write realtime applications?
Which programming languages are suitable for writing realtime applications?
It depends on your latency requirements, but usually these are advisable:
- C is preferred;
- C++ is possible if some constraints are taken into account (see HOWTO: Build an RT-application)
What are important things to keep in mind while writing realtime applications?
Taking care of the following during the initial startup phase:
- Call mlockall() as soon as possible from main().
- Create all threads at startup time of the application, and touch each page of the entire stack of each thread. Never start threads dynamically during RT show time, this will ruin RT behavior.
- Never use system calls that are known to generate page faults, such as fopen(). (Opening of files does the mmap() system call, which generates a page-fault).
- If you use 'compile time global variables' and/or 'compile time global arrays', then use mlockall() to prevent page faults when accessing them.
more information: HOWTO: Build an RT-application
Do I need to recompile my applications to get realtime performance?
Recompile is not necessary. Userland does not notice any difference if it is running on RT or not, except for a much better RT-responsiveness, only changing the kernel is enough.
But, usually, locking bugs in userland applications become much more visible due the more dynamic and fine grained scheduling of the kernel. These are not new bugs, but bugs already existing in the applications, which only become visible on RT (Notice that these bugs are also likely to occur on SMP systems).
- A bad designed application on non-RT will never behave realtime on RT. See also HOWTO: Build an RT-application.
- Never protect shared/global data in userland processes with semaphores (yes, many people do that...), but use PTHREAD_PRIO_INHERIT mutexes instead. (PTHREAD_PRIO_INHERIT mutexes are only supported in Glibc 2.5 and higher. uClibc based file systems do not support this)
Authors of this FAQ include: