HOWTO: Build an RT-application
Remy Bohmer (Software Engineer Linux RT platform)
|Draft / Work in progress|
This document describes the steps to writing hard realtime linux programs while using the Realtime Preemption Patch. It also describes the pitfalls that destroy the realtime responsiveness. It focusses on x86, as this is currently the most mature architecture.
Hardware causes of ISR latency
A good Realtime behaviour of a system depends a lot on low latency interrupt handling. Taking a look at the X86 platform, it shows that this platform is not really designed for RT usage. Several mechanisms cause ISR latencies that can run into the 10's of microseconds. Knowing them will enable you to make the best design choices on this platform to enable you to get rid of the negative impact.
- DMA busmastering: Bus mastering events can cause long-latency CPU stalls of many microseconds. It can be generated by every device that uses DMA, such as SATA/PATA/SCSI devices and even network adapters. Also video cards that insert wait cycles on the bus in response to a CPU access can cause this kind of latency. Sometimes the behavior of such peripherals can be controlled from the driver, trading off throughput for lower latency. The negative impact of busmastering is independant from the chosen OS, so this is not a unique problem for Linux-RT, even other RTOS-es experience these type of latency!
- On-demand CPU scaling: creates long-latency events when the CPU is put in a low-power-consumption state after a period of inactivity. Such problems are usually quite easy to detect. (e.g. On Fedora the 'cpuspeed' tool should be disabled, as this tool loads the ondemand scaling_governor driver)
- VGA Console: During RT behavior of the system the VGA Console must be left untouched. Nothing may be written to that console, also printk's are not allowed. This VGA console can cause very large latencies, up to more than a millisecond. It is better to use a serial console and have no login shell on the VGA console. Also SSH or Telnet sessions can be used. The 'quiet' option on the kernel command line could also work if login in on the console is disabled.
Whenever you ran into a page-fault the kernel freezes the entire program (with all its threads in it), until the kernel has backed up the memory in RAM. (A pagefault is a trap that occurs when you start using a memory page for the first time, or when it is not mapped in physical memory (anymore) and that it has to be retrieved from the swap-space) A pagefault trap can therefor cause very high latencies, especially when the page has to be retrieved from the swap space on disk, latencies of a second is no exception.
Once a RT application must show its RT requirements, page faults must be prevented.
This can be done by taking care of the following:
- Boot the application with the mlockall call.
- Directly after any malloc() or new() allocation, touch each page in the allocated memory space.
- Create all threads at startup time of the application, and touch each page of the entire stack of each thread.
- Never use system calls that are known to generate pagefaults, such as fopen(). (Opening of files does the mmap() system call, which generates a page-fault)
- Do not use 'compile time arrays' without initialising them directly on boot.