CPU shielding using /proc and /dev/cpuset

From RTwiki
(Difference between revisions)
Jump to: navigation, search
(Initial info)
 
(Process shielding)
 
(6 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
== Interrupt shielding ==
 
== Interrupt shielding ==
  
 +
=== Kernel Space ===
 
In order to shield CPUs from individual interrupts being serviced on them you have to make sure that the following kernel configuration parameter is set:
 
In order to shield CPUs from individual interrupts being serviced on them you have to make sure that the following kernel configuration parameter is set:
  
 
*CONFIG_IRQBALANCE
 
*CONFIG_IRQBALANCE
  
 +
 +
=== User Space ===
 
Then make sure that the interrupts are not automatically balanced by the irqbalance daemon. This daemon is started from the irqbalance init script. To disable once do:
 
Then make sure that the interrupts are not automatically balanced by the irqbalance daemon. This daemon is started from the irqbalance init script. To disable once do:
  
Line 15: Line 18:
 
After this you can change the CPU affinity mask of each interrupt by doing:
 
After this you can change the CPU affinity mask of each interrupt by doing:
  
{{cmdroot|echo ''hex_mask'' > /proc/irq/''irq_number''/smp_affinity}}
+
{{cmdroot|echo ''hex_mask'' > /proc/irq/''<irq_number>''/smp_affinity}}
  
 
To check that the affinity mask has been set you can check the contents of the smp_affinity file. {{NOTE|The mask is updated the next time an interrupt is serviced. So you may not see the change immediately.}}
 
To check that the affinity mask has been set you can check the contents of the smp_affinity file. {{NOTE|The mask is updated the next time an interrupt is serviced. So you may not see the change immediately.}}
Line 23: Line 26:
 
== Process shielding ==
 
== Process shielding ==
  
The kernel has an cpuset feature that allows you to create cpusets for real-time purposes. The kernel interface is proc filesystem based. It is described in {{path|/usr/src/kernel/Documentation/cpusets.txt}}.
+
The kernel has an cpuset feature that allows you to create cpusets for real-time purposes. The kernel interface is proc filesystem based. It is described in {{path|/usr/src/kernel/Documentation/cgroups/cpusets.txt}}.
 +
 
 +
Each cpuset is represented by a directory in the cgroup file system
 +
containing (on top of the standard cgroup files) the following
 +
files describing that cpuset:
 +
 
 +
<pre>
 +
  - cpus: list of CPUs in that cpuset
 +
  - mems: list of Memory Nodes in that cpuset
 +
  - memory_migrate flag: if set, move pages to cpusets nodes
 +
  - cpu_exclusive flag: is cpu placement exclusive?
 +
  - mem_exclusive flag: is memory placement exclusive?
 +
  - mem_hardwall flag:  is memory allocation hardwalled
 +
  - memory_pressure: measure of how much paging pressure in cpuset
 +
  - memory_spread_page flag: if set, spread page cache evenly on allowed nodes
 +
  - memory_spread_slab flag: if set, spread slab cache evenly on allowed nodes
 +
  - sched_load_balance flag: if set, load balance within CPUs on that cpuset
 +
  - sched_relax_domain_level: the searching range when migrating tasks
 +
</pre>
 +
 
 +
In addition, the root cpuset only has the following file:
 +
<pre>
 +
  - memory_pressure_enabled flag: compute memory_pressure?
 +
</pre>
 +
 
 +
Here is a quick example of how to use cpuset to reserve one cpu for your real-time process on a 4 cpu machine:
 +
 
 +
{{cmdroot|mkdir /dev/cpuset/rt0}}
 +
 
 +
{{cmdroot|echo 0 > /dev/cpuset/rt0/cpus}}
 +
 
 +
{{cmdroot|echo 0 > /dev/cpuset/rt0/mems}}
 +
 
 +
{{cmdroot|echo 1 > /dev/cpuset/rt0/cpu_exclusive}}
 +
 
 +
{{cmdroot|echo $RT_PROC_PID > /dev/cpuset/rt0/tasks}}
 +
 
 +
{{cmdroot|mkdir /dev/cpuset/system}}
 +
 
 +
{{cmdroot|echo 1-3 > /dev/cpuset/system/cpus}}
 +
 
 +
{{cmdroot|echo 0 > /dev/cpuset/system/mems}}
 +
 
 +
{{cmdroot|echo 1 > /dev/cpuset/system/cpu_exclusive}}
 +
 
 +
{{cmdroot|for pid in $(cat /dev/cpuset/tasks); do /bin/echo $pid > /dev/cpuset/system/tasks; done}}

Latest revision as of 20:25, 22 June 2010

Contents

[edit] Interrupt shielding

[edit] Kernel Space

In order to shield CPUs from individual interrupts being serviced on them you have to make sure that the following kernel configuration parameter is set:

  • CONFIG_IRQBALANCE


[edit] User Space

Then make sure that the interrupts are not automatically balanced by the irqbalance daemon. This daemon is started from the irqbalance init script. To disable once do:

# /etc/init.d/irqbalance stop

To disable after next reboot do:

# chkconfig irqbalance off

After this you can change the CPU affinity mask of each interrupt by doing:

# echo hex_mask > /proc/irq/<irq_number>/smp_affinity

To check that the affinity mask has been set you can check the contents of the smp_affinity file.
NOTE!
The mask is updated the next time an interrupt is serviced. So you may not see the change immediately.

More information can be found in /usr/src/kernel/Documentation/IRQ-affinity.txt.

[edit] Process shielding

The kernel has an cpuset feature that allows you to create cpusets for real-time purposes. The kernel interface is proc filesystem based. It is described in /usr/src/kernel/Documentation/cgroups/cpusets.txt.

Each cpuset is represented by a directory in the cgroup file system containing (on top of the standard cgroup files) the following files describing that cpuset:

 
  - cpus: list of CPUs in that cpuset
  - mems: list of Memory Nodes in that cpuset
  - memory_migrate flag: if set, move pages to cpusets nodes
  - cpu_exclusive flag: is cpu placement exclusive?
  - mem_exclusive flag: is memory placement exclusive?
  - mem_hardwall flag:  is memory allocation hardwalled
  - memory_pressure: measure of how much paging pressure in cpuset
  - memory_spread_page flag: if set, spread page cache evenly on allowed nodes
  - memory_spread_slab flag: if set, spread slab cache evenly on allowed nodes
  - sched_load_balance flag: if set, load balance within CPUs on that cpuset
  - sched_relax_domain_level: the searching range when migrating tasks

In addition, the root cpuset only has the following file:

  - memory_pressure_enabled flag: compute memory_pressure?

Here is a quick example of how to use cpuset to reserve one cpu for your real-time process on a 4 cpu machine:

# mkdir /dev/cpuset/rt0

# echo 0 > /dev/cpuset/rt0/cpus

# echo 0 > /dev/cpuset/rt0/mems

# echo 1 > /dev/cpuset/rt0/cpu_exclusive

# echo $RT_PROC_PID > /dev/cpuset/rt0/tasks

# mkdir /dev/cpuset/system

# echo 1-3 > /dev/cpuset/system/cpus

# echo 0 > /dev/cpuset/system/mems

# echo 1 > /dev/cpuset/system/cpu_exclusive

# for pid in $(cat /dev/cpuset/tasks); do /bin/echo $pid > /dev/cpuset/system/tasks; done

Personal tools