forked from Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
184 lines
6.3 KiB
184 lines
6.3 KiB
.. _psi: |
|
|
|
================================ |
|
PSI - Pressure Stall Information |
|
================================ |
|
|
|
:Date: April, 2018 |
|
:Author: Johannes Weiner <[email protected]> |
|
|
|
When CPU, memory or IO devices are contended, workloads experience |
|
latency spikes, throughput losses, and run the risk of OOM kills. |
|
|
|
Without an accurate measure of such contention, users are forced to |
|
either play it safe and under-utilize their hardware resources, or |
|
roll the dice and frequently suffer the disruptions resulting from |
|
excessive overcommit. |
|
|
|
The psi feature identifies and quantifies the disruptions caused by |
|
such resource crunches and the time impact it has on complex workloads |
|
or even entire systems. |
|
|
|
Having an accurate measure of productivity losses caused by resource |
|
scarcity aids users in sizing workloads to hardware--or provisioning |
|
hardware according to workload demand. |
|
|
|
As psi aggregates this information in realtime, systems can be managed |
|
dynamically using techniques such as load shedding, migrating jobs to |
|
other systems or data centers, or strategically pausing or killing low |
|
priority or restartable batch jobs. |
|
|
|
This allows maximizing hardware utilization without sacrificing |
|
workload health or risking major disruptions such as OOM kills. |
|
|
|
Pressure interface |
|
================== |
|
|
|
Pressure information for each resource is exported through the |
|
respective file in /proc/pressure/ -- cpu, memory, and io. |
|
|
|
The format for CPU is as such:: |
|
|
|
some avg10=0.00 avg60=0.00 avg300=0.00 total=0 |
|
|
|
and for memory and IO:: |
|
|
|
some avg10=0.00 avg60=0.00 avg300=0.00 total=0 |
|
full avg10=0.00 avg60=0.00 avg300=0.00 total=0 |
|
|
|
The "some" line indicates the share of time in which at least some |
|
tasks are stalled on a given resource. |
|
|
|
The "full" line indicates the share of time in which all non-idle |
|
tasks are stalled on a given resource simultaneously. In this state |
|
actual CPU cycles are going to waste, and a workload that spends |
|
extended time in this state is considered to be thrashing. This has |
|
severe impact on performance, and it's useful to distinguish this |
|
situation from a state where some tasks are stalled but the CPU is |
|
still doing productive work. As such, time spent in this subset of the |
|
stall state is tracked separately and exported in the "full" averages. |
|
|
|
The ratios (in %) are tracked as recent trends over ten, sixty, and |
|
three hundred second windows, which gives insight into short term events |
|
as well as medium and long term trends. The total absolute stall time |
|
(in us) is tracked and exported as well, to allow detection of latency |
|
spikes which wouldn't necessarily make a dent in the time averages, |
|
or to average trends over custom time frames. |
|
|
|
Monitoring for pressure thresholds |
|
================================== |
|
|
|
Users can register triggers and use poll() to be woken up when resource |
|
pressure exceeds certain thresholds. |
|
|
|
A trigger describes the maximum cumulative stall time over a specific |
|
time window, e.g. 100ms of total stall time within any 500ms window to |
|
generate a wakeup event. |
|
|
|
To register a trigger user has to open psi interface file under |
|
/proc/pressure/ representing the resource to be monitored and write the |
|
desired threshold and time window. The open file descriptor should be |
|
used to wait for trigger events using select(), poll() or epoll(). |
|
The following format is used:: |
|
|
|
<some|full> <stall amount in us> <time window in us> |
|
|
|
For example writing "some 150000 1000000" into /proc/pressure/memory |
|
would add 150ms threshold for partial memory stall measured within |
|
1sec time window. Writing "full 50000 1000000" into /proc/pressure/io |
|
would add 50ms threshold for full io stall measured within 1sec time window. |
|
|
|
Triggers can be set on more than one psi metric and more than one trigger |
|
for the same psi metric can be specified. However for each trigger a separate |
|
file descriptor is required to be able to poll it separately from others, |
|
therefore for each trigger a separate open() syscall should be made even |
|
when opening the same psi interface file. |
|
|
|
Monitors activate only when system enters stall state for the monitored |
|
psi metric and deactivates upon exit from the stall state. While system is |
|
in the stall state psi signal growth is monitored at a rate of 10 times per |
|
tracking window. |
|
|
|
The kernel accepts window sizes ranging from 500ms to 10s, therefore min |
|
monitoring update interval is 50ms and max is 1s. Min limit is set to |
|
prevent overly frequent polling. Max limit is chosen as a high enough number |
|
after which monitors are most likely not needed and psi averages can be used |
|
instead. |
|
|
|
When activated, psi monitor stays active for at least the duration of one |
|
tracking window to avoid repeated activations/deactivations when system is |
|
bouncing in and out of the stall state. |
|
|
|
Notifications to the userspace are rate-limited to one per tracking window. |
|
|
|
The trigger will de-register when the file descriptor used to define the |
|
trigger is closed. |
|
|
|
Userspace monitor usage example |
|
=============================== |
|
|
|
:: |
|
|
|
#include <errno.h> |
|
#include <fcntl.h> |
|
#include <stdio.h> |
|
#include <poll.h> |
|
#include <string.h> |
|
#include <unistd.h> |
|
|
|
/* |
|
* Monitor memory partial stall with 1s tracking window size |
|
* and 150ms threshold. |
|
*/ |
|
int main() { |
|
const char trig[] = "some 150000 1000000"; |
|
struct pollfd fds; |
|
int n; |
|
|
|
fds.fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK); |
|
if (fds.fd < 0) { |
|
printf("/proc/pressure/memory open error: %s\n", |
|
strerror(errno)); |
|
return 1; |
|
} |
|
fds.events = POLLPRI; |
|
|
|
if (write(fds.fd, trig, strlen(trig) + 1) < 0) { |
|
printf("/proc/pressure/memory write error: %s\n", |
|
strerror(errno)); |
|
return 1; |
|
} |
|
|
|
printf("waiting for events...\n"); |
|
while (1) { |
|
n = poll(&fds, 1, -1); |
|
if (n < 0) { |
|
printf("poll error: %s\n", strerror(errno)); |
|
return 1; |
|
} |
|
if (fds.revents & POLLERR) { |
|
printf("got POLLERR, event source is gone\n"); |
|
return 0; |
|
} |
|
if (fds.revents & POLLPRI) { |
|
printf("event triggered!\n"); |
|
} else { |
|
printf("unknown event received: 0x%x\n", fds.revents); |
|
return 1; |
|
} |
|
} |
|
|
|
return 0; |
|
} |
|
|
|
Cgroup2 interface |
|
================= |
|
|
|
In a system with a CONFIG_CGROUP=y kernel and the cgroup2 filesystem |
|
mounted, pressure stall information is also tracked for tasks grouped |
|
into cgroups. Each subdirectory in the cgroupfs mountpoint contains |
|
cpu.pressure, memory.pressure, and io.pressure files; the format is |
|
the same as the /proc/pressure/ files. |
|
|
|
Per-cgroup psi monitors can be specified and used the same way as |
|
system-wide ones.
|
|
|