Tutorial: Linux kernel profiling with perf
3 days ago
- #Linux
- #Performance
- #Profiling
- Perf is a profiler tool for Linux 2.6+ based systems that abstracts CPU hardware differences and provides a simple command-line interface.
- Perf is based on the perf_events interface exported by recent versions of the Linux kernel.
- The perf tool offers a rich set of commands to collect and analyze performance and trace data, similar to git in structure.
- Common perf commands include annotate, record, report, stat, and top, each serving different profiling and analysis purposes.
- Perf supports a variety of events including hardware events, software events, and tracepoint events, which can be listed using 'perf list'.
- Hardware events are CPU-specific and vary by processor type and model, while software events include context-switches and minor-faults.
- Tracepoint events are implemented by the kernel ftrace infrastructure and are available in Linux kernels 2.6.3x and newer.
- Perf stat is used to count events during process execution, providing aggregated statistics at the end of the run.
- Perf record collects samples and saves them into a perf.data file, which can later be analyzed with perf report or annotate.
- Perf top provides real-time analysis similar to the top tool, showing functions where most time is spent.
- Perf bench includes microbenchmarks for different subsystems in the Linux kernel, useful for stress testing and performance regression testing.
- Troubleshooting tips include managing open file limits and understanding build-id for binary identification.
- Perf can profile sleep times to show where and how long a program is sleeping or waiting, using sched_stat and sched_switch events.