Profiling Specific Code Segments of Applications

Understanding the interaction between software and hardware has become increasingly essential for building high-performance applications.
The architecture of modern hardware systems has grown significantly in complexity, including deep memory hierarchies and advanced CPUs with features like out-of-order execution and sophisticated branch prediction mechanisms.

Linux Perf, Intel VTune, and AMD μProf are helpful tools for understanding how applications use system resources.
However, as these tools are typically designed as external applications, they profile the entire program, making it difficult to focus on specific code segments like particular functions.
This limitation is particularly challenging when analyzing micro-benchmarks, where the measured code may represent only a fraction of the overall runtime, or distinguishing between different phases of an application's execution.

Counting Hardware Events

At their core, these tools leverage Performance Monitoring Units (PMUs)–specialized components designed to track hardware events like cache misses and branch mispredictions.
Although these tools are far more powerful, this discussion will focus on the essentials of hardware event counting.

Scenario: Random Access Pattern

Consider a random access micro-benchmark designed to access a set of cache lines in a random sequence—a scenario that typically baffles the data prefetcher (see the full source code).
The benchmark employs two distinct arrays: one holding the data and another containing indices that establish the random access pattern.
After initializing these arrays, we execute the micro-benchmark by sequentially scanning through the indices array and accessing data from the data array. This method generally leads to approximately one cache miss per access within the contiguous data array.

Perf Stat

To observe the underlying hardware dynamics, we utilize the perf stat command, which quantifies low-level hardware events such as L1 data cache accesses and references during the execution of the micro-benchmark:

perf stat -e instructions,cycles,L1-dcache-loads,L1-dcache-load-misses -- ./random-access-bench --size 16777216

After running, perf stat displays the results on the command line, in combination with metrics such as instructions per cycle:

Performance counter stats for './random-access-bench --size 16777216':

    3,697,089,032      instructions            #    0.63  insn per cycle            
    5,879,736,227      cycles                                                                
    1,186,826,319      L1-dcache-loads                                                       
      103,262,784      L1-dcache-load-misses   #    8.70% of all L1-dcache accesses 

      1.202831289 seconds time elapsed

      0.799309000 seconds user
      0.403155000 seconds sys

Zooming into details, the results reveal 103,262,784 L1d misses for 16,777,216 items, which translates to 103,262,784 / 16,777,216 ≈ 6 misses per item.
This number significantly surpasses the anticipated single cache miss per item.
The source of this discrepancy lies in the comprehensive scope of the perf stat command, which records events throughout the entire runtime of the benchmark.
This includes the initialization stage of the benchmark where both the data and pattern arrays are allocated and filled.
Ideally, however, profiling should be confined to the specific segment of the code that interacts directly with the data array to achieve more accurate metrics.

One effective strategy for more control over profiling is to start and stop hardware counters at specific code segments using file descriptors.
This technique is well-documented in the perf stat man page.
Pramod Kumbhar provides a practical guide to implementing this technique on his blog, though some might find the approach somewhat cumbersome to implement.

Controlling Performance Counters from C++ Applications

Another strategy for achieving refined control over PMUs is to leverage the perf subsystem directly from C and C++ applications through the perf_event_open system call.
Given the complexity of this interface, various libraries have been developed to simplify interaction by embedding the perf_event_open system call into their framework.
Notable examples include PAPI, PerfEvent, and perf-cpp, each designed to offer a more accessible gateway to these advanced functionalities.

This article will specifically explore perf-cpp and demonstrate practical examples of how to activate and deactivate hardware performance counters for targeted code segments.
The perf::EventCounter class in perf-cpp allows users to define which events to measure and provides start() and stop() methods to manage the counters.
Below is a code snippet that sets up the EventCounter and focuses the measurement on the desired code segment:

#include <perfcpp/event_counter.h>

/// Initialize the hardware event counter
auto counters = perf::CounterDefinition{};
auto event_counter = perf::EventCounter{ counters };

/// Specify hardware events to count
event_counter.add({"instructions", "cycles", "cache-references", "cache-misses"});

/// Setup benchmark here (this will not be measured)
struct alignas(64U) cache_line { std::uint64_t value; };
auto data = std::vector<cache_line>{};
auto indices = std::vector<std::uint64_t>{};
/// Fill both vectors here...

auto sum = 0ULL;

/// Run the workload and count hardware events
event_counter.start();
for (const auto index : indices) {
    sum += data[index]; // <-- critical memory access
}
asm volatile("" : : "r,m"(value) : "memory"); // Ensure the compiler will not optimize sum away
event_counter.stop();

Once the EventCounter is initiated and the events of interest are added, we set up the benchmark by initializing the data and pattern arrays.
Enclosing the workload we wish to measure with start() and stop() calls enables precise monitoring of that particular code segment.
Upon stopping the counter, the EventCounter can be queried to obtain the measured events:

const auto result = event_counter.result();

/// Print the performance counters.
for (const auto [name, value] : result)
{
    std::cout << value << " " << name << " (" << value / 16777216 << " per access)" << std::endl;
}

The output reflects only the activity during the benchmark, effectively excluding the initial setup phase where data is allocated, and patterns are established:

102,284,667 instructions            (6.09664 per access)
992,091,716 cycles                  (59.1333 per access)
 34,227,532 L1-dcache-loads         (2.04012 per access)
 18,944,008 L1-dcache-load-misses   (1.12915 per access)

The results obtained are markedly more explicable than those we got from the perf stat command.
We observe two L1d cache references per access: one for the randomly accessed cache line and another for the index of the pattern array.
Additionally, there are approximately 1.3 cache misses—one for each data cache line and 0.125 for the access index, as eight indices fit into a single cache line of the pattern array.

Hardware-specific Events

While basic performance metrics such as instructions, cycles, and cache misses shed light on the interplay of hardware and software, modern CPUs offer a far broader spectrum of events to monitor.
However, it's important to note that many of these events are specific to the underlying hardware substrate.
The perf subsystem standardizes only a select group of events universally supported across different processors (see a detailed list).
To discover the full range of events available on specific CPUs, one can utilize the perf list command.
Additionally, Intel provides an extensive catalog of events for various architectures on their perfmon website.

In order to use hardware-specific counters within applications, the readable event names need to be translated into event codes.
To that end, Libpfm4 provides a valuable tool that translates event names (from perf list) into codes.

Let us consider the event CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD on the AMD Zen4 architecture as an example.
The event quantifies the CPU cycles stalled due to pending memory requests, which is particularly insightful for assessing the effects of cache misses on modern systems.
Intel offers analogous events, such as CYCLE_ACTIVITY.STALLS_MEM_ANY on the Cascade Lake architecture, and both EXE_ACTIVITY.BOUND_ON_LOADS and EXE_ACTIVITY.BOUND_ON_STORES on the Sapphire Rapids architecture.

After downloading and compiling Libpfm4, developers can fetch the code for a specific event as shown below:

./examples/check_events CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD

Requested Event: CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD
Actual    Event: amd64_fam19h_zen4::CYCLES_NO_RETIRE:NOT_COMPLETE_MISSING_LOAD:k=1:u=1:e=0:i=0:c=0:h=0:g=0
PMU            : AMD64 Fam19h Zen4
IDX            : 1077936192
Codes          : 0x53a2d6

Incorporating hardware-specific events into an application with perf-cpp would look something like this:

#include <perfcpp/event_counter.h>

/// Initialize the hardware event counter
auto counters = perf::CounterDefinition{};
counters.add("CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD", 0x53a2d6); // <-- Event code from Libpfm4 output
auto event_counter = perf::EventCounter{ counters };

/// Specify hardware events to count
event_counter.add({"cycles", "CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD"});

/// Setup and execute the benchmark as demonstrated above...

This precise tracking reveals that approximately 57 of 59 CPU cycles are spent waiting for memory loads to complete–a finding consistent with the inability of the hardware to predict the benchmark’s random access pattern, relying instead on inherent memory latency:

992,091,716 cycles                                      (59.1333 per access)
967,301,682 CYCLES_NO_RETIRE.NOT_COMPLETE_MISSING_LOAD  (57.6557 per access)

However, thanks to sophisticated out-of-order execution, the hardware effectively masks much of this latency, which on the specific machine to execute the benchmark is around 700 cycles.

Summary

Profiling tools play a crucial role in identifying bottlenecks and aiding developers in optimizing their code.
Yet, the broad granularity often means that key code segments tracked with perf stat can be obscured by extraneous data.
Libraries like PAPI, PerfEvent, and perf-cpp offer a solution by allowing direct control over hardware performance counters from within the application itself.
By leveraging the perf subsystem (more precisely the perf_event_open system call), these tools enable precise measurements of only the code segments that are truly relevant.