Linux 7.3 SMP Improvement To Help Reduce Latency, Improve Real-Time Performance
Linux kernel 7.3 introduces SMP improvements from Bytedance engineers to reduce latency and enhance real-time performance.
Intelligence analysis by Gemini 2.5 Flash Lite
New SMP code in the Linux 7.3 kernel, developed by Bytedance engineers, aims to significantly reduce scheduling latency by allowing preemption during Inter-Processor Interrupt (IPI) completion waits.
Imagine a busy office where one person needs information from many others. If they have to wait for everyone to reply before doing anything else, the whole office slows down. This new computer code lets the person ask for information and then do other small tasks while waiting, making the office much faster.
Analysis
Bytedance Engineers
Engineers from Bytedance have spearheaded a significant rework of the Linux kernel's Symmetric Multiprocessing (SMP) code, specifically targeting the latency introduced by Inter-Processor Interrupts (IPIs). Their analysis revealed that the existing SMP behavior could lead to dramatic increases in scheduling latency, particularly on x86_64 systems where IPIs are used for critical operations like Translation Lookaside Buffer (TLB) flushes. This latency was found to be especially detrimental to performance-sensitive workloads such as those utilizing the Data Plane Development Kit (DPDK).
The core of their optimization lies in modifying how the kernel handles waiting for IPI completion. Previously, preemption was disabled throughout the entire IPI function execution, including the waiting period. This meant that if a remote CPU was slow to respond to an IPI, the entire system could experience significant latency spikes. The Bytedance solution introduces a per-task CPU mask to track which CPUs are being waited upon. This allows the kernel to re-enable preemption before the wait is fully completed, thereby mitigating the potential for large latency spikes.
IPI Completion Waiting
The impact of this change is substantial, with Bytedance reporting a nearly 90% reduction in P99 latency in their testing scenarios involving DPDK. Thomas Gleixner, a prominent figure in kernel development, highlighted the elegance of this approach in the pull request, noting that the moderate increase in memory cost for tracking the CPU mask is a worthwhile trade-off for the dramatic reduction in SMP function call induced latency. The improvements have moved the measured latency for high-priority tasks from approximately 17ms down to around 1.5ms, a critical improvement for real-time applications.
This rework addresses a long-standing issue where the blocking nature of IPI completion could severely impact system responsiveness. By allowing the system to remain preemptible during these waits, the kernel can better handle concurrent tasks and ensure smoother execution for demanding workloads. The changes are now awaiting integration into the mainline Linux kernel, with Linus Torvalds expected to review them during the Linux 7.3 merge window.
Linux 7.3 Merge Window
The integration of these SMP improvements into the Linux 7.3 kernel marks a significant step forward in optimizing real-time performance and reducing latency. As the merge window for Linux 7.3 opens, this contribution from Bytedance stands out as a key enhancement that will benefit a wide range of applications, particularly those requiring high throughput and low latency, such as networking and high-performance computing. The successful implementation of this optimization demonstrates the ongoing collaborative effort within the open-source community to continuously refine and improve the Linux kernel.
Barring any minor code adjustments requested by Linus Torvalds, these SMP enhancements are expected to be included in the upcoming Linux 7.3 release. This proactive approach to latency reduction underscores the kernel's adaptability and its capacity to evolve to meet the demands of modern computing environments. The Phoronix article notes that this is one of several improvements being prepared for Linux 7.3, indicating a busy development cycle for the next kernel iteration.
Key points
- Linux kernel 7.3 will include SMP improvements developed by Bytedance engineers.
- The changes aim to reduce scheduling latency by allowing preemption during IPI completion waits.
- Testing showed a nearly 90% reduction in P99 latency for DPDK workloads.
- This enhancement is expected to improve real-time performance for latency-sensitive applications.
This optimization could lead to significantly more responsive Linux systems for real-time applications, potentially enabling new use cases in areas like high-frequency trading, industrial automation, and advanced networking. Improved latency also means better overall system stability and performance for demanding workloads.
While the reported latency reduction is substantial, real-world performance may vary depending on specific hardware configurations and workload types. There's also a risk that the added complexity of per-task CPU masks could introduce subtle bugs or performance regressions in less common scenarios.