Linux 7.4 Workqueue Changes Bring A "10 To 100 Times Faster" Optimization
Linux 7.4 introduces a significant workqueue optimization, making flushing operations 10 to 100 times faster on large, high core count systems by only checking active queues.
Intelligence analysis by Gemini 2.5 Flash
A major performance improvement has been merged into Linux 7.4, specifically targeting workqueues, which are crucial for asynchronous kernel tasks. This optimization dramatically speeds up workqueue flushing, particularly benefiting large server environments with many CPUs, by streamlining the checking process.
Imagine a huge library with thousands of empty desks. When the librarian (the computer's brain) needs to check if any books were left on desks, they used to check *every single desk*, even the empty ones, which took ages. Now, the librarian only checks the desks where someone actually sat down recently. This makes checking for forgotten books super fast, especially in really, really big libraries!
Analysis
The upcoming Linux 7.4 kernel is set to receive a notable performance boost through an optimization to its workqueue mechanism. This change, described as yielding a "10 to 100 times faster" improvement, is particularly impactful for large-scale systems, such as those powered by high core count Intel Xeon and AMD EPYC processors. Workqueues are fundamental kernel components responsible for deferring tasks asynchronously, ensuring the kernel remains responsive while handling background operations.
Tejun Heo
Longtime Linux developer Tejun Heo is credited with authoring this significant improvement. His work focuses on refining how workqueues are flushed, a process that previously involved checking every possible CPU queue regardless of activity. Heo's optimization intelligently modifies this behavior, ensuring that the system only visits queues that have been active since the last flush. This targeted approach drastically reduces overhead, especially in scenarios where many CPU queues might be idle.
The commit message from Heo highlights the impressive gains, detailing how an idle per-CPU workqueue on a 192-CPU machine can jump from 40,000 to 4.2 million flushes per second. Similarly, an idle unbound workqueue sees an increase from 450,000 to 4.3 million flushes per second. These figures underscore the magnitude of the performance enhancement, demonstrating a substantial reduction in the time and resources required for workqueue management.
Yao Kai
The impetus for this optimization originated from a report by Huawei engineer Yao Kai in early September. Kai's findings brought attention to the significant overhead associated with unbounded workqueues, particularly in demanding use cases. His report specifically cited the XFS CIL workqueue, which, on a 128-CPU machine, was spending up to 64 microseconds per flush in its walk operation. This detailed feedback provided the crucial real-world context and data necessary to identify and address the performance bottleneck.
Kai's observations were instrumental in guiding the development of the new flushing mechanism. By pinpointing the specific areas of inefficiency, the Linux kernel community was able to focus on a solution that directly tackled the reported performance issues. The collaboration between engineers like Kai, who identify problems in production environments, and core kernel developers like Heo, who implement solutions, is a hallmark of the open-source development model, leading to continuous and impactful improvements.
XFS CIL workqueue
One of the most direct beneficiaries of this workqueue optimization is the XFS file system, specifically its CoW (Copy-on-Write) Intent Log (CIL) workqueue. The article notes that XFS hits this workqueue on every log force, meaning that the previous inefficiency was a constant drain on performance for systems utilizing XFS, particularly those with high I/O demands. The reported 64-microsecond overhead per flush on a 128-CPU machine for the XFS CIL workqueue illustrates the practical impact of the prior implementation.
With the new changes, the XFS CIL workqueue will experience significantly faster flushing times, directly contributing to improved file system performance and overall system responsiveness. This is particularly critical for enterprise storage solutions and databases that rely heavily on XFS for its robustness and scalability. While the optimization primarily targets idle flushes, the overall reduction in overhead for frequently used components like the XFS CIL workqueue represents a tangible enhancement for a wide array of Linux-based systems.
Key points
- Linux 7.4 includes a workqueue optimization making flushing operations 10 to 100 times faster.
- The improvement primarily benefits large systems with high core counts, such as Intel Xeon and AMD EPYC servers.
- The change involves visiting only workqueues used since the last flush, rather than checking every possible CPU queue.
- Longtime Linux developer Tejun Heo authored the optimization, stemming from a report by Huawei engineer Yao Kai.
- The XFS CIL workqueue, which flushes on every log force, is a key beneficiary of this speedup.
This optimization promises substantial performance gains for large server infrastructures, leading to more efficient resource utilization and faster processing for demanding workloads. It reinforces Linux's position as a high-performance operating system for enterprise and data center environments.
While largely positive, the article notes that dense flushes with every workqueue active remain unchanged on per-CPU workqueues and can be 15-20% slower on unbound ones, indicating some specific scenarios might not see benefits or could experience minor slowdowns.
