Following are some of the tips to improve overall performance of Linux application. It also includes tips for scalability of multi-threaded Linux application.

Reduce number of system calls

Any system call in user space application requires context switching to kernel mode and back to user mode. So the number system calls made in your most frequently executed code may adversely affect your application’s performance.

Use Futex

Unlike Mutex, Futex library attempts to avoid system calls whenever possible. This will improve the performance of the application.

Use Compare and Swap (CAS) and Lockless data structures

This is tricky and getting it right is difficult. However this will greatly improve performance of multi-threaded applications.

Avoid Disk access or Optimize disk access

Generally disk access will slow down the application very badly. If possible, completely avoid disk access in most frequently executed code path. Otherwise optimize the calls for disk write and read access in blocks instead of reading or writing small amounts of data. Some kind of buffering in the application would help in reducing read and write system calls

Set CPU affinity and priority of the application

If the system has multiple CPU cores, you can dedicate certain threads to specific CPU cores using CPU affinity setting. You can also isolate a CPU and dedicate it for a specific thread. However this decision should be made carefully based on the load pattern of the application threads.

Use shared memory for bulk amount of data transfer to other applications

Exchanging bulk data with other applications using standard IPC like pipe, FIFO, socket layer would require the data to be copied once from user space to kernel space and kernel space to the other application’s user space. This requires multiple copies and context switches. Using shared memory with a semaphore would be a better solution.

Use epoll instead of poll or select

Traditionally applications use poll or select system calls to wait on multiple file descriptors. However Linux provides a different mechanism epoll that would reduce system load.

Use Likely and Unlikely statements

The likely and unlikely conditions can help compiler optimize the code generation. This helps in keeping the likely code path together and improves the probability of hitting CPU cache.