Ubuntu Performance Tuning
- 1 Use The Latest Ubuntu Release and Kernel
- 2 Virtual Memory Tweaks
- 3 CPU Performance Governor
- 4 File System and Block Device Tweaks
- 5 Networking Sysctl Tweaks
- 6 Additional Performance Tuning Sources
Use The Latest Ubuntu Release and Kernel
Newer versions of Ubuntu run faster than older versions. So if you want the best performance you need to be running the newest software. Ubuntu 15.04 is just around the corner, and it'll probably be faster than Ubuntu 14.10 :) If possible, I suggest upgrading the kernel to the latest version, which can always be found here - https://www.kernel.org/
There have been some pretty major IO performance boosts in the past few kernel releases, in some cases you might see 25% more random read or write IOPS if you are using an SSD.
- Kernel 3.17 SCSI Multi-Queue - http://www.phoronix.com/scan.php?page=news_item&px=MTcyMjk
- Kernel 3.19 multi-queue block layer (blk-mq) - http://www.phoronix.com/scan.php?page=news_item&px=MTg2Mjg
If you're running Ubuntu with a kernel that's older than 3.19, and you have an SSD, you are probably only getting 50% - 75% of it's potential speed. I'm not joking, the performance gains are huge between Kernel 3.13 and 3.19 even for Ubuntu. CentOS 7 / RHEL just got to Kernel 3.10 which sucks slightly less than CentOS 6 2.6.something, which is just depressing when it comes to SSD IO performance.
You can tweak all the settings you want, but if you are looking for better disk IO the only thing that will make a significant difference is the kernel version you use.
Virtual Memory Tweaks
On most Linux based operating systems the sysctl configuration file can be found under /etc/ and I suggest using this file to make any changes instead of using the sysctl command. You can use sysctl -w $value to write changes, and then use sysctl -p to load the new settings but I always like to actually look at sysctl.conf before I change anything. I also like to note what the original value was set to and comment the line just so I can revert changes if needed.
The vm.swappiness value determines how aggressive Linux should be when it comes to swapping in active pages in Memory to Disk. The default value on most Linux distros such as Ubuntu is 60. Lowering this value to 10 will cause Linux to only swap out pages when it's close to utilizing all the available memory. This can help to reduce latency for most applications because more data should be kept in RAM instead of on disk. Be careful with adjusting this setting on a server that's already using a lot of memory / swap as a burst if IO activity could cause the system to hang. I suggest applying this change then rebooting the server or at least restarting the services to make sure the processes don't continue to use swap.
vim /etc/sysctl.conf ##Add this line or modify the existing line from 60 to 10 vm.swappiness = 10
To make sure the new settings is loaded run:
You can check what the value of vm.swappiness is set to by:
sysctl -a | grep vm.swappiness
- Redhat suggests a value of 10 -- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-tunables.html
- Brendan Gregg suggests a value of 0 -- http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html
The default value for vm.dirty_ratio on Ubuntu 14.10 is 20. The value set for the dirty ratio defines the maximum amount of memory that can be filled with dirty pages before they all must get flushed to the disk. With higher values, dirty pages will remain in memory longer, with lower values dirty pages gets flushed to disk more often. Higher values can cause applications that are writing data to disk to start writing data to dirty pages in memory which will later be flushed to disk.
If you are not worried about data loss / flushing data to disk all that often, then setting the dirty ratio to 50 or even 80 should provide better performance because data gets written to disk less frequently, however, this also means that if the server goes off line or hangs up for whatever reason you could potentially lose a lot of data since there will be more dirty pages in RAM.
If you have a lot of Memory and slow disks then you might not want to set this too high or else you may run into a scenario where the server needs to flush 100GB of dirty pages to disk (if the server has say, 256GB RAM and the dirty ratio is set to 50). This may not be ideal and could cause applications to hang until all the pages are on disk. If you have fast storage and small amounts of RAM this may not be as much of an issue.
TL;DR -- Percentage of total available memory that contains free and reclaimable pages at which a process that is generating disk writes will start writing out dirty data.
vim /etc/sysctl.conf ##Add this line## vm.dirty_ratio = 80
Apply the new setting after you have modified sysctl.conf
There are many different settings that can affect disk I/O so even if you set vm.dirty_ratio to a low value, other settings like vm.swappiness can still cause frequent disk writes. The bottom line is that you should measure your server's performance and application performance before you make any changes, then review the results a day later to decide if the change was good or bad. I use NewRelic to help monitor my application as well as my server stats.
vm.dirty_background_ratio is set to 10 by default, at least for Ubuntu 14.10. This value determines the percentage of memory that can contain dirty pages before the background kernel flusher threads start to write dirty pages to disk. If you have 100GB of RAM and you set this value to 10, then Linux will flush the dirty pages to disk once there is 10GB worth of dirty pages. If you have 1GB of RAM and you set this to 10 then it would take 100MB of dirty pages to begin the flush process.
If your priority is making sure data is written to disk as soon as possible then you want to set this to a low value, if you would rather have dirty pages hang out in memory for a while then you can raise this value.
TL;DR - The percentage of total available memory that contains free and reclaimable pages at which the background kernel flusher threads will start writing out dirty data.
vim /etc/sysctl.conf ##Add this line## vm.dirty_background_ratio = 5
Apply the new setting after you have modified sysctl.conf
This value determines the amount of centiseconds (1 centisecond is 1/100 of a second) that a page needs to be dirty for, once this age has been reached the dirty page is considered eligible to be flushed to disk by the kernel flusher threads. On Ubuntu 14.10 vm.dirty_expire_centisecs is set to 3000, or 30 seconds (3000 centiseconds / 100 = 30 seconds). If a dirty page exceeds this age, it will get flushed to disk the next time the flusher threads wake up. Larger values mean that data can be written to memory and remain there longer before the disk has to write it. This means that the longer this value is the higher the odds of data loss so be careful with raising this too high.
TL;DR -- Value is expressed in 100's of a second. Defines the age at which dirty pages are eligible to be written to disk by the kernel flusher threads.
vim /etc/sysctl.conf ##Add this line## vm.dirty_expire_centisecs = 12000
Apply the new setting after you have modified sysctl.conf
CPU Performance Governor
apt-get install cpufrequtils
If you have 4 CPUs, you can run a simple for loop to echo performance into all of them. The performance governor lets all the CPUs run at full speed instead of dialing down to low speeds to save power. If you only want the best performance and don't care about power usage, this is the way to go.
for each in 0 1 2 3; do echo performance > /sys/devices/system/cpu/cpu$each/cpufreq/scaling_governor; done
This will only apply until you reboot the PC, you can add this to /etc/rc.local so that the settings are applied on reboot.
File System and Block Device Tweaks
If you are using an SSD you may get better performance by using the noop scheduler instead of deadline. While deadline is much better than CFQ, noop can outperform both in some cases, especially if you have a PCI-E SSD. To modify the scheduler on the fly, simply echo noop into sda's scheduler file, if you have multiple SSDs you should make sure you set noop as the scheduler for each SSD (sdb, sdc, etc).
echo noop > /sys/block/sda/queue/scheduler
To make noop the permanent scheduler and persist after reboot you need to modify grub and add "elevator=noop" to the line below.
vim /etc/default/grub GRUB_CMDLINE_LINUX="find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US elevator=noop"
Run update-grub after you save the file, then reboot.
Additionally, you can just add a command to echo noop as the scheduler on reboot, you can add this command to /etc/rc.local
vim /etc/rc.local ##Add this line## echo noop > /sys/block/$DEVICE/queue/scheduler
Make sure that /etc/rc.local is executable, otherwise the changes will not get applied on reboot, a simple chmod +x /etc/rc.local should do the trick.
Disable add_random for Block Devices
To prevent the I/O on your SSD or HDD from contributing to the entropy pool you can echo 0 into the file below. Linux uses the entropy pool for things like generating SSH keys, SSL certificates or anything else crypto. Preventing your SSD from contributing to this pool probably isn't going to get yo kernel hacked so I do suggest disabling this as it will save you small amounts of IO.
echo 0 > /sys/block/sda/queue/add_random
If you want to make sure this remains disabled after reboots, you can add the command above to /etc/rc.local
vim /etc/rc.local ##Add this above "exit 0"## echo 0 > /sys/block/sda/queue/add_random exit 0
Save the file then make sure the file is executable
chmod +x /etc/rc.local
Make sure Ubuntu knows you are using an SSD and not a slow HDD
On newer versions of Ubuntu, this may not matter a whole lot, if at all, but for the sake of adding another section to this page I'm including it anyway. If this file outputs a 1 when you cat it, echo 0 right back in it's face.
echo 0 > /sys/block/sda/queue/rotational
Again, to make this a permanent change we will add it to /etc/rc.local and chmod +x that sucker so it can execute like a boss.
vim /etc/rc.local ##Add this above "exit 0"## echo 0 > /sys/block/sda/queue/add_random echo 0 > /sys/block/sda/queue/rotational exit 0
By default this value is set to 1, which means that once an I/O request has been completed by the block device, it will be sent back to the "group" of CPUs that the request came from. This can sometimes help to improve CPU performance due to caching on the CPU, which requires less cycles. If this value is set to 2, the block device will sent the completed request back to the actual CPU that requested it, not to the general group. If you have a beefy CPU and really want to utilize ALL THE CORES and spread the load around as much as possible then a value of 2 might provide better results.
echo 2 > /sys/block/sda/queue/rq_affinity
To make this a permanent change we will add it to /etc/rc.local
vim /etc/rc.local ##Add this above "exit 0"## echo 0 > /sys/block/sda/queue/add_random echo 0 > /sys/block/sda/queue/rotational echo 2 > /sys/block/sda/queue/rq_affinity exit 0
The value of nr_requests determines the amount of I/O requests that get buffered before the I/O scheduler sends / receives data to the block device, if you are using a RAID card / Block Device that can handle a larger queue than what the I/O scheduler is set to, raising the value of nr_requests may help to improve throughout and reduce server load when large amounts of I/O occur on the server. If you are using Deadline or CFQ as the scheduler, it is suggested that you should set the nr_request value to 2 times the value of queue depth.
echo 256 > /sys/block/sda/queue/nr_requests
For this setting to persist after a reboot you can add this command to /etc/rc.local
vim /etc/rc.local ##Add this line## echo 256 > /sys/block/sda/queue/nr_requests
The default value for read_ahead_kb on a CentOS 6 server is 128. This value determines the amount of extra data that will get read when the OS reads a file. If the next few requests are similar to the first request, the data may already be in RAM, especially if you set this to a large value. There are many opinions on what this value should be set to, and there is probably no right answer. I suggest testing out I/O performance with the default setting, then raise the value to 256, test again, then continue to raise this until you are happy with performance.
echo 256 > /sys/block/sda/queue/read_ahead_kb
- Oracle suggests a value of 64 -- https://docs.oracle.com/cd/E29584_01/webhelp/PerfTuning/src/tperf_tuning_the_read_ahead_kb_kernel_parameter.html
- Redhat suggests a value of 8192 -- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/ch06s04s03.html
Networking Sysctl Tweaks
Some of these values may already be set in sysctl.conf. If that is the case I suggest commenting out the original settings and pasting the new settings in. Be careful adding these tweaks!
net.core.somaxconn = 1000 net.core.netdev_max_backlog = 5000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_wmem = 4096 12582912 16777216 net.ipv4.tcp_rmem = 4096 12582912 16777216 net.ipv4.tcp_max_syn_backlog = 8096 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535
To apply the new settings
Additional Performance Tuning Sources
- Excellent blog that does a great job at explaining what some of the vm. settings do -- https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/
- Kernel Documentation on most of the tweaks covered in this wiki -- https://www.kernel.org/doc/Documentation/sysctl/vm.txt
- Much of the information in this wiki has come from watching Brendan Gregg speak on performance tuning at various conferences, you can find his blog here -- http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html