Benchmarking

From wiki.mikejung.biz
Jump to: navigation, search

Contents

How to break in SSDs before benchmarking

The dd utility is really not useful as a benchmarking tool, but it is an excellent tool to use to break in SSDs before you run a real benchmark like FIO or Sysbench :) I like to run dd at least 5 times, each time writing to the SSD until it's full. Notice I am not running dd on a partition or writing a file on a file system, I'm simply running dd against the block device. In this case I'm breaking in a new NVMe SSD, if you don't have an NVMe SSD, then replace "/dev/nvme0n1" with "/dev/sda" or whatever your drive is named.

note this will nuke the data on the devices so DO NOT RUN THIS on SSDs that have data on them, only do this when initially testing an SSD

for each in {1..5}; do dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct ; done & 

In this case I'm breaking in some new NVMe SSDs, and I have two of them so I'm launching two for loops into the background. If you don't have an NVMe SSD, then replace "/dev/nvme0n1" with "/dev/sda" or whatever your drive is named. note this will nuke the data on the devices so DO NOT RUN THIS on SSDs that have data on them, only do this when initially testing an SSD

for each in {1..5}; do dd if=/dev/zero of=/dev/nvme0n1 bs=1M oflag=direct ; done & for each in {1..5}; do dd if=/dev/zero of=/dev/nvme1n1 bs=1M oflag=direct ; done &

Linux Benchmarking Tools

Tools mentioned in this section should work for dedicated servers, home computers, cloud or virtual servers.

Linux has many different utilities which will measure performance of just about anything you want. The most common I/O utilities are Sysbench, FIO, and IOzone. There are also many benchmarking frameworks such as Phoronix Test Suite and Google's recently announced "PerfKitBenchmarker". Many of the frameworks use FIO to test I/O, but I prefer to run the utilities on their own, often times sequential I/O tests are run, and those results are worthless unless all the server does is stream large media files. Webservers and Database servers are usually dealing with random reads and writes, so if you do run I/O tests with something like Phoronix, take the results with a grain of salt until you run your own tests to verify that random workloads are being properly tested.

This page will cover the following utilities

  • FIO
  • Sysbench
  • Phoronix Test Suite (moved to it's own page)
  • IOzone
  • Ioping
  • UnixBench
  • Google's Perfkit Benchmarker

Liquidweb high performance banner.gif

Fio

Fio is one of my favorite tools to benchmark SSD or HDD I/O because it's so flexible. You can run sequential read and write tests, as well as random read and write tests using various block sizes like 4KB, 128KB, 512KB, or 1MB+. Flexibility is very important if you want to run tests that replicate the "real world" environment or workload that the drive(s) will be dealing with. If a server is not going to be streaming media files, then testing out sequential read or write performance is a waste of your time, you should be testing out random read or writes, depending on what the server will be doing.

Fio is really easy to install on Ubuntu / Debian. On CentOS 7 / Fedora you need to add an additional repo since the stock repos don't usually include benchmarking utilities. I strongly suggest using both FIO and Sysbench to measure I/O performance because every utility will provide a slightly different view of performance, and the results may not be the same between a FIO random write test and a Sysbench random write test, if you run both tests and the results are similar then you can consider the results to be accurate, however if the results are totally different then you might want to find out why this is the case before you can consider the test results to be accurate.

I suggest running FIO tests using various amounts of jobs and queue depths. Running a random write test with 1 job and 1 QD will produce significantly different results than a test with 4 jobs and 64 QD. You can think of a job as a process that is accessing a file / set of files. The more jobs you have, the higher the performance will be, up to a point. Usually I like to use at least 4 jobs and a QD of at least 16, but again, if your application / environment is not that demanding then you may not want to run demanding I/O tests (is your application really going to be running 16 processes (jobs) each with a queue depth of 64 all the time?)

Fio Installation

Install Fio on Ubuntu 14.10

Installing Fio on Ubuntu 14.10 is really easy, you can simply apt-get install fio.

apt-get install fio

Install Fio on CentOS 6.6

If you want to run FIO on CentOS you can either use a repo like EPEL, or just grab the actual RPM and dependency packages and install FIO that way.

yum install libibverbs.x86_64
wget http://pkgs.repoforge.org/fio/fio-2.1.10-1.el6.rf.x86_64.rpm
rpm -iv fio-2.1.10-1.el6.rf.x86_64.rpm

Install Fio on CentOS 7

Installing FIO on CentOS 7 is also pretty simple. You can grab the fio 2.1.10 rpm below, then you simply use RPM to install it.

wget http://pkgs.repoforge.org/fio/fio-2.1.10-1.el7.rf.x86_64.rpm
rpm -iv fio-2.1.10-1.el7.rf.x86_64.rpm

Fio Test Options and Examples

blocksize

This options determines the block size for the I/O units used during the test. The default value for blocksize is 4k (4KB). This option can be set for both read and write tests. For random workloads, the default value of 4k is typically used, for sequential workloads, a value of 1M (MB) is usually used. Change this value to whatever your production environment uses so that you are replicating the real world scenario as much as possible. If your servers deal with 4K block sizes 99% of the time, then why test out performance using 1MB blocksize?

--blocksize=4k (default)

ioengine

By default, Fio will run tests using the sync io engine, but if you want to change the engine used, you can. There are many different options you could change this value to, but on Linux the most common options are sync or libaio if the kernel supports it.

--ioengine=sync (default)

iodepth

The iodepth option defines the amount of IO units that will continue to hammer a file with requests during the test. If you are using the default sync ioengine, then increasing the iodepth beyond the default value of 1 will not have an effect. Even if you change the ioengine to use something like libaio the OS might restrict the maximum iodepth and ignore the specified value. Because of this I recommend starting off testing with an iodepth of 1 and raise this to something like 16 and test again, if you do not see any performance differences then you may not want to even specify this option, especially if you have set directio to a value of 1. Again, every server / OS is different so test out a few combinations of options before you start recording results.

--iodepth=1 (default)

direct

This option tells Fio whether or not it should use direct IO, or buffered IO. The default value is "0" which means that Fio will use use buffered I/O for the test. If you set this value to 1 then Fio will avoid using buffered IO, usually this is similar to O_DIRECT. Using buffered IO will almost always provide better performance than non-buffered IO, especially for read tests, or if you are testing out a server with a very large amount of RAM, using non-buffered IO helps to avoid inflated results. If you every run a test and Fio tells you that an SSD performed 600,000 IOPs, odds are it's not, and Fio is reading out of RAM, which will obviously be faster.

--direct=0 (default)

fsync

The fsync option tells Fio how often it should use fsync to flush "dirty data" to disk. By default this value is set to 0 which means "don't sync". Many applications perform like this and leave it up to Linux to figure out when to flush data from memory to disk. If your application or server always flushes every write to disk (meta-data and data) then you should include this option and set it to a 1. If your application does not flush data to disk after each write, or you aren't too worried about potential data loss, then leave this value alone. Setting fsync to 1 will completely avoid the buffering of writes, so if you want to see the "worst case" performance IO performance for a block device, set fsync to 1 and run a random write test. Results will be much lower than without fsync, but since every single write operation has to get flushed to disk, the disk will be stressed.

--fsync=0 (default)

Fio Random Write and Random Read Command Line Examples

Random Write

The command below will run a Fio random write test. This test writes a total of 4GB files (8 jobs x 512MB each = 4GB total size being accessed) running 8 processes at once. If you are testing a server with say 8GB of RAM, you would want to adjust the file size to be double the RAM to avoid excessive buffering, if there is 8GB of RAM, then set size to 2G, or leave the file size alone and double the amount of jobs. If you don't have that much space to test you can change "--direct=0" to "--direct=1" which will help to avoid caching the writes / buffering them, this might happen in the real world, but if you just want to isolate the block device performance without Linux caching / buffering the data, either use direct=1 or use a data set that is 2 times larger than the amount of RAM to make it impossible to cache / buffer all of the writes. By using the Group reporting option FIO will combine each job's stats into one aggregate result so the output is much easier to read. I'm using a QD of 1 for this example and am using buffered IO, both of these settings are the FIO defaults, so this is the most basic random write test you can run.

fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --numjobs=8 --runtime=240 --group_reporting

For example, I would run the command below on a 1GB SSD LiquidWeb StormVPS to get a quick idea of it's random write performance. Here I am only running the test for 60 seconds, just to gather the results quickly for this example.

fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=256M --numjobs=8 --runtime=60 --group_reporting

Once I run the command above, the output will look like this. The test begins as soon as the last of the 8 files is laid out by FIO.

randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.0.13
Starting 8 processes
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
randwrite: Laying out IO file(s) (1 file(s) / 256MB)
Jobs: 5 (f=5): [w_w_www_] [86.7% done] [0K/286.8M/0K /s] [0 /73.5K/0  iops] [eta 00m:02s]

Once the test is complete, FIO will output the test results, which looks like the output below. There are a ton of stats here and at first it's a little overwhelming. When I run a FIO test I always record the full results and store them somewhere. Even if I only care about 1 or two stats like iops or 95% clat, I still store all the results somewhere in case I need to grab another stat later on. Usually I'll store the full results in a Google Sheet, in a note in the same cell as the iops. If you don't store all of this data and only record the iops, what happens if you need to recall the date you ran the test? I placed a * next to the lines that I usually pay attention to.

randwrite: (groupid=0, jobs=8): err= 0: pid=22394: Sun Mar  1 13:13:18 2015
*  write: io=2048.0MB, bw=169426KB/s, iops=42356 , runt= 12378msec
    slat (usec): min=1 , max=771608 , avg=177.53, stdev=5435.09
    clat (usec): min=0 , max=12601 , avg= 1.46, stdev=65.22
     lat (usec): min=2 , max=771614 , avg=180.20, stdev=5436.13
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    1], 60.00th=[    1],
*     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   70], 99.95th=[  217],
     | 99.99th=[ 1160]
    bw (KB/s)  : min=  154, max=80272, per=12.29%, avg=20817.29, stdev=16052.89
    lat (usec) : 2=95.46%, 4=4.06%, 10=0.16%, 20=0.13%, 50=0.07%
    lat (usec) : 100=0.04%, 250=0.04%, 500=0.03%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.94%, sys=15.12%, ctx=153139, majf=0, minf=201
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=524288/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=2048.0MB, aggrb=169425KB/s, minb=169425KB/s, maxb=169425KB/s, mint=12378msec, maxt=12378msec

Disk stats (read/write):
*  vda: ios=62/444879, merge=28/119022, ticks=168/530425, in_queue=530499, util=89.71%

The key things to look for from these results are:

  • - On the 1GB LiquidWeb SSD VPS, the fio test was able to achieve A Lot more than this run displays random write IOPS with buffered IO, a QD of 1, and 8 jobs writing to 8 x 256MB files for up to 60 seconds. Almost everyone uses the IOPS stat instead of the BW stat for random reads or writes since BW is usually used for sequential tests. An IOP is 1 input or output operation, IOPS is the amount of those operations performed in 1 second. The more IOPS the better.
  • clat percentiles (usec) 95.00th=[ 1] - I prefer to look at clat instead of slat because clat is the time between submission and completion, which is more accurate than just slat which just tells you the submission latency. I like to use the 95% value which tells you that 95% of all IO operations completed in under this time. It does not count the slowest 5%, but there will always be some requests that are slower, we just want to find out how fast most requests would be complete. If you use an average or max number, it's a lot harder to understand how quickly most requests complete. The values for clat are in microseconds, 1 microsecond (1 us) means that the request took 1/1000000 of a second to complete. Don't get this value mixed up with 1 Millisecond which is 1/1000 of a second, and is a few orders of magnitude slower. This value may not always be accurate, especially if you are testing out a VPS / Cloud Server. Anytime you get a hypervisor involved with IO there will be some wonkyness since the guest instance can't directly access the block device (at least in most cases), the times may be slightly off compared to testing on Bare Metal hardware.
  • util=89.71% - Once we know how many IOPS the device can handle, and how quickly 95% of the operations complete, the last thing I usually want to know is if the device was maxed out during the test. In this case the block storage device for my 1GB VPS was only 90% utilized which means it still has capacity to serve some requests even while I was running the test. Usually you want to push the device to 100% utilization during the test or you won't get it's true performance potential.

Random Read

Random read test. This reads a total of 4GB files, running 8 processes at once. If you are testing a server with say 8GB of RAM, you would want to adjust the file size to be double the RAM to avoid excessive buffering, if there is 8GB of RAM, then set size to 2G. Group reporting will combine each job's stats so you get an overall result.

fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=8 --runtime=240 --group_reporting

Google Compute SSD FIO Script

This FIO script is meant to be run on Ubuntu, but can be quickly converted to work with any OS, such as CentOS 7. The script can be found on Google's local ssd documentation page. I've modified the commands slightly to use less space. The original script uses 100GB of space, but on many smaller VPS / Cloud servers there is not this much free space available, so to prevent people from reaching 100% storage utilization I've lowered the size to 10GB. If you want to test out a smaller or larger size just modify the script below.

# Change this variable to the path of the device you want to test
block_dev=/$mount/$point

# install dependencies
sudo apt-get -y update
sudo apt-get install -y fio

# full write pass
sudo fio --name=writefile --size=10G --filesize=10G \
--filename=$block_dev --bs=1M --nrfiles=1 \
--direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
--iodepth=200 --ioengine=libaio

# rand read
sudo fio --time_based --name=benchmark --size=10G --runtime=30 \
--filename=$block_dev --ioengine=libaio --randrepeat=0 \
--iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
--numjobs=4 --rw=randread --blocksize=4k --group_reporting

# rand write
sudo fio --time_based --name=benchmark --size=10G --runtime=30 \
--filename=$block_dev --ioengine=libaio --randrepeat=0 \
--iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
--numjobs=4 --rw=randwrite --blocksize=4k --group_reporting

AWS EBS FIO Tests

AWS tests:

##The following command performs 16 KB random write operations.

fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting

##The following command performs 16 KB random read operations.

fio --name fio_test_file --direct=1 --rw=randread --bs=4k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting

Fio Random Write Test using libaio and direct flags

This Fio test will use libaio for the ioengine, it will also use direct io for the tests. This means the test will avoid using the host's page cache for writes and write directly to disk. This may produce lower test results, however they will be much more consistent and represent a more real world scenario.

fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=8 --runtime=240 --group_reporting

Fio Random Read Test using libaio and direct flags

This Fio test will use libaio for the ioengine, it will also use direct io for the tests. This means the test will avoid using the host's page cache for reads and will read directly from disk. This may produce lower test results, however they will be much more consistent and represent a more real world scenario.

The nice thing about using directfor random read tests is that you do not have to create some massive test file to avoid inflated test results due to reading out of RAM.

fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=8 --runtime=240 --group_reporting

Sysbench

Sysbench is one of a few other Benchmarking utilities that can be used to test out VPS and Cloud Server performance. Sysbench is one of the most common benchmarking utilities, so it's easy to find and compare performance results since a lot of people use Sysbench and similar tests.

Sysbench allows you to test out performance in the following areas:

  • CPU -- This is a CPU performance test which uses a maximum prime number value that's defined when the test is run to determine how long it takes the system to calculate the numbers. You can specify how many threads to use for this test, making it a quick and easy way to test CPU performance for various amounts of threads.
  • OLTP -- This is a Database / MySQL performance test which helps to estimate best case scenario MySQL performance. You can simulate read only workloads, by only using SELECT queries, or you can test mixed read and write environments. Almost anyone who writes about MySQL performance uses Sysbench OLTP for performance tests and results.
  • fileio -- This is your disk IO test. Similar to FIO, but much less complex in terms of output and options. You can test sequential reads and writes, as well as random reads and writes. Sysbench fileio allows you to modify block size for IO, and the ability to toggle direct io, sync, async, and various other IO related functions.

Installing Sysbench

You can install Sysbench via apt-get, yum, or .rpm / .deb packages. You can also install Sysbench from source, either way installing Sysbench is usually pretty simple. The latest version of Sysbench in 0.5 however the most popular version is 0.4.12. There are some slight performance differences between 0.4.12 and 0.5, specifically with the OLTP test, Sysbench 0.5 tends to provide slightly lower results, this may change in the future, but if you are comparing Sysbench 0.4.12 results to Sysbench 0.5 results, make sure you run some additional tests to verify the results are correct.

Sysbench 0.4.12 Source Install On CentOS 6.5 and CentOS 6.6

wget the source files for Sysbench, unzip the tar.gz file, then install the dependencies needed for Sysbench. This should work on most Redhat / CentOS based systems.

wget http://pkgs.fedoraproject.org/repo/pkgs/sysbench/sysbench-0.4.12.tar.gz/3a6d54fdd3fe002328e4458206392b9d/sysbench-0.4.12.tar.gz
tar zxvf sysbench-0.4.12.tar.gz
yum install libtool.x86_64 openssl-devel.x86_64 openssl-static.x86_64
cd sysbench-0.4.12/
libtoolize --force --copy
./autogen.sh
./configure
make
make install

/usr/bin/ld: cannot find -lssl If you see this error, make sure that you have the openssl-devel and openssl-static packages installed.

Install Sysbench 0.4.12 via RPM on CentOS 6.6

To install Sysbench 0.4.12 on CentOS 6.x or Redhat you can wget the latest remi and epel repos, install them, then install sysbench.

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
yum install sysbench

Install Sysbench on CentOS 7

This should work ok for CentOS 7, verified it does indeed install sysbench 0.4.12

wget ftp://ftp.gnome.org/mirror/fedora/epel/6/x86_64/sysbench-0.4.12-5.el6.x86_64.rpm
wget http://downloads.mysql.com/archives/mysql-5.1/MySQL-shared-compat-5.1.49-1.rhel5.x86_64.rpm
rpm -iv MySQL-shared-compat-5.1.49-1.rhel5.x86_64.rpm
yum install postgresql-libs.x86_64
rpm -iv sysbench-0.4.12-5.el6.x86_64.rpm

Install Sysbench 0.4.12 on Ubuntu 14.04 and 14.10

If you want to stick with the slightly older 0.4.12 Sysbench you can easily install it on most modern Debian based distros by simply apt-get installing sysbench.

apt-get install sysbench

Install Sysbench 0.5 via Percona Repo on Ubuntu 14.04

Percona includes the latest version of sysbench in their repos, and if you care about performance, chances are you are using Percona, if not, then you may not want to install it this way.

echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
apt-get update
apt-get install sysbench

Sysbench OLTP / MySQL Tests

Creating a test database with Sysbench

Assuming you have Sysbench installed, you can just login to MySQL if you don't save the root mysql password in /root/.my.cnf then you will need to use mysql -p and enter in the root mysql password. Create a new database and user to use as the test database, generally I just use sysbench for the user and the db and stick with a simple password since I am only testing on a temporary server. Make sure you replace my weak password with an actual secure password, otherwise you could get compromised in a hurry especially if you configure MySQL to listen on a public IP

##Create Database 
mysql -u root -ppassword -e "CREATE DATABASE sysbench;"

##Create User
mysql -u root -ppassword -e "CREATE USER 'sysbench'@'localhost' IDENTIFIED BY 'password';"

##Grant Access 
mysql -u root -ppassword -e "GRANT ALL PRIVILEGES ON *.* TO 'sysbench'@'localhost' IDENTIFIED  BY 'password';"

Once you have created the test database and user, run the command below to fill up the database with some data. If you use an oltp table size of 10000000, sysbench will create a database that is about 2GB in size. If you double the table size to 20000000 the database will be around 4GB in size. I recommend creating a test database that is close to the size of databases that you normally see in production.

sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password prepare

Read Only OLTP tests

This command will run a Read-Only Sysbench OLTP test without the "BEGIN" statement. I like to run OLTP tests at various thread levels to get an idea of how MySQL will perform under certain workloads. This test will start off running a single threaded OLTP read only test, sleep for 10 seconds, then run a 4 threaded OLTP read only test, and will continue to do that until the 64 threaded test has finished. If you only have a few CPUs you may not need to run 64 threaded tests.

for each in 1 4 8 16 32 64; do sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=$password --max-time=240 --max-requests=0 --oltp-read-only --oltp-skip-trx  --oltp-nontrx-mode=select --num-threads=$each run; sleep 10; done


Similar to the test above, this is also a Read-Only Sysbench OLTP test, but without the extra options.

for each in 1 4 8 16 32 64; do sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=240 --oltp-read-only=on --max-requests=0 --num-threads=$each run; sleep 10; done

Mixed Read and Write OLTP tests

Read and Write OLTP Mixed Test. Usually about 70% read (select) and 30% write (insert, update, delete)

sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --max-requests=0 run

Read and Write OLTP test with 16 threaded and random connection mode

sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --max-requests=0 --num-threads=16 --oltp-reconnect-mode=random run

Simple Sysbench OLTP Test Script for Ubuntu and Debian

Make sure the system is updated. Then simply install Sysbench, and MySQL Server if you plan on running OLTP tests.

apt-get update
apt-get install -y sysbench
apt-get install -y mysql-server
sysbench --test=fileio --file-total-size=4G --file-num=64 prepare

mysql -u root -ppassword -e "CREATE DATABASE sysbench;"
mysql -u root -ppassword -e "CREATE USER 'sysbench'@'localhost' IDENTIFIED BY 'password';"
mysql -u root -ppassword -e "GRANT ALL PRIVILEGES ON *.* TO 'sysbench'@'localhost' IDENTIFIED  BY 'password';"
sysbench --test=oltp --db-driver=mysql --oltp-table-size=40000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password prepare


You can then run this script which will run each test 3 times at each thread level listed. This can help to find the "sweet spot" in terms of performance.

#!/bin/bash

for run in 1 2 3 ;do
for thread in 1 4 8 16 32 ;do

echo "Performing test RW-${thread}T-${run}"
sysbench --test=fileio --file-total-size=4G --file-test-mode=rndwr --max-time=60 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=${thread} run > /root/RW-${thread}T-${run}

echo "Performing test RR-${thread}T-${run}"
sysbench --test=fileio --file-total-size=4G --file-test-mode=rndrd --max-time=60 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=${thread} run > /root/RR-${thread}T-${run}

echo "Performing test SQ-${thread}T-${run}"
sysbench --test=oltp --db-driver=mysql --oltp-table-size=40000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=60 --max-requests=0 --num-threads=${thread} run > /root/SQ-${thread}T-${run}

done
done


Sysbench 0.5 OLTP Read and Write Test

This script will run 6 tests, each lasting 4 minutes. It will run 1 through 64 threaded tests, which seem to be the most common tests to run. This test does selects, updates, and various other things and is considered to be a "read / write" MySQL mixed workload.

for each in 1 4 8 16 32 64; do sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --oltp-table-size=20000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=240 --max-requests=0 --num-threads=$each run; done

Sysbench 0.5 OLTP Read Only Test

This script will run 6 tests, each lasting 4 minutes. It will run 1 through 64 threaded tests, which seem to be the most common tests to run. This runs selects only, and is considered to be "read only".

for each in 1 4 8 16 32 64; do sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --oltp-table-size=20000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --oltp-read-only=on --max-time=240 --max-requests=0 --num-threads=$each run; done

Test Script

Prepare the server

apt-get update
apt-get install -y sysbench 
apt-get install -y mysql-server
sysbench --test=fileio --file-total-size=4G --file-num=64 prepare

mysql -u root -ppassword -e "CREATE DATABASE sysbench;"
mysql -u root -ppassword -e "CREATE USER 'sysbench'@'localhost' IDENTIFIED BY 'password';"
mysql -u root -ppassword -e "GRANT ALL PRIVILEGES ON *.* TO 'sysbench'@'localhost' IDENTIFIED  BY 'password';"
sysbench --test=oltp --db-driver=mysql --oltp-table-size=40000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password prepare


#!/bin/bash

for run in 1 2 3 ;do
for thread in 1 4 8 16 32 ;do

echo "Performing test RW-${thread}T-${run}"
sysbench --test=fileio --file-total-size=4G --file-test-mode=rndwr --max-time=60 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=${thread} run > /root/RW-${thread}T-${run}

echo "Performing test RR-${thread}T-${run}"
sysbench --test=fileio --file-total-size=4G --file-test-mode=rndrd --max-time=60 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=${thread} run > /root/RR-${thread}T-${run}

echo "Performing test SQ-${thread}T-${run}"
sysbench --test=oltp --db-driver=mysql --oltp-table-size=40000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=60 --max-requests=0 --num-threads=${thread} run > /root/SQ-${thread}T-${run}

done
done

Various Tests

Disk Tests

##The size of the file depends on the amount of RAM in the system, so if you have 32GB RAM installed, the test file needs to be larger than 32GB
sysbench --test=fileio --file-total-size=60G --file-num=64 prepare

##Random Write using no fsync (Should show the highest potential random writes)
sysbench --test=fileio --file-total-size=60G --file-test-mode=rndwr --max-time=720 --max-requests=0 --num-threads=8 --file-num=64 --file-io-mode=async --file-extra-flags=direct --file-fsync-freq=0 run

##Random Write using default fsync (Should show typical fsync behavior)
sysbench --test=fileio --file-total-size=60G --file-test-mode=rndwr --max-time=720 --max-requests=0 --num-threads=8 --file-num=64 --file-io-mode=async --file-extra-flags=direct --file-fsync-freq=0 run


##Random Read Only
sysbench --test=fileio --file-total-size=20G --file-test-mode=rndrd --max-time=240 --max-requests=0 --num-threads=${1,8,64} run

##Mixed Random Read and Write
sysbench --test=fileio --file-total-size=20G --file-test-mode=rndrw --max-time=240 --file-fsync-all --max-requests=0 --num-threads=${1,8,64} run


###For Loops

###RAND WRITE 1 - 16 thread
for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run; done; sleep 10; for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=4 run; done; sleep 10; for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=8 run; done; sleep 10; for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=16 run; done;

###RAND READ 1 - 16 thread
for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run; done; sleep 10;for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=4 run; done; sleep 10;for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=8 run; done; sleep 10;for each in {1..3} ; do sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=16 run; done; sleep 10;

CPU Tests

##This test defaults to using 1 thread, good to run with one thread, then 32 or whatever
sysbench --test=cpu --cpu-max-prime=20000 run

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=32 run

RAM Tests

sysbench --test=memory --memory-block-size=1K --memory-scope=global --memory-total-size=100G --memory-oper=read run

sysbench --test=memory --memory-block-size=1K --memory-scope=global --memory-total-size=100G --memory-oper=write run

Using OLTP with custom .lua scripts

##Install and move to the db test directory##
wget wget http://www.lefred.be/files/sysbench-0.5-3.el6_.x86_64.rpm
rpm -iv sysbench-0.5-3.el6_.x86_64.rpm
cd /usr/share/doc/sysbench/tests/db

##Prepare a test table on the target system##
sysbench --test=parallel_prepare.lua --oltp-table-size=10000000 --mysql-host=$remote_host --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --oltp-tables-count=1 --num-threads=32 prepare

##Run Tests based off of .lua##
sysbench --test=oltp_simple.lua --oltp-table-size=10000000 --mysql-host=$remote_host --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --oltp-read-only=on --oltp-skip-trx=on    --max-time=120 --max-requests=0 --max-time=120 --num-threads=512 run


MySQL Tests

##Create Database 
CREATE DATABASE sysbench;

##Create User
CREATE USER 'sysbench'@'localhost' IDENTIFIED BY '$password';

##Grant Access 
GRANT ALL PRIVILEGES ON *.* TO 'sysbench'@'localhost' IDENTIFIED  BY '$password';

##Prepare a test table
sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password prepare

##Read-Only Test without "BEGIN"
sysbench --db-driver=mysql --test=oltp --oltp-table-size=10000000 --mysql-host=$IP --mysql-db=sysbench --mysql-user=sysbench --mysql-password=$password --max-time=120 --max-requests=0 --oltp-read-only --oltp-skip-trx  --oltp-nontrx-mode=select --num-threads=256 run

##Read-Only Test
sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

##Mixed Test
sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --max-requests=0 run

##Mixed Test 8 threads
sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --max-requests=0 --num-threads=8 run

##16 threads, Random connect / disconnect for threads
sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password --max-time=60 --max-requests=0 --num-threads=16 --oltp-reconnect-mode=random run
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/common.lua-20110427224028-j1y84gokmt93n8xn-1/common.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/sysbenchtestsdbinser-20090113060453-687cibzqa6u5f1i4-115/insert.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/oltp.lua-20110427224028-j1y84gokmt93n8xn-2/oltp.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/sysbenchtestsdboltp_-20090113060453-687cibzqa6u5f1i4-118/oltp_simple.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/sysbenchtestsdbselec-20090113060453-687cibzqa6u5f1i4-119/select.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/select_random_points-20101028165209-ue1iedn6sn5azlvx-1/select_random_points.lua
wget http://bazaar.launchpad.net/~vadim-tk/sysbench/sysbench-db-scripts/download/head:/select_random_ranges-20101028165209-ue1iedn6sn5azlvx-2/select_random_ranges.lua

Sysbench CPU Tests

To run a single threaded CPU test, run the following command. Generally speaking it's a good idea to run a single threaded sysbench CPU test, as well as another rest with threads equaling the amount of CPUs the server has. If it has 4 cores and 8 threads then I suggest running a single threaded test, 4 threads, and 8 threads to get an idea of how threaded process scales for your server.

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

To run a 4 threaded Sysbench CPU test, use this command.

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run

You can use the script below to run multi threaded Sysbench CPU tests

for each in 1 2 4 8 16 32 64; do sysbench --test=cpu --cpu-max-prime=20000 --num-threads=$each run; done

The lower the result, the faster the CPU. Multithreaded CPUs will not show their true performance unless you run the test with the correct amount of threads. If you have an 8 core CPU you should run an 8 thread CPU test to get the true performance number.

Cloud / VPS CPU Performance Test Results with Sysbench

If you only have a single CPU server, there no real reason to run CPU tests with thread levels that are well above your CPU count as performance won't scale past the amount of CPUs you have (at least in most cases). This is not the case with Disk IO and OLTP (MySQL) tests which usually scale to a thread count of something like:

2 x vCPU #

So when I run Sysbench's CPU test, I'll use --num-threads=1 for VPS servers with only one CPU. The total count of vCPUs is determined by running

cat /proc/cpuinfo


Random Section - If you see only 1 CPU, you have 1 vCPU. Whether or not that virtual CPU has an entire physical CPU to itself is not up to you, it's up to whoever is hosting your VPS. You can slice up, divide, split up, or segment a Physical Server into many pieces using software like KVM or Xen. A Hypervisor keeps track of these slices and can enforce resource limits to make sure no single VM can use all the resources at once. The problem with this is that the definition of a vCPU or CPU is changing, and never the same, at least when a hypervisor is involved.

Below are some example results using 1 and 2 threaded Sysbench CPU tests. Both of these servers are virtualized instances. Azure instance is using HyperV for the hypervisor and the LiquidWeb VPS server is using KVM for the hypervisor. As you can see, since the Azure VPS only has a single vCPU the performance between the 1 and 2 threaded tests are basically the same. With the LiquidWeb VPS performance scaled as expected between the 1 and 2 threaded test.

This is why it's important to test out multiple threaded CPU tests since not all vps or cloud vps servers are the same in terms of vCPU count and CPU utilization limits. As you can see here, the LiquidWeb 1GB SSD VPS]is almost 4 times faster than the Azure A1 instance in terms of CPU performance. Again, lower times are better, longer test times means slower. The vps CPU performance results are not surprising. LiquidWeb's VPS has 2 E3-1270 vCPUs which are much newer and faster than Azure's AMD Opteron.

  • Azure A1 Standard with 1 vCPU (AMD Opteron 4171 HE @ 2.1GHz) (Tested on 01-12-2015)
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

1 thread 49.3510s

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

2 thread 49.7294s
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

1 thread 24.3093s

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

2 thread 13.2732s
  • Digital Ocean 1GB VPS with 1 vCPU (E5-2630L) (Tested on 10-13-2014)
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

1 thread 35.69s

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

2 thread 35.32s
  • Rackspace 1GB Standard VPS 1 vCPU (E5-2670v1) (Tested on 10-13-2014)
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

1 thread 28.85s

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

2 thread 28.87s
  • AWS t2.micro 1 vCPU (E5-2670v2) (Tested on 10-13-2014)
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run

1 thread 26.77s

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

2 thread 26.73s

As you can see, CPU performance can be all over the place, especially with VPS and Cloud servers, which are often times sharing the same CPU with many other instances. If you notice a large variation in CPU performance between CPU test runs then you may be on a server with lots of "noisy neighbors" who are also using lots of CPU.

As you can see, LiquidWeb VPS has the fastest CPU for 1 and 2 threaded CPU performance. AWS come in a close second with Rackspace slightly behind, but only with single threaded performance. The 1GB LiquidWeb VPS was two times faster than AWS and Rackspace for 2 threaded CPU performance, at least for the 1GB VPS tier. Hardware changes often, so these tests are meant to be an example of what Sysbench can tell you about performance. Please test CPU performance and see for yourself!


You can use the script below to run multi threaded Sysbench CPU tests. If you want to run a more demanding test, modify the cpu max prime value to a larger number, such as 80000 or higher. The higher this value is the longer the tests take to complete. Typically with a value of 20000, single threaded tests can take anywhere from 20 seconds to a minute or more. The shorter the time, the better the result, so faster CPUs will take less time to run the test.

for each in 1 2 4 8 16; do sysbench --test=cpu --cpu-max-prime=20000 --num-threads=$each run; done

The lower the result, the faster the CPU. Multithreaded CPUs will not show their true performance unless you run the test with the correct amount of threads. If you have an 8 core CPU you should run an 8 thread CPU test to get the true performance number.

Sysbench Fileio Options

This section has been moved to the main Sysbench Wiki Page

Sysbench Fileio Tests

First prepare files to use for the test, you can change the size to whatever you want, here I am using 8GB file size, with 64 files total.

sysbench --test=fileio --file-total-size=8G --file-num=64 prepare

This will run 1 - 64 threaded IO tests for random read, random write with default fsync which means data is flushed to disk ever 100 write requests. I am also testing random writes with fsync all which means that after each write data is flushed to disk. I like to run both random write tests because the first is more "real world" the second is much harsher, but for servers / applications that ensure every write is safe, the fsync all IO test is a good way to get a general idea of IO performance.

for each in 1 4 8 16 32 64; do sysbench --test=fileio --file-total-size=8G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=$each run; sleep 10; done; for each in 1 4 8 16 32 64; do sysbench --test=fileio --file-total-size=8G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --file-fsync-all --num-threads=$each run; sleep 10; done; for each in 1 4 8 16 32 64; do sysbench --test=fileio --file-total-size=8G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=$each run; sleep 10; done

Sysbench Prepare

Generally, I like to create / prepare test files before I run the read and write tests. To prepare test files, run the command below. Typically I will prepare a test file with a size that is about two times more than the amount of RAM the server has. If it's a 2GB vps then I'll use a 4GB file size, if it's a 1GB vps then I'll use 2GB file size. I do this so that the ratio between file size and RAM is always roughly the same. You can force direct io to get around inflated cached results, but if you always use a test file thats 2 times larger than RAM you will avoid having all reads come from the cache in RAM which will give you super inflated results. If you notice very high numbers for IO tests odds are you need to flush caches and buffers before testing IO again.

sysbench --test=fileio --file-total-size=2G --file-num=64 prepare
  • file-total-size Can be adjusted to whatever size you want. You can use M for Megabytes as well. I like to use a file size of at least 2 times the amount of RAM that the box has.
  • file-num Can be adjusted as well. The default is 128. but you can raise or lower this value to more closely match your workload.

Sysbench Random Write Test

Below is a basic Sysbench random write test. This test uses a file size of 2GB, and will run for a maximum of 240 seconds before finishing. For larger tests you might want to enforce a max time so that the tests do not run for hours and hours. The example below is only running a singled threaded test. Typically, the higher you set num-threads the faster the result will be. However, this is only true up to a point, usually performance will start to cap out once you reach a thread count that is 2 x the number of CPUs on the test system. I suggest doubling the amount of threads to use for random read or write tests until you hit the "cap" in terms of performance. If you only run a single threaded random read test on an SSD you will get lower performance results than if you used 16 threads for the test. Usually SSDs cap out around 64 threads, but faster PCIe Flash devices scale up to 128 or even 256 threads if you have a fast enough CPU to handle the IO.

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run

Make sure you change file-total-size to match whatever size you used to prepare the sysbench files, again, I like to use a file test size of roughly 2 x the amount of VPS RAM to avoid having all reads come from the cache. Write speed can also be effected by the file size as well, so you can get somewhat inflated results if you write a 2GB file on a server with 16GB of ram since a lot of the writes will hit the buffer before hitting the disk, while this is normal behavior it does not isolate the underlying block device's performance unless you use direct io flags, or use a file size that is large enough to not fit into RAM.

I prefer to run the script below, which will run the same random write test multiple times. This test will run 1, 4, 8, 16, 32, and 64 threaded random write tests.

for each in 1 4 8 16 32 64; do sysbench --test=fileio --file-total-size=2G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=$each run; sleep 10; done

Sysbench Random Read Test

Below is a basic Sysbench random read test. This test uses a file size of 2GB, and will run for a maximum of 240 seconds before finishing. For larger tests you might want to enforce a max time so that the tests do not run for hours and hours. The example below is only running a singled threaded test. Typically, the higher you set num-threads the faster the result will be. However, this is only true up to a point, usually performance will start to cap out once you reach a thread count that is 2 x the number of CPUs on the test system.

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run

Make sure you change file-total-size to match whatever size you used to prepare the sysbench files, again, I like to use a file test size of roughly 2 x the amount of VPS RAM to avoid having all reads come from the cache. Write speed can also be effected by the file size as well, so you can get somewhat inflated results if you write a 2GB file on a server with 16GB of ram since a lot of the writes will hit the buffer before hitting the disk, while this is normal behavior it does not isolate the underlying block device's performance unless you use direct io flags, or use a file size that is large enough to not fit into RAM.


I prefer to run the script below, which will run the same random read test multiple times. This test will run 1, 4, 8, 16, 32, and 64 threaded random read file io tests.

for each in 1 4 8 16 32 64; do sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=$each run; sleep 10; done

VPS and Cloud Server Random Write Performance Tests Using Sysbench

Generally I like to prepare a set of files to use for random read, or write tests. For random write tests you could potentially skip the prepare step, but if you are going to run read and write tests you might as well just prepare the test files first. I'm testing out 1GB VPS performance, so I am using a test file size of roughly twice the amount VPS RAM. I do this to avoid the possibility of Linux caching all the data in RAM, at least this way the ratio between file size and RAM is always 2:1 to avoid all reads coming from RAM. I don't want to entirely eliminate caching effects, just reduce them to a more realistic scenario. If you don't think 2:1 is a good ratio then feel free to adjust the file size to whatever you want, this is just an example.

sysbench --test=fileio --file-total-size=2G --file-num=64 prepare

This is one of the more simple random write tests. The test runs for 240 seconds, or 4 minutes, uses a block size of 4KB, using 64 files that are 32MB in size each. By default Sysbench will use FSYNC after every 100 requests. Syncronous I/O mode is used by default unless you specify async, but that is not supported on all OSs.

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run

This is what the test result will look like once the test is done. I've removed some of the output to make it easier to display the results as well as to focus on the results that really matter. The results below were collected from LiquidWeb's 1GB SSD VPS

Operations performed:  0 Read, 2242700 Write, 1435295 Other = 3677995 Total
Read 0b  Written 8.5552Gb  Total transferred 8.5552Gb  (36.502Mb/sec)
 9344.54 Requests/sec executed

Test execution summary:
    total time:                          240.0010s
    total number of events:              2242700
    total time taken by event execution: 10.4937
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.00ms
         max:                                  8.28ms
         approx.  95 percentile:               0.01ms

The most important information to look at from these results are

  • Requests/sec executed: 9344.54

Requests/s executed is more or less IOPS or input / output operations per second. In this case it means that the 1GB LiquidWeb SSD VPS is able to perform 9,344(iops) 4KB write operations per second. Some people prefer to view performance in MB/s or even GB/s. Sysbench displays both results (36.502Mb/sec) is roughly equal to 4KB x 9344 = 37376KB/s or about 36MB/s. IOPS is much more useful to know, but you must also include the block size used or else the results will be meaningless. A 4KB IOP is not equal to a 16KB IOP. Often times people will use 16KB or 1MB block sizes for random write tests which will inflate results because each IOP is transferring more data, so the throughput is higher.

Most of the time you will be dealing with 4KB block sizes so if you are testing random write performance you should really stick with 4KB unless your environment deals with different block sizes.

  • approx. 95 percentile: 0.01ms

IOPS only tells you so much about performance, to get a full picture of IO performance you want to look at the 95% response time as well. This number tells you what 95% of of response times looks like. This is a much more useful metric than "average" or "minimum" response times. Maximum response time is another good metric, but most benchmarking sites and results typically show IOPS and 95% response times.

In the example above, the LiquidWeb 1GB SSD VPS had a 95% response time of 0.01ms (milliseconds) for write operations. Depending on the hypervisor in use, these times may not always be accurate, sysbench usually warns of this when you run the test. If you run this test on a server that is not running on top of a hypervisor then the response times will be lower and probably more accurate since there isn't a hypervisor involved so keeping track of time is easier.

If the hypervisor is configured to use Write-Back Caching then response times will significantly improve, however it's important to keep in mind that just because the Guest VPS thinks a write is on disk, doesn't mean that is actually the case on the physical host. There is no right way to configure the hypervisor, and enabling write-back caching is fine if the underlying storage is protected from power loss via BBU or other non-volitile flash storage type solutions.

Usually random write performance will be very low if you only use 1 or 2 threads to test performance. Especially if you have a fast CPU. If I run a 4 threaded random write test I get about 2 x performance, at least for the results.

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndwr --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=4 run

The results below were collected from the same LiquidWeb 1GB SSD VPS from the single threaded test. Notice that the results look much better! IO performance for this vps doubled from almost 10,000 iops to over 20,000 iops.

Operations performed:  0 Read, 5396917 Write, 3445004 Other = 8841921 Total
Read 0b  Written 20.588Gb  Total transferred 20.588Gb  (87.837Mb/sec)
22486.39 Requests/sec executed

Test execution summary:
    total time:                          240.0081s
    total number of events:              5396917
    total time taken by event execution: 56.6653
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.01ms
         max:                                 41.72ms
         approx.  95 percentile:               0.01ms

Generally, you want to keep increasing thread count until your results top out. On most VPS / Cloud Servers this usually happens when the thread count is 2x - 4x server CPU count. This is not always the case though, so I usually run 1,4,8,16,32, and 64 threaded random read and write tests to see how performance scales.

Most VPS instances are on shared physical hosts, so IO will almost always be changing based on what other guests are doing. I suggest running each test 3 or 4 times and then averaging those results to get a general idea on performance. I like to test performance at different times of the day for these averages. 3 tests at noon on a sunday are not going to be the same as 3 tests on monday, tuesday and wednesday at 3PM. The point being, don't trust every benchmark result you see (mine included), test out performance for yourself, using tests that simulate your application / environment.

VPS and Cloud Server Random Read Performance Tests Using Sysbench

As mentioned in the previous section, I like to prepare a set of files to use for random read, or write tests.

For random read tests you need to prepare the test files before you can actually run the random test. I'm using a LiquidWeb 1GB SSD VPS for these examples. I am using a test file size of roughly twice the amount VPS RAM. I do this to avoid the possibility of Linux caching all the data in RAM, at least this way the ratio between file size and RAM is always 2:1 to avoid all reads coming from RAM. I don't want to entirely eliminate caching effects, just reduce them to a more realistic scenario. If you don't think 2:1 is a good ratio then feel free to adjust the file size to whatever you want, this is just an example.

First up, 'run this command to create a total test file size of 2GB.

sysbench --test=fileio --file-total-size=2G --file-num=64 prepare

This is one of the more simple random read tests. The test runs for 240 seconds, uses 1 thread, uses a block size of 4KB, using 64 files that are 32MB in size each. By default Sysbench will use FSYNC after every 100 requests, even if it's a read only test. Syncronous I/O mode is used by default unless you specify async, but that is not supported on all OSs.

You will notice this is almost the exact same command I used for the random write test, the only difference being file-test-mode which is rndrd (random read).

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=1 run

This is what the test result will look like once the test is done. I've removed some of the output to make it easier to display the results as well as to focus on the results that really matter. The results below were collected from LiquidWeb's 1GB SSD VPS I used a single thread for this random read test.

Operations performed:  1013246 Read, 0 Write, 0 Other = 1013246 Total
Read 3.8652Gb  Written 0b  Total transferred 3.8652Gb  (16.492Mb/sec)
 4221.85 Requests/sec executed

Test execution summary:
    total time:                          240.0004s
    total number of events:              1013246
    total time taken by event execution: 238.9431
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.24ms
         max:                                  8.73ms
         approx.  95 percentile:               0.36ms

The most important information to look at from these results are

  • Requests/sec executed: 4221.85

Requests/s executed is more or less IOPS or input / output operations per second. In this case it means that the 1GB LiquidWeb SSD VPS is able to perform 4221.85 (iops) 4KB read operations per second. Some people prefer to view performance in MB/s or even GB/s. Sysbench displays both results (16.492Mb/sec) is roughly equal to 4KB x 4221 = 16,884KB or about 16MB/s. IOPS is much more useful to know, but you must also include the block size used or else the results will be meaningless. A 4KB IOP is not equal to a 16KB IOP. Often times people will use 16KB or 1MB block sizes for random read or write tests which will inflate results because each IOP is transferring more data, so the throughput is higher.

Most of the time you will be dealing with 4KB block sizes so if you are testing random read performance you should really stick with 4KB unless your environment deals with different block sizes.

  • approx. 95 percentile: 0.36ms

IOPS only tells you so much about performance, to get a full picture of IO performance you want to look at the 95% response time as well. This number tells you what 95% of of response times looks like. This is a much more useful metric than "average" or "minimum" response times. Maximum response time is another good metric, but most benchmarking sites and results typically show IOPS and 95% response times. Obviously the goal is to read as much data from RAM as possible, but understanding what non-cached random read performance looks like is pretty important, especially if your application is mainly read only.


Just like the random write test, it's important to run random read tests using different amount of threads. Performance will almost always be higher if you run 4 threaded tests versus single threaded IO tests. Typically random read performance will scale up to around 64 or 128 threads even if you have 1 or 2 CPUs.

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=4 run

With 4 threads, the LiquidWeb 1GB SSD VPS gets about 17,000 IOPS for random read performance. With 95% of the requests completing in under 0.36ms

Operations performed:  4085976 Read, 0 Write, 0 Other = 4085976 Total
Read 15.587Gb  Written 0b  Total transferred 15.587Gb  (66.503Mb/sec)
17024.87 Requests/sec executed

Test execution summary:
    total time:                          240.0004s
    total number of events:              4085976
    total time taken by event execution: 956.7898
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.23ms
         max:                                 38.60ms
         approx.  95 percentile:               0.36ms

Let's get to the point and run a 16 threaded random read test. Keep in mind the server I am running this on has 2 vCPUs and 50GB of SSD storage

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=16 run

This 1GB VPS is almost at 50,000 IOPS for random reads. Not too shabby!

Operations performed:  11679232 Read, 0 Write, 0 Other = 11679232 Total
Read 44.553Gb  Written 0b  Total transferred 44.553Gb  (190.09Mb/sec)
48662.93 Requests/sec executed

Test execution summary:
    total time:                          240.0027s
    total number of events:              11679232
    total time taken by event execution: 3829.1727
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.33ms
         max:                                 51.21ms
         approx.  95 percentile:               0.68ms

Since performance continues to scale I'm going to run a 32 threaded random read test to see what kind of results I get

sysbench --test=fileio --file-total-size=2G --file-test-mode=rndrd --max-time=240 --max-requests=0 --file-block-size=4K --file-num=64 --num-threads=32 run

LiquidWeb's 1GB SSD VPS performed pretty well for the 32 threaded random read test, reaching almost 64,000 IOPS for random reads.

Operations performed:  15348543 Read, 0 Write, 0 Other = 15348543 Total
Read 58.55Gb  Written 0b  Total transferred 58.55Gb  (249.81Mb/sec)
63951.43 Requests/sec executed

Test execution summary:
    total time:                          240.0031s
    total number of events:              15348543
    total time taken by event execution: 7663.7240
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.50ms
         max:                                103.57ms
         approx.  95 percentile:               1.27ms

Threads fairness:
    events (avg/stddev):           479641.9688/3418.13
    execution time (avg/stddev):   239.4914/0.02

I could continue to run more tests, but I think you get the idea. Make sure you test IO performance at various thread levels, don't just assume 1 thread will be good enough, many SSD VPS will perform, or at least appear to perform similarly when running light IO tests, but to get an idea of VPS IO performance under heavy IO you need to run at least 8 or 16 threaded tests to get an idea.

Anyway, LiquidWeb's 1GB SSD VPS appear to perform very well.

Intel E5-2690v2 KVM Sysbench CPU Pinning Results

  • Testing using a single VM located on a CentOS 6 box. Running KVM, allocated all 40 vCPUs to the guest instance.

Running Sysbench CPU with 40 threads

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=40 run
  • CPU Affinity all Total Time Taken: 0.9807s

By default each vCPU's CPU affinity will be set to each Host CPU. You can change force this by running the command below

for each in {0..39}; do virsh vcpupin $instance_ID $each 0-39 ; done
  • Each vCPU Pinned to Corresponding Host CPU Total Time Taken: 0.9218s

To optimize performance, I pinned each vCPU to the corresponding host CPU. So vCPU 0 affinity is set to Host CPU 0, etc, etc

for each in {0..39}; do virsh vcpupin $instance_ID $each $each ; done

Using Sysbench 0.5 LUA

Phoronix Test Suite

The Phoronix Test Suite section was getting too large, it has been moved it it's own page which can be found here Phoronix Test Suite

IOzone

This will run a ton of IOzone tests, sleeping for 60 seconds in-between them

iozone -l 8 -i 0 -i 1 -i 2 -e -+n -r 4K -s 1G -O ;
sleep 60 ;
iozone -l 8 -i 0 -i 1 -i 2 -e -+n -r 4K -s 1G ;
sleep 60 ; 
iozone -e -s 12G -r 64K -t 1 -i 0 -i 1 -+n ; 
sleep 60 ; 
iozone -e -s 12G -r 64K -t 1 -i 0 -i 1 -+n ;
sleep 60 ; 
iozone -e -s 12G -r 64K -t 1 -i 0 -i 1 -+n ; 
sleep 60 ; 
iozone -t 1 -i 0 -i 1 -i 2 -e -+n -r 4K -s 12G -O ; 
sleep 60 ;
iozone -t 1 -i 0 -i 1 -i 2 -e -+n -r 4K -s 12G -O

IOzone commands

##Sequential Write, 64K requests, 32 threads:
iozone -I -t 32 -M -O -r 64k -s 500m -+u -w -i 0

##Sequential Read, 64K requests, 32 threads:
iozone -I -t 32 -M -O -r 64k -s 500m -+u -w -i 1

##Random Read/Write, 4K requests, 32 threads:
iozone -I -t 32 -M -O -r 4k -s 500m -+u -w -i 2

IOzone command line options

##Save output to spreadsheet
-b $filename

##Specify test types
-i [1-13]

    0=write/rewrite
    1=read/re-read
    2=random-read/write
    3=Read-backwards
    4=Re-write-record
    5=stride-read
    6=fwrite/re-fwrite
    7=fread/Re-fread,
    8=random mix
    9=pwrite/Re-pwrite
    10=pread/Re-pread
    11=pwritev/Re-pwritev
    12=preadv/Re-preadv

Specify the file size using iozone (file size used to test)
-s $number (1M, 512M, 4G, etc)

Specify the record size for testing
-r $number (1K, 4K, 64K, etc)

Throughput test
-t $threads

Increase the file size
-g $number (usually should be 3x amount of RAM)


Personal tests (work in progress). This should give a good measurement of sequencial reads and writes, the file size will need to be larger than RAM on the server, but 12GB is usually a good starting point.

##Use fsync,file size is 12GB, record size is 64K, use 1 thread for throughput, test read and write, no re-tests, save to file.

iozone -e -s 12G -r 64K -t 1 -i 0 -i 1 -+n -b 12GB_64K_RW.txt

##Use fsync,file size is 12GB, record size is 64K, use 1 thread for throughput, test read and write, no re-tests, save to file, output in OPS/s

iozone -e -s 12G -r 64K -t 1 -i 0 -i 1 -+n -O -b 12GB_64K_RW_OPS.txt

UnixBench Installation and Commands

To run a quick test with Unix bench, you simply need to wget the file, extract it and then use ./Run -i $Number of Runs. Each run can take between 10 and 30 minutes so if you just want a quick idea of performance use -i 1, if you want to run the test more times simply replace "1" with how many times you want to run the test.

apt-get install make gcc
wget https://byte-unixbench.googlecode.com/files/UnixBench5.1.3.tgz
tar zxvf UnixBench5.1.3.tgz
cd UnixBench
make
./Run -i 1

If you run into issue with UnixBench on Ubuntu 15.04 while running make, you will need to install gcc as well as build-essential.

apt-get install make gcc build-essential
wget https://byte-unixbench.googlecode.com/files/UnixBench5.1.3.tgz
tar zxvf UnixBench5.1.3.tgz
cd UnixBench
make
./Run -i 1

If you get a make error while running make that looks like:

./src/arith.c:32:19: fatal error: stdio.h: No such file or directory

or that pgms/arithoh failed / is not found

compilation terminated.
Makefile:185: recipe for target 'pgms/arithoh' failed
make: *** [pgms/arithoh] Error 1

Installing build-essentials will resolve the issue and UnixBench will be able to be built with make without issue. I've only noticed the make issue with UnixBench on Ubuntu 15.04 on some VPS servers, so this issue doesn't appear to be specific to Ubuntu. Anyway, make sure that gcc and build-essentials are installed and you should be able to make and run UnixBench 5.1.3 without issues on Ubuntu 15.04.

ioping Installation and Commands

The latest version of ioping can be found here -- https://code.google.com/p/ioping/downloads/list

To install and run ioping you can simply wget the file, extract it cd into the directory and run "./ioping". The -c option lets you specify the amount of times ioping will attempt a read request. The more times you let ioping request data, the more accurate the test results will be. The other option to specify is . the option after -c, this is the location to run the test. If you only have a single volume, using . will just run the test in your current directory. If you want to run the test in say, /home, you would use a command like this ./ioping -c 1000 /home/. If you want to run the ioping test in the current directory, replace /home/ with "."

wget https://ioping.googlecode.com/files/ioping-0.8.tar.gz
tar zxvf ioping-0.8.tar.gz
cd ioping-0.8/
make
make install
./ioping -c 1000 /home/

##To run ioping in current directory

./ioping -c 1000 .

By default, ioping will use 4KiB block size for each request. If you wish to change the way ioping runs tests you can modify the options shown below

Usage: ioping [-LABCDWRq] [-c count] [-w deadline] [-pP period] [-i interval]
               [-s size] [-S wsize] [-o offset] directory|file|device
        ioping -h | -v

      -c <count>      stop after <count> requests
      -w <deadline>   stop after <deadline>
      -p <period>     print raw statistics for every <period> requests
      -P <period>     print raw statistics for every <period> in time
      -i <interval>   interval between requests (1s)
      -s <size>       request size (4k)
      -S <wsize>      working set size (1m)
      -o <offset>     working set offset (0)
      -k              keep and reuse temporary working file
      -L              use sequential operations (includes -s 256k)
      -A              use asynchronous I/O
      -C              use cached I/O
      -D              use direct I/O
      -W              use write I/O *DANGEROUS*
      -R              seek rate test (same as -q -i 0 -w 3 -S 64m)
      -B              print final statistics in raw format
      -q              suppress human-readable output
      -h              display this message and exit
      -v              display version and exit

When ioping is finished, it will display this information. The most relevant information that ioping displays is iops and the average and mdev response time per request. The individual request stats will usually be in us (microsecond) if you have really fast storage, or ms (millisecond). Please keep in mind that a microsecond(us) is 1/1,000,000 of a second, and a millisecond(ms) is 1/1000 of a second. Another way to look at it is that there are 1000us in 1ms.

1000 requests completed in 1.7 min, 109 iops, 440.0 KiB/s
min/avg/max/mdev = 190 us / 9.1 ms / 291.1 ms / 41.7 ms

In the example above, ioping is telling me that on average I have a response time of 9.1ms, with a maximum response time of 291ms, and mdev of 41.7ms, which I believe means that a majority of responses fall under this response time. This is important to take note of, if the mdev is much higher than the average then it means there are some responses that are taking significantly more time to complete than others, depending on how frequently this happens your system could experience slowness or even lock up if there is significant IO happening all the time.

How to run ioping with cached I/O

If you want to run an ioping test that uses cached I/O you can use the -C option, notice that this is different than -c which is the option that lets you determine the count, or number of times to measure performance. Cached I/O will typically perform better than using direct I/O so if you want to test out direct requests to storage, you probably do not want to use -C with ioping

./ioping -C -c 1000 .

How to run ioping with direct I/O

If you want to run an ioping test that uses direct I/O you can use the -D option. Direct I/O will typically perform slower than using cached I/O since memory is waaay faster than HDDs or even SSDs.

./ioping -D -c 1000 .

How to run ioping with async I/O

If you want to run an ioping test that uses asynchronous I/O you can use the -A option. Some Linux distros may not support async I/O and some virtual machines / VPS may not correctly implement / support this so make sure you test with and without this option in case there is a performance difference.

./ioping -A -c 1000 .

PerfKitBenchmarker

You can find out how to install, configure and use PerfKitBenchmarker by visiting this page.

How to Break In an SSD before Benchmarking

This simple script should work ok with most single SSDs / SSD RAID. The reason for doing this is because brand new SSDs will always perform extremely well initially, but over time performance will decrease slightly until the SSD has reached it's "steady state". An SSD with no data on it performs 10% - 50% faster than an SSD with 50% storage space used. This is because an empty SSD does not need to worry about things like garbage collection, or removing old blocks to make way for new data. If you run performance tests on an SSD when it's brand new, you won't get an accurate idea of what performance will look like in the real world.

So, to break the SSD in before testing you might want to run these commands a few times and continue benchmarking the SSD until the results are stable. This could take a few hours of testing to complete depending on the size of the SSD, the dd commands below will write 100GB of either zeros, or random data, remove the file, then do it over again. You could clean the script up and use a for loop, but I want to illustrate what I mean by "breaking in an ssd" by showing how many times I do this.

I suggest using something like Sysbench to test out read and writes once the script is done, record the results, and run the script again and test random read and writes, compare them to the previous set until the results are withing 5% or less of each other. If one run provides significantly different results than the previous run, then odds are the SSDs are still not at their steady state.

#!/bin/bash
dd if=/dev/random of=bigfile bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/random of=bigfile1 bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/random of=bigfile2 bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/zero of=bigfile3 bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/random of=bigfile4 bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/random of=bigfile5 bs=1M count=100000;
rm -f bigfile*;
dd if=/dev/zero of=bigfile6 bs=1M count=100000;
rm -f bigfile*;

Again, the main reason for the script above is to make sure that the SSDs you are testing have hit their "steady state performance", I do NOT use dd to measure performance because the results are almost always meaningless since sequential writes rarely happen as much as random writes (at least in the server world). If you do not break in the ssds and fill them up a few times with writes you will get higher than normal performance numbers, which is not going to tell you what the real world performance actually is. Even worse, you might start running benchmarks and wonder why performance continues to drop even as you tweak and optimize settings to improve performance. This will throw a lot of people off, so it's best to write to the SSD until performance has stabilized.

DD Write Tests

You can use this to break in new SSDs, ideally you will want to fill up the drive to about 90% remove the files, and repeat a few times to cause the garbage collection to kick in.

dd if=/dev/zero of=ddtest1 bs=1M count=8192 conv=fdatasync
dd if=/dev/zero of=ddtest2 bs=1M count=8192 conv=fdatasync
dd if=/dev/zero of=ddtest3 bs=1M count=8192 conv=fdatasync
rm -f ddtest1 ddtest2 ddtest3
dd if=/dev/zero of=ddtest1 bs=1M count=8192 conv=fdatasync
dd if=/dev/zero of=ddtest2 bs=1M count=8192 conv=fdatasync
dd if=/dev/zero of=ddtest3 bs=1M count=8192 conv=fdatasync
rm -f ddtest1 ddtest2 ddtest3

This is a test for small writes.

dd if=/dev/zero of=test bs=4K count=2000000 conv=fdatasync ; sleep 10 ; dd if=/dev/zero of=test bs=4K count=2000000 conv=fdatasync ; sleep 10 ; dd if=/dev/zero of=test bs=4K count=2000000 conv=fdatasync

Benchmarking Videos and Links

GTAC 2014: The Challenge of Fairly Comparing Cloud Providers

Liquidweb 728x90.jpg