Docker

From wiki.mikejung.biz
Jump to: navigation, search

LiquidWeb API Banner.jpg

Use Kubernetes with Docker

Docker Videos

Docker 1.0.1 MySQL OLTP Benchmarks

Now that docker has hit the "production" ready milestone I decided to re-run some performance tests to see if anything has changed in terms of overall docker performance, especially MySQL performance inside of a container. Previously Docker was considered "not production ready". That still might be the case, but I wanted to update this page with some newer Docker tests.

Docker 1.0.1 MySQL OLTP Benchmarks Test Specifics

I'm running these tests on a Storm Dedicated Server with an E3-1220V2, 8GB RAM, 2 x 240GB SSDs in Raid 1. I am using Ubuntu 14.04 for the OS. This is a virtual server that is dedicated entirely to me, so there are no other instances on the server. There will be a slight performance hit as KVM does reduce performance, but then again you don't really want to be running docker containers without some sort of isolation, at least publicly available Docker deployments.


I'm using Percona 5.6 for these tests. I am using the same installation method on the host, and inside of the docker container.

echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list                                      
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list                                  
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A                                     
apt-get update                                  
apt-get install percona-server-common-5.6 percona-server-client-5.6 percona-server-server-5.6                

I'm using a blank my.cnf configuration for the most part. I remove all of the existing performance settings that come with the default install and I am using these optimizations.

key_buffer =     64M                             
max_allowed_packet =      16M                             
innodb_buffer_pool_size = 4G                              
innodb_file_per_table                                   
innodb_log_file_size =    256MB                           
innodb_io_capacity =      10000                           
max_connections = 1000                            

I'm using Sysbench 0.5 because that is what the Percona repo wants to use. The command below is what I am using for the prepare statement.

sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --db-driver=mysql --oltp-table-size=20000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password prepare

DOCKER 1.0.1 Setup and ConfigInitially I created a single container and ran both MySQL and Sysbench inside the same container. Many will say that this is pointless and not a use case for Docker. I agree, but I still wanted to test as many different ways as possible. I created another container that just contained Sysbench, which connected to the MySQL container to create a more "real world" type of test.

docker run -i -t ubuntu:latest bash

##INSIDE CONTAINER

echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list                                   
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list                               
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
apt-get update
apt-get install percona-server-common-5.6 percona-server-client-5.6 percona-server-server-5.6 sysbench         


##Applied same MySQL opts as above

Docker 1.0.1 and Percona 5.6 OLTP Performance Results

Here are the results. There is still a small performance hit, but it doesn't seem to be massive. I used the following commands for the Mixed OLTP and Read Only OLTP tests.

for each in 1 4 8 16 32 64; do sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --oltp-table-size=20000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=240 --max-requests=0 --num-threads=$each run; done

for each in 1 4 8 16 32 64; do sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --oltp-table-size=20000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --oltp-read-only=on --max-time=240 --max-requests=0 --num-threads=$each run; done

The Blue column results are from a virtualized instance on top of KVM. This is exactly how most people currently use MySQL, in a virtualized environment. I ran Sysbench on the same host, so both MySQL and Sysbench were using the same resources.

The Orange column results were recorded from inside a Docker container on the same virtualized instances as the blue test run. I used Percona 5.6 with the same my.cnf settings as before, and I ran Sysbench inside of the container as well. I stopped MySQL on the host before running the tests inside of the container.

The Yellow column results were recorded from a second container, which just had sysbench installed in it. I then connected to a "remote" database IP, which was the IP from the first container, which has Percona 5.6 running. This simulates a more "real world" use case, however the networking performance would be a lot worse if I was connecting from a different host. Regardless, Docker didn't kill performance all that much as you can see from the results.

Docker MySQL Benchmarks.png

How to Create a Docker Sysbench Container

This container is meant for some basic benchmarking using Sysbench. For this example I am using CentOS on the host and in the container. The first step is to install Docker, and pull down the image we want to use.

wget http://mirrors.liquidweb.com/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -i epel-release-6-8.noarch.rpm
yum install docker-io
service docker start
docker pull centos:latest
docker run -i -v /home:/home -t centos /bin/bash

Inside of the container, I run the following

cd home
yum install wget
wget http://mirrors.liquidweb.com/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -i epel-release-6-8.noarch.rpm 
yum install sysbench
wget http://dev.mysql.com/get/mysql-community-release-el6-5.noarch.rpm
rpm -i mysql-community-release-el6-5.noarch.rpm
yum install mysql-server
yum install sysbench

MySQL Docker Performance Benchmarks

I decided to run some basic performance tests for MySQL to see if there was a huge difference in performance between running MySQL inside a Docker container and outside of a container, as you normally would on a VPS / Dedicated server.

MySQL Docker Benchmarking Performance Results

For this test I plan on using an 8GB StormVPS, using CentOS 6.5 for the host OS. I will be running MySQL 5.6 with some performance tweaks that can be located here:

http://wiki.mikejung.biz/index.php?title=MySQL_Optimization#my.cnf_template

The main goal is to determine how much of a performance overhead Docker adds to the application. I will be using Sysbench 0.4.12 to test MySQL performance. I will be running the client both locally (on the same vps that MySQL is running) as well as remotely, from another VPS connecting via Private Network.

MySQL Packages installed: Version: 5.6.17

wget http://dev.mysql.com/get/mysql-community-release-el6-5.noarch.rpm
rpm -ivh mysql-community-release-el6-5.noarch.rpm
yum install mysql.x86_64 mysql-community-client.x86_64 mysql-community-common.x86_64 mysql-server.x86_64

my.cnf

[mysqld]
###########################################################
###########################################################
### Begin Client Section ###
[client]
port                                    = 3306
socket                                  = /var/lib/mysql/mysql.sock
default-character-set                   = utf8
## End Client Section ###
###########################################################
###########################################################

###########################################################
###########################################################
### Begin MySQL Server Section ###
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

## General Settings ##
port                                    = 3306
tmpdir                                  = /tmp
skip_external_locking

## Performance and Capacity Tunables (These settings matter!)##
max_connections                         = 1000

## Raise to 128M for 2GB RAM, 256M for 4GB RAM and 512M for 8GB RAM
key_buffer                              = 64M

## Raise to 128M for 2GB RAM, 256M for 4GB RAM and 512M for 8GB RAM
innodb_buffer_pool_size                 = 4096M

## Misc Tunables (Don't touch these unless you know why you would want to touch these)##
max_allowed_packet                      = 16M
table_open_cache                        = 1024
myisam_sort_buffer_size                 = 32M
innodb_additional_mem_pool_size         = 8M
#innodb_flush_log_at_trx_commit          = 1
innodb_log_buffer_size                  = 8M
innodb_file_per_table

## Changing this setting requires you to stop MySQL, move the current logs out of the way, and then starting MySQL ##
innodb_log_file_size                    = 128M

Sysbench Database Info

sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=root --mysql-password=password prepare

DOCKER Setup and Config Using Docker Version docker-io-0.9.0-3.el6.x86_64

docker pull centos 
docker run -i -t centos /bin/bash

##INSIDE CONTAINER

yum update (updated to cent 6.5)
wget http://dev.mysql.com/get/mysql-community-release-el6-5.noarch.rpm
rpm -ivh mysql-community-release-el6-5.noarch.rpm
yum install mysql.x86_64 mysql-community-client.x86_64 mysql-community-common.x86_64 mysql-server.x86_64

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
yum install sysbench

##Applied same MySQL opts as above

MySQL Docker OLTP Performance Results

I Ran 5 different configurations to try and test all of the possible combinations for connecting to MySQL. The first two results are run on an 8GB Storm VPS. Sysbench was run on the same host that the MySQL DB was on for the first test (L). I created another 8GB VPS and connected both VPS via private Network for the second test. Sysbench was ran on the 2nd VPS and connected to MySQL on the first VPS (R).

For the docker tests, I used the same VPS. I created a docker container using the CentOS image from the Docker Index. Info on the container is listed above. The first test involved running Sysbench and MySQL inside the same container (LI). The second test involved running MySQL inside the container, but running Sysbench from outside the container, but on the same VPS (LO). The last test involved running MySQL inside the container on the first VPS, then running Sysbench on the second VPS, connecting remotely (RO).

  • 8GB VPS Sysbench run locally (L)
  • 8GB VPS Sysbench run remotely on another 8GB VPS (R)
  • 8GB VPS + Docker Sysbench run inside same container (LI)
  • 8GB VPS + Docker Sysbench run outside container, same host (LO)
  • 8GB VPS + Docker Sysbench run from remote host (RO)

I used the following command to test performance. I ran each test 5 times and averaged the results accross 5 runs. I used 1,4,8,16 threads, again, each time I recorded the results from 5 runs. I also used --oltp-read-only for the last read only test.

sysbench --test=oltp --oltp-table-size=10000000 --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --max-time=240 --max-requests=0 --num-threads=1 run
Config R/W 1 Thread R/W 4 Thread R/W 8 Thread R/W 16 Thread ReadOnly 16 Thread
8GB VPS Sysbench (L) 7,355 18,320 22,114 24,878 33,277
8GB VPS Sysbench (R) 1,958 7,064 11,225 15,414 17,807
8GB Docker Sysbench (LI) 5,887 15,932 17,826 21,343 31,392
8GB Docker Sysbench (LO) 3,670 7,170 9,190 11,168 13,312
8GB Docker Sysbench (RO) 1,774 5,494 8,517 10,041 15,514

Docker Local Performance Difference

1 thread: 25% performance overhead
4 Thread: 15% performance overhead
8 Thread: 24% performance overhead
16 Thread: 17% performance overhead
16 RO: 6% performance overhead

Docker Remote Performance Difference

1 Thread: 10% performance overhead
4 Thread: 28% performance overhead
8 Thread: 32% performance overhead
16 Thread: 53% performance overhead
16 RO: 15% performance overhead


As you can see, there is a pretty large performance overhead with MySQL. This could just be due to the fact that I am already running MySQL on a VPS, however Docker clearly does reduce performance by a notable amount. This is for the 0.9 release, so it will be interesting to see how performance changes down the road.

Common Docker Commands

Inspect Multiple Docker Containers at once

This will parse all running container IDs, then inspect all of them at once.

for each in `docker ps | awk {'print $1'} | grep -v CONTAINER`; do docker inspect $each; done

Find Multiple Docker Container IPs at once

This will parse all running container IDs, then inspect all of them at once, then only display the IPs of the containers.

for each in `docker ps | awk {'print $1'} | grep -v CONTAINER`; do docker inspect $each; done | grep "IAddress"

Find Multiple Docker Container Names at once

This will parse all running container IDs, then inspect all of them at once, then only display the IPs of the containers.

for each in `docker ps | awk {'print $1'} | grep -v CONTAINER`; do docker inspect $each; done | grep "Name"


Create Multiple Docker Containers at once

Below is an example of a simple for loop that will create 3 Docker containers that are running Percona 5.6. Replace "galera_main:latest bash" with the build you want to use, and replace "bash" with whatever you want the containers to actually run. Obviously in this case we would probably want MySQL to run, however if you still need to do some fine tuning and want to mess around in the container, using bash is the best option.

for n in {1..3}; do docker run --name dockerpxc$n -i -t -d galera_main:latest bash; done

Stop a specific docker container

Simply find the running container ID using docker ps, then replace "$ID" with the container ID and the container will stop.

docker stop $ID

Exit Running Docker Container Without Killing It

Simply press Ctrl+p then press Ctrl+q. At this point you should be out of the container.

List all running docker containers

Pretty simple, just run docker ps to get a list of currently running containers.

docker ps

Commit a running docker container

If you want to save the current state of a container you can use the commit command to save the container, so that you can roll back later on if you want to, or you can use the committed container to build off of.

docker commit $container_id $name_to_commit_as

Install Docker on CentOS 7

Docker is included in the default CentOS repos! (yay) Simply yum install to get it up and running

yum install docker


Creating a 3 node Percona Xtradb Cluster with Docker containers

This is based off of a slightly modified version of the guide above. I ran into a few issues when following that guide, so I am posting a different method here. I'm using Ubuntu 14.04 instead of 12.04 because new and shiny things are always better.

Build a Ubuntu 14.04 Docker Container

You could just use a docker file for all of this, but for now I am covering the manual steps for people who have not really used docker before. This command will pull down the latest version of Ubuntu and will start bash. NOTE Once this command finishes you will be placed inside of the container.

docker run -i -t ubuntu:latest bash

Now that we are in the container, we are going to run updates just because. We are also going to add Percona repos, grab a key and then install the stuff we need to Percona.

apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y
echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
apt-get update
apt-get install -y percona-xtradb-cluster-56 qpress xtrabackup
apt-get install -y python-software-properties vim wget curl netcat

Percona Cluster my.cnf Template

After we get everything installed, we need to replace / update the my.cnf. Use the example listed below. I've already entered in some IPs, read below to see what needs to be replaced for your cluster. It is ok if you don't know the other two container IPs yet, so if you have not created more than one container you still want to paste in the contents of this file, we will update it later on.

[client]
port		= 3306
socket		= /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
user = mysql
default_storage_engine = InnoDB
basedir = /usr
datadir = /var/lib/mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
innodb_autoinc_lock_mode = 2
log_queries_not_using_indexes = 1
max_allowed_packet = 128M
binlog_format = ROW
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_node_address = 172.17.0.5
wsrep_cluster_name="mydockerpxc"
wsrep_cluster_address=gcomm://172.17.0.4,172.17.0.5,172.17.0.6
wsrep_node_name = dockerpxc2
wsrep_slave_threads = 4
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = pxcuser:pxcpass
 
[sst]
streamfmt = xbstream
 
[xtrabackup]
compress
compact
parallel = 2
compress_threads = 2
rebuild_threads = 2 

[mysqldump]
quick
quote-names
max_allowed_packet	= 16M

[mysql]
#no-auto-rehash	# faster start of mysql but no tab completition

[isamchk]
key_buffer		= 16M

!includedir /etc/mysql/conf.d/
  • wsrep_node_address = $This_Nodes_IP
  • wsrep_cluster_name="$Whatever_you_want_to_call_the_cluster"
  • wsrep_cluster_address=gcomm://$Node1_IP,$Node2_IP,$Node3_IP
  • wsrep_node_name = $Whatever_you_want_to_call_the_node
  • wsrep_sst_auth = $pxcuser:$pxcpass


  • Once the file is in place and saved, we are going to exit the container, and commit the image. To exit the container without killing it off, do the following Ctrl+p then Ctrl+q.

Exit running container and commit first docker container

You should be back on the host at this time. Make sure the container is still running

docker ps

There should be an ID listed for the container. We are going to commit it so we can start other containers based off this image.

docker commit $container_ID percona_cluster_orig

Just for fun, and to make sure we are starting off on with a clean state, stop the current container, then start up 3 new containers all with the previously committed container.

docker stop $container_ID

Start 3 Percona Cluster docker containers and modify mysql my.cnf

Start 3 containers

for n in {1..3}; do docker run --name dockerpxc$n -i -t -d percona_cluster_orig:latest bash; done

Make sure all 3 containers are up

docker ps

If they are up, lets get their IPs so we can modify the my.cnf in each container.

##List names

for each in `docker ps | awk {'print $1'} | grep -v CONTAINER`; do docker inspect $each; done | grep "Name"

##List ips

for each in `docker ps | awk {'print $1'} | grep -v CONTAINER`; do docker inspect $each; done | grep "IPAddress"

At this point you can re-enter each container, and update the my.cnf file with the IPs listed. Enter each container, update the file, then exit using Ctrl+p then Ctrl+q. You might as well commit each container at this point. Usually it's better to commit too often than not at all.

##modify first my.cnf
docker attach dockerpxc1
docker commit dockerpxc1 dockerpxc1

##modify second my.cnf
docker attach dockerpxc2
docker commit dockerpxc2 dockerpxc2

##modify third my.cnf
docker attach dockerpxc3
docker commit dockerpxc3 dockerpxc3

Bootstrap First Percona Cluster Node

On the first node Attach to the container, and then run the bootstrap for mysql cluster

/etc/init.d/mysql bootstrap-pxc

On the first node Once the server has started, we need to make a user for replication, we should also make sure the other two containers can connect. Replace the IPs used below with whatever the other two containers are using.

mysql -p

CREATE USER 'pxcuser'@'localhost' IDENTIFIED BY 'pxcpass';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'pxcuser'@'localhost';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'pxcuser'@'172.17.0.5';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'pxcuser'@'172.17.0.6';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'pxcuser'@'$IP_N...';
FLUSH PRIVILEGES;

Start Up All Percona Cluster Nodes

On the other two nodes Attach to the container, and start up MySQL in each container. If you get any failed starts, make sure the my.cnf file is correct on all 3 nodes (sometimes paste fails cause a lot of headaches). If my.cnf looks ok, you may want to apt-get install telnet on each container, then attempt to telnet to each container to make sure you can actually connect.

service mysql start

On the first node If the other two containers started up correctly, and the config files were edited correctly, you should see a cluster size of 3 on the "master node" or in this case the "original" node.

mysql
show status like 'wsrep%';

## Important values to look out for are:


| wsrep_incoming_addresses     | 172.17.0.6:3306,172.17.0.4:3306,172.17.0.5:3306 |
| wsrep_cluster_size           | 3                                               |
| wsrep_cluster_status         | Primary                                         |
| wsrep_connected              | ON                                              |
  • You should now be able to commit all the containers again, just to finalize the configs in case we need to stop / start / remove the containers and bring them up later.

Test All Docker Percona Cluster Nodes

At this point you should be able to create a database on any one of the three containers, and you should then be able to see the database on all the other nodes

create database sysbench;

Then on another node

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sysbench           |
| test               |
+--------------------+
5 rows in set (0.00 sec)

Use Kubernetes with Docker