Varnish
Contents
- 1 Varnish Overview
- 2 Varnish 4.0 Configuration Files
- 3 Varnish Storage Backends
- 4 Varnish 3 Configuration
- 5 Varnish 4.0 Configuration
- 6 Varnish backend Misc section
- 7 default.vcl
- 8 Varnish Commands
- 9 Install Varnish 4.0.3 on CentOS 7
- 10 Install Varnish 3.0.5 On CentOS 6.5
- 11 Install Varnish 4.0.2 On CentOS 6.5
- 12 Upgrade Varnish 3.0.5 to Varnish 4.0.2 CentOS 6.5
- 13 Troubleshooting
- 14 Benchmarks
- 15 How to Configure Varnish for MediaWiki
- 16 Varnish req.http.X-Forwarded-For Guide with Apache Example
Varnish Overview
Varnish is a "web accelerator" that acts as an HTTP reverse proxy, basically Varnish is the front of the line for serving requests. Apache sits behind Varnish and serves requests only to Varnish if the content is not already in the Varnish cache, or cannot be cached for whatever reason. Images, css, html, and a lot of other stuff can be cached by Varnish which means Apache does less work, which generally means a happier server and a fast website.
Any time you access something from RAM the access time for that file becomes faster, since Varnish can store a large amount of files in RAM, it can significantly improve performance. I've seen performance boosts of 10x to 100x simply by sticking Varnish in front of WordPress. There is a VCL language that you can use to set rules about what to do with a certain type of request, what files to cache, what files not to cache, and for how long to cache said file.
If you like what you see so far, then continue on reading to find out how to make the cloud faster, or something.
Installing Varnish is a rather simple process. Configuration can sometimes be a little more tricky, especially with the default.vcl file. While you can create some very complex VCL rules and complex configurations, you can also just leave the default.vcl alone and run with the defaults, it may not be optimal, but it'll still probably be faster than just sticking with Apache!
If you aren't already using something like Php-fpm or HHVM for PHP, you may still notice slow loading times. Varnish can only help so much if your database server is slow or you are using something like SuPHP to run PHP.
Varnish can also be used to load balance HTTP requests between multiple backend servers. Varnish is obviously RAM hungry, the whole point of caching is to use as much RAM as possible so you should give Varnish enough RAM to fit most of your www/ content into it's cache. Varnish is not very CPU intesive unless you are getting large amounts of requests. Varnish running with a single core on a VPS can handle at least 100 request/s if not much more.
Varnish 4.0 Configuration Files
Ubuntu and Debian Varnish Server Config
On most Debian and Ubuntu servers you can find the main configuration file for varnish in
vim /etc/default/varnish
CentOS 6 and 7 Varnish Server Config
On most CentOS 6.x servers you can find the main configuration file for varnish in
/etc/sysconfig/varnish
For CentOS 7 the file is located in
/etc/varnish/varnish.params
Varnish 4.0.3 CentOS 7 Main Configuration Options
If you want to have Varnish use malloc storage you can edit /etc/varnish/varnish.params and change the storage option. In this example I am using 1GB of memory and the malloc storage option, by default Varnish will use the file backend by default.
RELOAD_VCL=1 VARNISH_VCL_CONF=/etc/varnish/default.vcl VARNISH_LISTEN_ADDRESS=$IP_Varnish_Should_Listen_On VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 VARNISH_SECRET_FILE=/etc/varnish/secret VARNISH_STORAGE="malloc,1G" VARNISH_TTL=120 VARNISH_USER=varnish VARNISH_GROUP=varnish VARNISH_STORAGE="malloc,512m"
Varnish Storage Backends
malloc
Varnish will request the entire size of the cache with a malloc call. Every object stored by Varnish will be stored in RAM. Keep in mind that if you do not limit RAM usage Varnish might use up all available RAM, forcing the OS to start to swap, at this point performance may degrade. Be sure to size Varnish Malloc allocation accordingly.
Each object that is stored in Varnish will consume about 1KB of RAM, regardless of the size. So many small files may start to bloat memory usage in a hurry. It's best to set a limit when using Malloc.
Malloc is the fastest option to use with Varnish
You can configure Varnish to use Malloc as backend storage by modifying /etc/sysconfig/varnish. You can either define the variable, and pass to the DAEMON_OPTS, or just define DAEMON_OPTS
vim /etc/sysconfig/varnish
VARNISH_STORAGE="malloc,100M" *OR* DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,100M \ -u www-data \ -g www-data"
Then restart Varnish
service varnish restart
file
Varnish creates a file on the filesystem to contain the entire cache. It then tells the OS via mmap() to map the entire file into memory if possible. Keep in mind that if you are still using spinning disks, you may see lower performance than using malloc due to write latency. If you are using SSDs, or have a large RAID that can handle some extra IO, the File option may be a good option. You can modify the file location to whatever you want, so if you have a spare disk, create the file and update VARNISH_STORAGE_FILE with the full path to the file.
- Choose file if you have a large cache that will not fit entirely into RAM AND have fast disks (ssds)
You can configure Varnish to use File as backend storage by modifying /etc/sysconfig/varnish. You can either define the variable, and pass to the DAEMON_OPTS, or just define DAEMON_OPTS
vim /etc/sysconfig/varnish
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin VARNISH_STORAGE="file,100M,${VARNISH_STORAGE_FILE}" *OR* DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s file,100M,/var/lib/varnish/varnish_storage.bin \ -u www-data \ -g www-data"
Then restart Varnish
service varnish restart
Shared memory log -- Not much needs to be done with this besides making sure the log is stored in RAM (/dev/shm)
Varnish 3 Configuration
VARNISH_MIN_THREADS
For Varnish 3.0 the default value is 2 (according to man page). If you have a very busy server, you may want to raise this value. Keep in mind that this is only the minimum value, and really only matters if you get frequent bursts of traffic. This can be found and configured in /etc/sysconfig/varnish
VARNISH_MIN_THREADS=2
VARNISH_MAX_THREADS
For Varnish 3.0 the default value is 500 (according to man page). If you have a very busy server, you may want to raise this value. Raising this to very high levels may cause some issues, so you want to watch the server after you change this value to make sure varnish doesn't go all crazy on you. This can be found and configured in /etc/sysconfig/varnish
VARNISH_MAX_THREADS=500
VARNISH_THREAD_TIMEOUT
For Varnish 3.0 the default value is 300 seconds (according to man page). If you prefer to kill your threads while they are young, you can lower the value. If you want your threads to have a long, fulfilling life and experience all the static files they can, raise this value. This can be found and configured in /etc/sysconfig/varnish
VARNISH_THREAD_TIMEOUT=300
VARNISH_TTL
For Varnish 3.0 the default value is 120 seconds (according to man page). This is the default TTL that is assigned to new objects that no one cared about, or specified a TTL for. This is a minimum, hard value to live in the cache. If you want to clear out the cache more frequently, and don't like to specify ttls elsewhere, then lowering or raising this value is your best bet. This can be found and configured in /etc/sysconfig/varnish
VARNISH_TTL=120
Varnish 4.0 Configuration
thread_pools
By default there are 2 thread pools. The minimum value is 1. This setting sets the amount of worker pools that varnish will use. You can increase this value on the fly without a restart, however you must restart the varnish server if you decrease this value. Technically this is still flagged as an experimental setting, so be careful if you decide to raise the value.
If you have many CPUs, then raising this value to the amount of CPUs probably makes sense because the more thread pools there are, the less locking / contention Varnish has to deal with. Raising thread_pools too high, or well about the amount of CPUs is not a good idea and could end up hurting performance and wasting resources. Increasing this to a sane value will reduce lock contention and therefore performance.
thread_pools=2
thread_pool_max
The Varnish thread_pool_max value is the maximum number of workers per pool. The default value is 5000. The minimum value is 100. This defines the number of worker threads per pool. You may not want to raise this value at all, as more threads can cause threads to start to step on each others toes. Do you really need more than 5000 threads? Do you?
thread_pool_max=5000
thread_pool_min
The Varnish thread_pool_min value is the minimum number of workers per pool. The default value is 100 and the maximum value is 5000. This setting works with thread_pool_max. You cannot set the minimum value to be higher than the max value. The default setting is probably fine in most cases, and you may find better performance by creating more pools with less workers, or vice versa. Varnish is already pretty damn fast.
thread_pool_min=100
Varnish backend Misc section
To configure Varnish using the file backend, do nothing, it's the default. However, you can change the size of RAM to use by changing the "1GB" value to whatever is appropriate.
vim /etc/sysconfig/varnish # # Cache file location VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin # # # Cache file size: in bytes, optionally using k / M / G / T suffix, # # or in percentage of available disk space using the % suffix. VARNISH_STORAGE_SIZE=1GB # # # Backend storage specification VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
To configure Varnish to use the malloc backend, you will need to comment out the original VARNISH_STORAGE setting, create a new one, specify "malloc,$size_of_cache"
# # Backend storage specification #VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" VARNISH_STORAGE="malloc,256MB"
default.vcl
- This contains your VCL and backend-definitions. After changing this, you can run either service varnish reload, which will not restart Varnish, or you can run service varnish restart, which empties the cache.
/etc/varnish/default.vcl
backend default { .host = "localhost"; .port = "8080"; }
- A backend server is the server providing the content Varnish will accelerate.
.host = The host that varnish will talk to get content. If this is installed on the same server that Apache is using, this would be localhost or 127.0.0.1. If the Apache server is on another server, you would put in the IP of that server, private network is preferable, but a public IP will work as well.
.port = The port that Varnish will connect to to get content to cache. If Apache is listening on port 80, then you would configure this to listen on port 80. If Apache is listening on port 8080, then you would set this to port 8080.
Varnish Commands
Restart Varnish
This will restart varnish, obviously. If you prefer a more graceful approach and just want to update VCL without clearing cache, try the reload command.
service varnish restart
Reload Varnish
Reloads the vcl file, the cache is not affected, which is obviously preferred over a full restart.
service varnish reload
Varnishstat
Shows a TON of useful information such as cache hits, misses, requests, backend connections and lots of other info.
varnishstat
Varnish Command Line Options
Command line options.
Listen address -a <[hostname]:port> Specifies the vcl file location -f <filename> Set the tunable parameters -p <parameter=value> Authentication secret for management -S <secretfile> Management interface -T <hostname:port> Where and how to store objects -s <storagetype,options>
Install Varnish 4.0.3 on CentOS 7
Installing Varnish 4.0.3 on CentOS 7 can be tricky since the repos don't always have up to date packages. You can grab all 3 Varnish rpms from Varnish's el7 repo and manually install them. I've found that as of Feb 18th 2015 this is the easiest way to install the latest version of Varnish.
yum install gcc wget https://dl.fedoraproject.org/pub/epel/6/x86_64/jemalloc-3.6.0-1.el6.x86_64.rpm wget https://repo.varnish-cache.org/redhat/varnish-4.0/el7/x86_64/varnish/varnish-4.0.3-1.el7.centos.x86_64.rpm wget https://repo.varnish-cache.org/redhat/varnish-4.0/el7/x86_64/varnish/varnish-libs-4.0.3-1.el7.centos.x86_64.rpm wget https://repo.varnish-cache.org/redhat/varnish-4.0/el7/x86_64/varnish/varnish-libs-devel-4.0.3-1.el7.centos.x86_64.rpm rpm -iv jemalloc-3.6.0-1.el6.x86_64.rpm rpm -iv varn*.rpm
Over time, the versions will be updated, so please make sure you check that the versions are the same by visiting this link - https://repo.varnish-cache.org/redhat/varnish-4.0/el7/x86_64/varnish/
If there are newer packages than what I have, simply replace the wget commands with the newer packages and the installation process should remain the same.
Install Varnish 3.0.5 On CentOS 6.5
Download and Install the official Repo from Varnish. The repo below is for Varnish 3, if you want to use varnish 4 please use the next repo down this page.
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-3.0.el6.rpm
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
Install Varnish
yum install varnish
Enable Varnish on Reboot
chkconfig varnish on
Change Apache Listen Port to 8080
vim /etc/httpd/conf/httpd.conf Listen 80 ##change to Listen 8080
Configure Varnish
vim /etc/sysconfig/varnish VARNISH_LISTEN_PORT=6081 ##change to VARNISH_LISTEN_ADDRESS=$IP_you_want_varnish_to_listen_on VARNISH_LISTEN_PORT=80
Configure Varnish Backend
vim /etc/varnish/default.vcl backend default { .host = "127.0.0.1"; .port = "80"; } ##change to backend default { .host = "127.0.0.1"; .port = "8080"; }
Restart Apache and Varnish to enable changes made to configuration
service httpd start service varnish start
Make sure Varnish and Apache are listening on the correct ports
netstat -plan | grep 80
Should look like:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1868/varnishd tcp 0 0 :::80 :::* LISTEN 1868/varnishd tcp 0 0 :::8080 :::* LISTEN 1825/httpd
Create a test file to make sure Varnish is working correctly
touch /var/www/html/test.txt echo "test" > /var/www/html/test.txt
- You should be able to view the file in your browser:
- You should also be able to verify that Varnish is serving the file:
curl -I http://$IP/test.txt HTTP/1.1 200 OK Server: Apache/2.2.15 (CentOS) Last-Modified: Tue, 01 Jul 2014 16:33:52 GMT ETag: "41cf0-5-4fd2457515364" Content-Type: text/plain; charset=UTF-8 Content-Length: 5 Accept-Ranges: bytes Date: Tue, 01 Jul 2014 16:36:35 GMT X-Varnish: 677655568 677655567 Age: 53 Via: 1.1 varnish Connection: keep-alive
Install Varnish 4.0.2 On CentOS 6.5
Download and Install the Repo from Varnish
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
Grab jemalloc since CentOS includes nothing useful by default for anything ever....
wget https://dl.fedoraproject.org/pub/epel/6/x86_64/jemalloc-3.6.0-1.el6.x86_64.rpm rpm -i jemalloc-3.6.0-1.el6.x86_64.rpm
Install Varnish
yum install varnish
Enable Varnish on Reboot
chkconfig varnish on
Change Apache Listen Port to 8080
vim /etc/httpd/conf/httpd.conf Listen 80 ##change to Listen 8080
Configure Varnish
vim /etc/sysconfig/varnish VARNISH_LISTEN_PORT=6081 ##change to VARNISH_LISTEN_PORT=80
Configure Varnish Backend
vim /etc/varnish/default.vcl backend default { .host = "127.0.0.1"; .port = "80"; } ##change to backend default { .host = "127.0.0.1"; .port = "8080"; }
Restart Apache and Varnish to enable changes made to configuration
service httpd start service varnish start
Make sure Varnish and Apache are listening on the correct ports
netstat -plan | grep 80
Should look like:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1868/varnishd tcp 0 0 :::80 :::* LISTEN 1868/varnishd tcp 0 0 :::8080 :::* LISTEN 1825/httpd
Create a test file to make sure Varnish is working correctly
touch /var/www/html/test.txt echo "test" > /var/www/html/test.txt
- You should be able to view the file in your browser:
- You should also be able to verify that Varnish is serving the file:
curl -I http://$IP/test.txt HTTP/1.1 200 OK Server: Apache/2.2.15 (CentOS) Last-Modified: Tue, 01 Jul 2014 16:33:52 GMT ETag: "41cf0-5-4fd2457515364" Content-Type: text/plain; charset=UTF-8 Content-Length: 5 Accept-Ranges: bytes Date: Tue, 01 Jul 2014 16:36:35 GMT X-Varnish: 677655568 677655567 Age: 53 Via: 1.1 varnish Connection: keep-alive
Upgrade Varnish 3.0.5 to Varnish 4.0.2 CentOS 6.5
This is not exactly seamless, or at least as simple as it sounds. All you should really have to do is grab the new repo, install, remove varnish 3, install varnish 4, however if you don't have a few things in place your site is going to be down.
Install jemalloc because CentOS never contains anything useful in its repos. Luckily this is easy once you head over to Fedora land. Install this sucker because Varnish 4 needs it. Varnish 3 didn't require this, so you just try and do a quick upgrade, you are going to get pissed without this.
wget https://dl.fedoraproject.org/pub/epel/6/x86_64/jemalloc-3.6.0-1.el6.x86_64.rpm rpm -i jemalloc-3.6.0-1.el6.x86_64.rpm
Backup existing config files. A backup is made when you remove Varnish 3, but do it anyway! Varnish 4 is going to get installed with default configs, so it has no idea what the backends are, or were, and all the VCL is going to be gone, so having the original configs to fall back on is nice.
cp /etc/sysconfig/varnish /etc/sysconfig/varnish.3bak cp /etc/varnish/default.vcl /etc/varnish/default.vcl.3back
Remove Varnish. If you don't want any downtime, you should probably update Apache or Nginx to listen on port 80, stop Varnish, restart the web server, then remove varnish. Or if you don't care about downtime, and want to do it live, just remove varnish.
yum remove varnish ##Check for any old repos or misc varnish things, and remove them as well rpm -qa | grep var
Once Varnish 3 is removed, grab the varnish 4 repo and install it
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
Now install Varnish 4
yum install varnish
At this point, you can start up varnish. Replace the new Varnish config files with the old ones and you should be back up and running. VCL changes will probably cause varnish to fail if you have a ton of crazy vcl, so you will want to test it out before you do this live.
Troubleshooting
nf_conntrack
If you notice some odd issues with Varnish, check dmesg, if you see this you should raise the limit.
# dmesg nf_conntrack: table full, dropping packet. nf_conntrack: table full, dropping packet. nf_conntrack: table full, dropping packet.
To raise the limit you can modify the sysctl.conf file as follows
#See what the current limit is and note it: cat /proc/sys/net/nf_conntrack_max vim /etc/sysctl.conf #Add the following net.nf_conntrack_max = 100000 #Once that is added, run this to make the change permanent. sysctl -p
Benchmarks
Setup
- Running on CentOS 6
- Apache 2.2 is being used behind Varnish on the same server.
- Varnish 3.0 using the following settings: VARNISH_STORAGE="malloc,256MB"
- This is on a 1GB instance
Test files:
http:///index.html (static, basic html page) http:///index.php (phpinfo page) http:///perl.php (grabs perl version)
Varnish
10,000 requests, 10 concurrency
ab -n 10000 -c 10 http:///index.html Document Length: 169 bytes Time taken for tests: 1.029 seconds Requests per second: 9716.42 [#/sec] (mean) Transfer rate: 4725.37 [Kbytes/sec] received ab -n 10000 -c 10 http:///index.php Document Length: 44303 bytes Time taken for tests: 1.202 seconds Requests per second: 8316.46 [#/sec] (mean) Transfer rate: 362058.58 [Kbytes/sec] received ab -n 10000 -c 10 http:///perl.php Document Length: 237 bytes Time taken for tests: 1.033 seconds Requests per second: 9678.93 [#/sec] (mean) Transfer rate: 4839.45 [Kbytes/sec] received
50,000 requests, 20 concurrency
ab -n 50000 -c 20 http:///index.html Document Length: 169 bytes Time taken for tests: 5.176 seconds Requests per second: 9659.51 [#/sec] (mean) Transfer rate: 4688.26 [Kbytes/sec] received ab -n 50000 -c 20 http:///index.php Document Length: 44303 bytes Time taken for tests: 6.064 seconds Requests per second: 8245.66 [#/sec] (mean) Transfer rate: 358976.27 [Kbytes/sec] received ab -n 50000 -c 20 http:///perl.php Document Length: 237 bytes Time taken for tests: 5.190 seconds Requests per second: 9633.36 [#/sec] (mean) Transfer rate: 4816.68 [Kbytes/sec] received
Apache
10,000 requests, 10 concurrency
ab -n 10000 -c 10 http:///index.html Document Length: 169 bytes Time taken for tests: 1.416 seconds Requests per second: 7063.33 [#/sec] (mean) Transfer rate: 3015.24 [Kbytes/sec] received ab -n 10000 -c 10 http:///index.php Document Length: 43540 bytes Time taken for tests: 6.479 seconds Requests per second: 1543.56 [#/sec] (mean) Transfer rate: 65943.43 [Kbytes/sec] received ab -n 10000 -c 10 http:///perl.php Document Length: 237 bytes Time taken for tests: 1.792 seconds Requests per second: 5578.88 [#/sec] (mean) Transfer rate: 2344.57 [Kbytes/sec] received
50,000 requests, 20 concurrency
ab -n 50000 -c 20 http:///index.html Document Length: 169 bytes Time taken for tests: 7.119 seconds Requests per second: 7023.53 [#/sec] (mean) Transfer rate: 2997.59 [Kbytes/sec] received ab -n 50000 -c 20 http:///index.php Document Length: 43540 bytes Time taken for tests: 34.014 seconds Requests per second: 1470.00 [#/sec] (mean) Transfer rate: 62750.60 [Kbytes/sec] received ab -n 50000 -c 20 http:///perl.php Document Length: 237 bytes Time taken for tests: 9.256 seconds Requests per second: 5402.16 [#/sec] (mean) Transfer rate: 2268.62 [Kbytes/sec] received
How to Configure Varnish for MediaWiki
Visit this link for a Varnish 4 VCL Example MediaWiki Example VCL
Varnish req.http.X-Forwarded-For Guide with Apache Example
Please visit the link below to view the page on how to configure Varnish to pass along IP header info to Apache. This will make it so that the actual IP is logged and not 127.0.0.1