From wiki.mikejung.biz
Jump to: navigation, search

Liquidweb high performance banner.gif

Quick Primer on Website Performance

Website performance has almost nothing to do with Apache. Apache is only a small factor when it comes to loading a webpage in your browser. Most of the server side processing time is spent in the PHP and MySQL world, and even then, the amount of time spent serving the request is a fraction of the time spent in the web browser. Often times the best performance tweaks involve using something like mod_expires to enforce Browser Caching for static files such as images, videos, or css. Making sure that you are running the latest stable version of Apache, which at this time is version [2.4.23] can help significantly in terms of improving performance and stability.

To prove this point, take a look at this example website. The chart was generated by NewRelic's free service which I have installed on my own VPS. As you can see, most of the time my server takes under 100ms, or 0.100 seconds to generate the response. During the idle times of the day it takes PHP 68ms to complete, and it takes MySQL about 10ms to resond. You will notice that there is not a section for how long it took Apache to respond. That's because Apache would be a tiny slice of the PHP processing time which is 68ms. Even if you optimize Apache to be 10x faster you would only be shaving off 1ms or maybe 2ms from the total response time of 100ms. The wiki in this example is pretty much as optimized as you can get, but there is still what appears to be a "load spike". This load spike was caused mainly by MySQL during a busy time of day.

Since PHP communicates with MySQL, the speed at which PHP can complete a request is dependent on MySQL's speed. Nowhere in this equation is Apache. Because Apache doesn't really matter when it comes to performance.

Wiki app response time.png

So, using this example, we have established that it takes under 100ms for a 1GB server to generate a response to send back to the client that is trying to load a page on my wiki. That includes the time it took for Apache to handle the HTTP request, pass PHP to FCGI which had to speak with MySQL about things, then heads back over to Apache to get sent back to the browser.

Now that we have an idea about how long it takes a server to generate a response back, let's take a look at how long it takes that page to load in a browser.

Wiki page load time.png

As you can see, the time it takes to load a page can vary wildly. In general though a page will FULLY LOAD in about 3 seconds. So much for the fast response time on my server! I must be able to optimize Apache to make it faster, right? WRONG. Even if you could make Apache, PHP and MySQL all respond in 1ms, instead of 100ms, you only shave off 1/10 of a second from the page load time since all the other time is spent off the server.

Almost all of the time it takes to load a website is spent traveling across the network, and rendering inside the client browser, rendering the page. What about the server though, maybe there are load spikes that are causing higher than normal page loading time? Not really, for the most part the load average is well under 0.30.

So what's the point of all this? The point is that there are many factors when it comes to website response times, but generally speaking the server side processing is not a huge factor for website performance. Even then, PHP and MySQL take much more time than Apache does, so don't waste too much time on Apache!

Apache Tuning Overview

Let's make one thing clear, Apache does not really need to be optimized, in fact it's pretty hard to make Apache faster than it already is. Apache handles HTTP requests made by clients (typically a browser) and responds back with the requested resources. That's really all Apache does. The amount of time it takes Apache to fetch resources, or pass along requests to other services is very small. Even if you could make Apache 100% faster, you would only be shaving a few milliseconds off each request.

Generally speaking the best way to optimize Apache for performance is to:

1) Give Apache less work to do by using FCGI or PHP-FPM. This offloads PHP processing and frees up Apache significantly. If the server is using DSO or SuPHP the first step would be to suggest using FCGI. DSO and SuPHP are both tried and true handlers, however this does not mean they are the best. Now that FCGI and PHP-FPM are around you will see significant performance improvements as well as reduced resouce usage by using FCGI or PHP-FPM. Since PHP-FPM is not currently supported by cPanel, try out FCGI first since it's fully supported by cPanel.

2) Make sure that the software is up to date, as of 2014 Apache 2.4 is the latest stable version. The MPM used by Apache can have some performance impact as well. Typically Event is the best MPM to use unless there is some compatibility issue and you need to use Prefork.

tl;dr Using FCGI with Apache 2.4 with the Event MPM, and FCGI as the PHP handler is about as optimized as Apache gets (at least for those of you with cPanel servers). The best optimization for Apache is to offload PHP handling to either FCGI or PHP-FPM. If Apache is older than version 2.4, or it's using SuPHP as a handler then Apache IS NOT OPTIMIZED and will never be optimized.

The only Apache directives that really matter are MaxRequestWorkers, MaxClients(same as MaxRequestWorkers, but for old versions of apache). Apache Event uses very small amounts of RAM when the server is idle and Apache is very good at self regulation so it'll kill off idle processes if needed and create more processes when needed. If the server is using Apache 2.4 with Event you don't need to worry about optimizing Apache in terms of performance, the focus should be on setting resource limits based on the amount of RAM that the server has.

Apache uses very little resources because, in general, it's only acting as a proxy between a browser and the rest of your application, which usually includes PHP code, which generally talks to MySQL. The average Event process uses about 14MB of RAM, typically there are between 3 and 6 Event processes running on an Idle server, this comes to a total of about 60MB of RAM. Even on a busy 1GB server, Apache will use around 100MB, or about 10% of the servers total RAM.

The average PHP process uses anywhere from 32MB to 128MB of RAM. Often times a single PHP process uses more RAM than all Apache processes combined. The only time this is not true is if Apache is running with mod_php, since Apache runs PHP inside of the Apache process. That type of setup is not recommended as it can quickly utilize RAM and cause the server to go OOM if configured incorrectly. Because of that, we are going to focus on Apache Event and FCGI (or PHP-FPM). If the server isn't using Event and FCGI then it's not optimized, if the server is using Event and FCGI it is optimized. That's really all there is to it.

Front and Back End Tuning Basics

“Start Optimizing the Front End before the Back End.” --Some NewRelic guy at a conference

80% - 90% of website load time is on the front end. To improve front end performance you would do things like reduce the amount of requests needed for the browser to render the page, reduce the page size so that less MBs have to be transferred, or using a CDN to reduce latency for static content. These are things that you can’t improve by tuning “server” settings.

The other 10% - 20% of website load time is the back end. The back end consists of databases, web servers and all that stuff.

Basically if a webpage is 10MB in size and has 10000 requests per page, all the MySQL tuning in the world won’t help the page load any faster. Even optimizing Apache won’t do you any good. How would you make the page load faster? Enable Browser Caching in .htaccess, use a CDN, resize some images or split up the page.

1 Process with 1000 threads is going to use a lot of Memory, especially with Apache. Be sure to set processes and threads to sane values, too many threads, or processes can be bad, so try and find a safe mix.

Apache DirectoryIndex Priority Tuning

Recently, I noticed that cPanel decided it would be a good idea to place a TON of possible directory indexes that Apache should look out for and specify them in the main httpd.conf file. This is great for compatibility, however I found out that if you don't place the most common indexes first, Apache will search each directory under the request path for every index option listed in this section. I was able to see this activity when I used sysdig to view system calls that Apache makes when a webpage is requested.

DirectoryIndex  index.html.var index.htm index.html index.shtml index.xhtml index.wml index.perl index.pl index.plx index.ppl index.cgi index.jsp index.js index.jp index.php4 index.php3 index.phtml default.htm default.html home.htm index.php5 Default.html Default.htm home.html index.php

What that means is that if my www/ directory for my site is located in


Apache is going to look for every possible index file until it finds a match. If I use "index.php" but I don't specify it first in the list, then apache is going to search for like 20 files every time a request is made.

does /home/$user/www/index.html.var exist?
does /home/$user/www/index.htm exist?
does /home/$user/www/index.html exist?
does /home/$user/www/index.shtml exist?
does ........
OMG index.php exists, time to serve it up!

The point is that if you only use index.php or index.html and you never use any of the other types of indexes, then make sure you put index.php and index.html at the front of the list so that Apache doesnt have to search for 20 something types of indexes every time you make a request.

Apache Threads Versus Processes

Processes can be much faster than threads, but are constrained by memory. In some cases changing from 3 process with 3 threads each to 9 processes with 1 thread each significantly improved stability at higher concurrency levels, so that is worth testing out.

If the application tends to be very CPU intensive then trying to raise the process number could help instead of just raising threads.

If the application does a lot of IO and talks to other servers or databases then you are better off using more threads because there is not as much locking going on when you have lots of threads and fewer processes.

To see how many Apache processes are currently running, use this command

ps faux | grep httpd

To see how many total Apache threads are running under all processes

ps -eLf | grep httpd

To see how many threads are running under a specific Apache process

ps -eLf | grep $PID | wc -l

Apache MPM Overview

There are currently 3 different MPMs you can use with Apache. An MPM is responsible for binding to network ports and accepting requests from the user, then passing that request to one of the children.

Prefork: Each child process handles one connection at a time, this has been the default MPM for Apache for a long time.

Worker: Uses multiple child processes which each have multiple threads.

Event: Designed to improve performance and serve more requests at the same time by passing off some of the processing to supporting threads which helps to keep the main thread available to accept more requests.



  • Multi-Processing Module (MPM)
  • Impliments a NON-threaded, PRE-forking webserver
  • Each server process can answer incoming requests
  • A parent process manages the size of the server pool.
  • Used to avoid threading for compatibilty with NON-thread-safe libraries
  • Best MPM for isolating each request

How it works

  • Single control process launches child processes with listen for and serve connections
  • Always tries to maintain several spare or idle server processes so that new child processes do not need to be created.
  • The StartServers, MinSpareServers, MaxSpareServers, and MaxRequestWorkers regulate how the parent process creates children to serve requests
  • MaxConnectionsPerChild controls how frequently the server recycles processes by killing old ones and launching new ones.



  • This Multi-Processing Module (MPM) implements a hybrid multi-process multi-threaded server.
  • By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server.
  • It retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.

How it works

  • A single control process (the parent) is responsible for launching child processes.
  • Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread which listens for connections and passes them to a server thread for processing when they arrive.
  • Apache HTTP Server always tries to maintain a pool of spare or idle server threads, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new threads or processes to be created before their requests can be served.
  • The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxRequestWorkers directive.
  • The maximum number of active child processes is determined by the MaxRequestWorkers directive divided by the ThreadsPerChild directive.
  • ServerLimit is a hard limit on the number of active child processes, and must be greater than or equal to the MaxRequestWorkers directive divided by the ThreadsPerChild directive.
  • ThreadLimit is a hard limit of the number of server threads, and must be greater than or equal to the ThreadsPerChild directive.
  • A typical configuration of the process-thread controls in the worker MPM could look as follows:
ServerLimit         16
StartServers         2
MaxRequestWorkers  400
ThreadsPerChild     25

Event MPM

The Event MPM is preferred for high traffic sites because it is able to serve a lot of requests using little RAM. It does this by using threads to process requests. There is still some isolation since Event spawns multiple processes which each have their own threads. By using multiple threads instead of just processes libraries do not need to be loaded as much so less RAM is used.

If a single thread was causing issues it would only effect other threads within that process, it would not effect the other processes and their threads.

Event has a dedicated thread that only handles keep alive connections. It then passes requests to the child threads if one of the keep alive connections actually makes another request. Because the child threads don't care about keep alive, they can immediately move on after they process the request instead of having to wait to see if another request comes.

PHP is NOT thread safe so it's not possible to embed the PHP interpreter into each process. To get around this issue you would use something like PHP-FPM to handle the PHP processes. This means that Event is really only handling static content and passing along the PHP processing to PHP-FPM which runs it's own set of processes.

In general, EVENT uses less memory than PREFORK and provides much better performance in terms of requests per second.

How to Optimize Apache Event Configuration in httpd.conf

Besides adding an Ifmodule for event, there's not much more to do to really optimize Apache. Apache Event does a great job at regulating itself and killing off idle processes and idle threads. Apache will utilize around 60MB of memory when it's idle and under slight amounts of load. Even if you are blasting Apache left and right you won't see it use much more than 200MB - 300MB of Memory. Most of the time a majority of CPU and Memory is going to be used by PHP and MySQL. Apache generally doesn't have to do much if you have properly optimized the rest of the server. I've found that for most websites you can stick with a server limit of 16 and 25 threads per child, which is a total of 400 threads / maxrequest workers.

<IfModule event.c>
ServerLimit           16
MaxClients           400
StartServers           3
ThreadsPerChild       25
ThreadLimit           64

If you don't want to worry about PHP or MySQL and simply want to "optimize" apache to not cause a server to run out of memory, then try these settings. If your server is constantly freezing up or your website is slow, changing Apache settings isn't going to do anything to help your cause, try using PHP-FPM or update PHP before you try and squeeze performance out of apache.

<IfModule event.c>
ServerLimit            8         
MaxClients           200          
StartServers           3   
ThreadsPerChild       25
ThreadLimit           64

Apache and PHP

By default, any Apache version older than 2.4 uses the Prefork MPM and also runs with an embedded PHP interpreter (mod_php). This is nice in terms of compatibility, however this combination is not very fast and isn't very efficient in terms of resource utilization.

Apache must spawn a new process every time a new connection occurs, when there are a lot of simultaneous requests there ends up being a lot of separate processes that each use up a decent amount of RAM. Because each process needs to be isolated, the PHP interpreter must be loaded for every process, along with all the libraries needed. This is obviously not the most efficient way to utilize RAM.

On top of all this, each process will remain tied up while the connection is active so the process is not destroyed until after the connection is closed so the likely hood of a server running out of RAM is much greater if you use Prefork and mod_php.

Apache Prefork and PHP Memory Usage

I did a quick test to see exactly how much Memory Apache Prefork processes use. The idle usage showed that there were 4 httpd processes running, each using about 8MB of Memory. Idle Apache uses about 32MB of RAM. Then I installed wordpress and run AB at 1 concurrent connection, 100 requests, then at 10 concurrent connections and recorded the results. As you can see Apache really doesn't use much RAM at all compared to IDLE. It's PHP that is using the memory. Now, keep in mind that your numbers will look different depending on what you set your start, min and max servers to, in this case it looks like it's set to 4.

Idle Prefork RAM Usage: 7.8MB per process (4 processes total = 32MB)
ab -c 1 -n 100 Prefork RAM Usage: 9.2MB per process (5 processes total = 46MB)
ab -c 10 -n 100 Prefork RAM Usage: 10MB per process (5 processes total =  51MB)

I used this script and used watch -n 1 to view memory usage in real time

ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print "Number of processes =",count; print "Memory usage per process =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'

ps -C $2 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print "Number of processes =",count; print "Memory usage per process =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'

If you look at PHP memory usage you can see that this is where RAM gets used in a hurry. The argument can be made that since Apache is spawning the PHP processes, the memory for both PHP and Apache should be combined, but the point is to isolate processes and get a breakdown of what uses what in terms of resources.

Idle PHP RAM Usage: 0MB (no PHP processes unless apache spawns them)
ab -c 1 -n 100 PHP RAM Usage: 22MB per process (1 processes total = 22MB)
ab -c 10 -n 100 PHP RAM Usage: 22MB per process (10 processes total =  220MB)


[for more information on PHP-FPM see this page]

"PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation with some additional features useful for heavy-loaded sites." PHP-FPM has been included with PHP since 5.3.3.

Click here for more information on configuring [PHP-FPM]


"mod_fcgid is a high performance alternative to mod_cgi or mod_cgid, which starts a sufficient number instances of the CGI program to handle concurrent requests, and these programs remain running to handle further incoming requests. It is favored by the PHP developers, for example, as a preferred alternative to running mod_php in-process, delivering very similar performance."


When you use mod_php Apache runs PHP inside it's process, so there is no external PHP process running. This means PHP is run with the Apache user and group. This can be a huge security concern if permissions are not set correctly.

Apache 2.4 MPM Directives

MaxRequestWorkers (Previously MaxClients)

MaxRequestWorkers 400 (default = ServerLimit x ThreadsPerChild) restricts the total number of threads that will be available to serve clients. MaxRequestWorkers was previously called MaxClients before Apache 2.3.13.

The Event default settings allow for 400 total threads to serve requests. If you want to lower or raise this value just multiply ServerLimit by ThreadsPerChild to get the value to use for MaxRequestWorkers

MaxRequestWorkers 400 (16 x 25 = 400)
ServerLimit 16 (default)
ThreadsPerChild 25 (default)

Since MaxRequestWorkers defaults to ServerLimit x ThreadsPerChild you shouldn't have to specify this value. You might want to do so just to make it clear what the max value is without having to do math.



For the worker and Event MPM, this directive is multiplied by ThreadLimit which sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. You must stop and start Apache to change this value, changes for this value are ignored during a restart.

If ServerLimit is set to a value much higher than necessary, extra, unused shared memory will be allocated.

You should only modify this limit if you need to have more than 16 processes running. If you want to handle more requests, try raising ThreadsPerChild first.

ServerLimit 16 (default)



Worker defaults to 3 StartServers. The Apache documentation says there is little reason to change this value. I'm not going to even tell you how to change this value because odds are there is no need to change this value.



ThreadLimit sets the limit for the maximum value of ThreadsPerChild. If you set ThreadLimit to a value higher than ThreadsPerChild then you are wasting resources.

For Event and Worker the default value is 64. There should be no reason to change this limit. With 16 servers running with 64 threads each you are able to set Apache to be able to handle 1024 requests. This is a lot, so unless you have a massive website you shouldn't ever need to raise this value.

ThreadLimit 64 (default)



This directive sets the amount of threads created under each child process. By default this value is set to 25. You cannot raise this value unless you raise the ThreadLimit value first.

ThreadsPerChild 25 (default)

How to enable Event MPM for Apache 2.4 on CentOS 7

If you install Apache 2.4 on CentOS 7 the Prefork MPM will be used by default. Prefork is not ideal for a fast webserver, at least in my opinion. I prefer to use the Event MPM which is an improvement over the Worker MPM. The main difference between the Worker and Event MPM is that Event has a dedicated thread which handles all Keep Alive connections and requests, the Worker MPM does not have this dedicated thread, so each worker thread keeps track of any KeepAlive connections within the single thread.

Anyway, to change the Apache MPM from Prefork to Event on CentOS 7 you can modify 00-mpm.conf, comment out the Prefork line and then un-comment out the Event line. Stop and start Apache and you should see that Event is now in use.

vim /etc/httpd/conf.modules.d/00-mpm.conf

# Select the MPM module which should be used by uncommenting exactly
# one of the following LoadModule lines:

# prefork MPM: Implements a non-threaded, pre-forking web server
# See: http://httpd.apache.org/docs/2.4/mod/prefork.html
#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

# worker MPM: Multi-Processing Module implementing a hybrid
# multi-threaded multi-process web server
# See: http://httpd.apache.org/docs/2.4/mod/worker.html
#LoadModule mpm_worker_module modules/mod_mpm_worker.so

# event MPM: A variant of the worker MPM with the goal of consuming
# threads only for connections with active processing
# See: http://httpd.apache.org/docs/2.4/mod/event.html
LoadModule mpm_event_module modules/mod_mpm_event.so

Run this command to confirm that Apache is using Event

httpd -V | grep MPM

Server MPM:     event

Installing Apache 2.4 from Source Cent6.4

How to install apache 2.4 from source on CentOS 6.4 has been moved here

Build in APC with Apache and PHP 5.3

  • The Alternative PHP Cache (APC) is a free and open opcode cache for PHP. Its goal is to provide a free, open, and robust framework for caching and optimizing PHP intermediate code.
  • Only works with DSO or FCGI

Cent5/6 Install

cd /usr/local/src
wget http://pecl.php.net/get/APC-3.1.13.tgz
tar -zvxf APC-3.1.13.tgz
cd APC-3.1.13

./configure --enable-apc --enable-apc-mmap --with-apxs=/usr/local/apache/bin/apxs
make install

vim /usr/local/lib/php.ini

##Add the following
;specifies the size for each shared memory segment 8MB to start
;max amount of memory a script can occupy
; means we are always atomically editing the files

##comment out zend-extensions and you're all done.
; zend_extension_manager.optimizer = "/usr/local/Zend/lib/Optimizer-3.3.3"
; zend_extension_manager.optimizer_ts = "/usr/local/Zend/lib/Optimizer_TS-3.3.3"

##Confirm that the module is installed
/etc/init.d/httpd restart
php -m |grep apc
  • There will be a apc.php file in the apc directory (depends on where installed) Move this to a web accessible directory to get access to the web interface

Apache One Liners and Troubleshooting Commands

One liner that shows Apache connections for a certain domain during a time period

One liner that shows connections to all domains during a certain time. Change the "hour" variable to the hour you want to search. 16= 4PM

cd /usr/local/apache/domlogs

hour=16;for domain in $(cat /etc/userdomains | grep -v nobody |cut -d':' -f1); do if [ -e "$domain" ]; then for minute in $(seq 10 59); do count=$(cat $domain | grep "$hour:$minute"|wc -l);if [ "$count" -gt 1 ]; then echo "$domain : $hour:$minute : $count" >> /home/domlogreport.$hour;fi;done;echo;echo;fi;done
cat /home/domlogreport.$hour | sort -g -k 3

Report is in /home/domlogreport

How to get IP counts from Apache logs

To get a list of the top 50 IPs with the most connections to Apache, you can run the command below.

 /usr/bin/lynx -dump -width 500 | awk '{print $11}'| awk NF |grep [0-9].[0-9].[0-9].[0-9]|sort -n|uniq -c|tail -50 | sort -rn

How to check for excessive connections to Apache

To see a list of the IPs that have the most connections to port 80 you can use netstat, grep apache's listening port (80) then use awk and cut to grab the IP column, then sort the IPs and only display unique IPs, then count them and sort from highest to lowest connections. If you notice a few IPs with lots and lots of connections then you may want to investigate further and block if necessary.

netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head  

How to check for SYN floods

You can check for SYN flood attacks by using the command below. If you notice a ton of port 80 SYNs and Apache is slow to respond / failing to respond, this command should show you that.

netstat -nap | grep SYN | wc -l

How to view the sites with most connections on WHM / cPanel server

The command below can be ran on a cPanel server, it will check the Apache server status, print the vhost column (column 12), then will remove extra white space, then sort, finally count all unique connections and add them up. This will give you a good idea of what sites are the most active.

/usr/bin/lynx -dump -width 500 | awk  'BEGIN { FS = " " } ; { print $12 }' | sed '/^$/d' | sort | uniq -c       

The command below is similar to the one above, slightly different output.

/usr/bin/lynx -dump -width 500 | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head                

How to view the busiest script / file for a cPanel Server

The command below will look at all the current Apache requests and will parse them to get a list of the most requested script or file.

/usr/bin/lynx -dump -width 500 | grep GET | awk '{print $14}' | sort | uniq -c | sort -rn | head                

Useful Apache Links on Performance and Optimization


  • Bitnami uses Apache Event and [PHP-FPM]. This combination resulted in lower memory usage and the ability to process more requests.