Main Page

Revision as of 01:43, 17 January 2012 by Admin (talk | contribs) (Hardware Checks and Commands)
Jump to navigation Jump to search


"Can you take a look at my server and tell me what's going on?"

Starting Points

Who is on the server?


Show top processes.

top -c

Load averages.

sar -q

Ram usage.

sar -r

I/O wait

sar -s


HTTPD='/usr/local/apache/conf/httpd.conf'; PHP=`php -i | grep php.ini | grep "Configuration" | cut -d ">" -f2 | cut -c 2- | tail -n 1`; MYSQL='/etc/my.cnf'; IOSTAT=( $(iostat 1 2 | grep iowait -A1 | awk '{print $4}' | grep -v system) ); echo -e "\n=== SERVER STATS ===\n"; echo -e "Host: `hostname`"; echo "CPUs: `cat /proc/cpuinfo | grep processor -c`"; echo "I/O Wait: `echo ${IOSTAT[1]}`"; echo -e "\n=== Disk Space Usage ===\n"; df -h; echo -e "\n=== MySQL Database queries ===\n"; mysqladmin proc stat;  echo -e "\n=== Exim Stats ===\n"; echo -e "Emails in queue: `exim -bpc`"; echo -e "Exim procs: `ps faux | grep exim -c`"; echo -e "\n=== Number of SYN connections ===\n"; netstat -nap | grep SYN | wc -l; echo -e "\n=== Top 10 SYN Flood Conections ===\n"; netstat -tn 2>/dev/null | grep SYN | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head; echo -e "\n=== PHP Info ===\n"; egrep 'max_execution_time|max_input_time|memory_limit' $PHP; echo -e "\n=== Number of Apache Processes ===\n"; ps faux | grep httpd -c | grep -v grep; echo -e "\n=== Top 10 connections to apache ===\n"; netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head; echo -e "\n=== Current Memory Usage ===\n"; free -m;  echo -e "\n=== Apache Configuation ===\n"; httpd -V | grep MPM; egrep 'MaxClients|KeepAlive|MaxRequestsPerChild|Timeout|Servers|Threads|ServerLimit' $HTTPD | grep -v SSL; echo -e "\n=== MySQL Configuration ===\n"; grep max_connections $MYSQL;


Apache Status

/usr/bin/lynx -dump -width 500 | less

Apache connection

/usr/bin/lynx -dump -width 500 | awk '{print $11" "$12}'| awk NF |grep [0-9].[0-9].[0-9].[0-9]|sort|uniq -c|sort -n|tail -50

Check settings in httpd.conf. Added +160 usually located around that line number.

vim /usr/local/apache/conf/httpd.conf +160

Think Apache is causing server to go OOM? Check PHP memory limit. If it's above 32M ask client if they need it this high.

grep memory_limit /usr/local/lib/php.ini

Find all users php.ini files.

find /home/*/public_html/* -name php.ini

Dos Script

netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head

Get a list of top IPs accessing the server (some false positives)

tail -n50000 access_log | grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" | sort -n | uniq -c | sort -n


Useful for seeing what queries are doing what.

watch -n 1 'mysqladmin proc stat'

Check /tmp for sess_* files. Can mean tables are corrupt.

ll /tmp/

Check the logs.


Then check MySQL settings

vim /etc/my.cnf

Network Stuff

New server missing some IPs it was supposed to come with?

 service ipaliases restart

Ubuntu Networking

Where all the nics are actually configured:

vim /etc/udev/rules.d/70-persistent-net.rules

Bonding info

cat /proc/net/bonding/bond0

Interface config

vim /etc/network/interfaces


One Liners

See MySQL status. Updates every 1 s.

watch -n 1 'mysqladmin proc stat'

Optimize Tables.

for i in $(mysql -e "show databases;" | sed 's/Database//') ; do for each in $(mysql -e "use $i; show tables;" \
| sed 's/Tables.*//' ;) ; do mysql -e "use $i ; optimize table $each" ; done ; done

MySQL check that will: check all databases, analyze, optimize and repair. Pretty useful, usually safe.

mysqlcheck -Aaor

Dump a database.

mysqldump database > database.sql

Import a database.

mysql database < database.sql

Connect to a database.

mysql -u user -h ip -p databasename

Viewing and Deleting Tables and Databases

Look at databases and tables.

> use databasename;
> show tables;

Drop (delete) a database. Can be useful if importing a database and it gives you an error.

> drop database databasename;

Optimization Scripts

While the default configs here are a good starting point. These scripts will help in finding any issues with the users current MySQL config.

Note: I like to run these like : '/scripts/ > /root/tuning-primer.txt[n]' This saves the output, so you don't feel compelled to add it as a note to a ticket or admin comments. You can also use it to compare the results after 48 hours. This is a great way to document these changes.

wget -O /scripts/
chmod +x /scripts/

If is down, try:

wget -O /scripts/

wget -O /scripts/
chmod +x /scripts/

Upgrading MySQL

Template:Box Warning

If pre-MySQL 5.0:

 mysqlcheck -Aaor

If MySQL 5.0 to 5.1:

 mysqlcheck -Agr

And back up all the databases:

 mkdir -p /backup/mysqldumps
 cd /backup/mysqldumps
 for i in $(mysql -e "show databases;" | cut -d ' ' -f2 | grep -v Database); do `mysqldump $i > $i.sql`; done

Enabling a Slow Query Log

How to enable a slow query log

touch /var/lib/mysql/slow.log

chown mysql. /var/lib/mysql/slow.log

In the my.cnf file under the mysqld section add this:


Then restart mysql and you have a slow query log.

If you want to specify the number of seconds that indicates a long or slow query, use this line in /etc/my.cnf :

long_query_time = 5

changing 5 to whatever number of seconds you want.


Parse Error

Parse error: syntax error, unexpected T_STRING

Check the file and remove <?xml version="1.0" encoding="utf-8"?>

Force PHP5

Add to .htaccess:

AddType application/x-httpd-php5 .html .htm

Apache PHP Handlers

Can use this command to change owner and group

chown -R user:group /directory/

Install Zip from source

cd /usr/local/src
tar -zxvf zip-1.10.2.tgz
cd zip-*
make && make install

Install any extension from source:

cd /usr/local/src
wget somthing
tar -zxvf something.tgz
cd something-*
make && make install
echo "extension =" >> /etc/php.ini

May need to do these additional steps:

cd /usr/local/src/php-5.2.11
make clean
php -i | grep configure | sed s/\'//g | sed s/'Configure Command =>  '//g
add --enable-zip to output
make install


Ownership - permissions should be 755



Ownership - permissions should be 755



Email accounts not showing up in cPanel.

Check /home/user/etc Make sure the passwd file and shadow file have proper permissions also make sure they are located in



Can't find file: 'horde_sessionhandler.MYI'

/etc/init.d/mysqld stop
rm /var/lib/mysql/horde/horde_sessionhandler.frm
/etc/init.d/mysqld start

>CREATE TABLE horde_sessionhandler (session_id VARCHAR(32) NOT NULL, session_lastmodified INT NOT NULL, session_data LONGBLOB, PRIMARY KEY 
(session_id)) ENGINE = InnoDB;

>GRANT SELECT, INSERT, UPDATE, DELETE ON horde_sessionhandler TO [email protected];


Find top sending IPs in exim logs:

grep "SMTP connection from" /var/log/exim_mainlog |grep "connection count" |awk '{print $7}' |cut -d ":" -f 1 |cut -d "[" -f 2 |cut -d "]" -f 1 |sort -n |uniq -c | sort -n

Find authenticated users who may be spamming:

find /var/spool/exim/input/ -name '*-H' | xargs grep 'auth_id'

Spam comming from scripts:

grep cwd=\/home\/ /var/log/exim_mainlog| cut -d' ' -f4 | sort | uniq -c | sort -n

Removing all queued messages at once in a safe way:

exim -bp | awk '/^ *[0-9]+[mhd]/{print "exim -Mrm " $3}' | sh

Or you can do the same from the mail queue manager in WHM.

APF SMTP tweak enables mail to be sent only from the mail or mailman GID, and blocks all outbound SMTP, except through the sendmail binary. Add this bold line of code to /etc/init.d/apf , right underneath the start) case:

/usr/local/sbin/apf --start >> /dev/null 2>&1
'''/scripts/smtpmailgidonly on'''


Add relaying from another server:

Add the IP to the "remote service IPs" in cPanel


Find Spam in the queue:

egrep -l "user" /var/spool/clientmqueue/Q* | wc -l


To search for available packages:

yum search example

Find packages and where they lead to:

rpm -qa | grep example
rpm -ql example

Java + Tomcat

Regular install:

yum install java-1.6.0-openjdk.x86_64
yum install tomcat5

Stripped down server install:

yum install tomcat5
yum install java-1.6.0-openjdk.x86_64
yum install tomcat5-webapps.x86_64
yum install httpd-devel.x86_64
tar -zxvf tomcat-connectors-1.2.32-src.tar.gz
cd tomcat-connectors-1.2.32-src
cd native/
./configure --with-apxs=/usr/sbin/apxs
make install
vim /etc/httpd/conf/httpd.conf
add LoadModule jk_module modules/
httpd -M or /etc/init.d/httpd -l


Configuration file, lots of settings can be changed here:

vim /usr/local/apache/conf/httpd.conf

Includes (external settings that Apache reads in case the conf was rebuilt)

cd /usr/local/apache/conf/includes

Check for a basic Dos, or heavy traffic:

netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head  

Count the processes:

ps aux | grep httpd | wc -l
ps aux | grep php | wc -l


cPanel not working for some accounts on some servers:

chgrp user /var/cpanel/users/username
vim /etc/proftpd/username


PureFTP using FTPES

Edit /etc/pure-ftpd.conf and uncomment (enable) the PassivePortRange line, like below.

# Port range for passive connections replies. - for firewalling.
PassivePortRange          30000 50000

APF - /etc/apf/conf.apf

# Common ingress (inbound) TCP ports

# Common egress (outbound) TCP ports

CSF - /etc/csf/csf.conf

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,953,993,995,2077,2078,2082,2083,2086,2087,2095,2096,30000:50000"

# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,37,43,53,80,110,113,443,587,873,953,2087,2089,2703,30000:50000"

Packages / Yum

Install clamd on a coremanaged server:

yum --disablerepo=\* --enablerepo=epel install clamd

Adding RPMforge repos:

rpm --import
rpm -K rpmforge-release-0.5.2-2.el5.rf.*.rpm
rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm

Once this is complete you can now install other packages via yum!

Load Balanced Troubleshooting

Reboot Process

Check to make sure SAN is mounted:

netstat -lpn | grep 192.168

Stop OCFS2 before reboot:

/etc/init.d/ocfs2 stop

Start OCFS2:

/etc/init.d/ocfs2 start

Make sure man is mounted:

mount | grep san

Restart Apache:

service httpd restart


Find all index.* files then remove bad things

find /home/*/public_html/ -name index.* > /root/list
for each in `cat /root/list` ; do sed -i.lwbak 's/Badthing\/script>//g' $each ; done

Find symlinks

find /home/*/public_html/ -type l | cut -d / -f3 | sort -nr | uniq -c



Nmap commands:
-s what type of scan
-T TCP connect
-p scan all the things
-PN scan all and pretend they are alive

TCP scan:
nmap -sT -p -PN

SYN scan:
nmap -sS -p -PN

UDP scan:
nmap -sU

Versioning UDP scan:
nmap -sUV

XMAS scan:
nmap -sX -p -PN

NULL scan:
nmap -sN -p -PN


Conflicting Server Name Error

Check for duplicates/system users:

grep -i /var/cpanel/users/*

If there is a domain entry owned by "system" remove this file:

rm /var/cpanel/users/system

Then run:


Hardware Checks and Commands

Check for disk age:

smartctl -a /dev/sda | grep Power_On_Hours

Check disk performance:

hdparm -tT /dev/whateverdevice

Run a smart test (quick)

smartctl -t short /dev/whateverdevice

Kaspid issues

Can disable:

vim /boot/grub/grub.conf

Server will then need to be rebooted

cPanel Tips and Tricks

Exclude files from being updated.

vim /etc/cpanelsync.exclude

Then add the absolute path for the file. An example would be Roundcube webmail settings:



Disable zone transfers with named.conf

acl can_axfr {;

options {
    allow-recursion { trusted; };
    allow-transfer { can_axfr; };