Performance Troubleshooting Methodologies

Jump to: navigation, search

Liquidweb 728x90.jpg


Will be filling this in later. But for now, check out the link above to learn more. This guy is basically a wizard and this entire page will basically be based off some of the methodologies he talks about in his latest performance book.

CPU Analysis


vmstat (run per second) and see how much IDLE headroom there is, if less than 10% there might be an issue, or it might be time to upgrade the CPU.

vmstat 1


mpstat will show you the current activity of each CPU. This can help to identify "hot" CPUs and will show if a process is only utilizing a single CPU, or if it is utilizing many CPUs.

mpstat -P ALL 1


To record what is going on when a process runs, specifically where the most CPU time is being spent, you can run a command then start perf to record what is going on. An example would be to start sysbench at 1 thread, record the results, then start sysbench again at 32 threads to see if there is a threading issue.

### In one screen / terminal run:

sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=$db --mysql-user=sysbench --mysql-password=$password --max-time=120 --oltp-read-only=on --max-requests=0 --num-threads=1 run

### Right before / after you run the above command, run this to record the data

perf record -a sleep 120

### To view the data run:

perf report --stdio

You can also specify which command you want to record by running the following

perf record -g $command

### Example
perf record -g sysbench --test=cpu --cpu-max-prime=20000 --num-threads=64 run

To sort results by CPU:

perf report --stdio --sort=cpu

You can also view live information using perf top

perf top