Performance Troubleshooting Methodologies

From wiki.mikejung.biz
Jump to: navigation, search

Liquidweb 728x90.jpg

Intro

Will be filling this in later. But for now, check out the link above to learn more. This guy is basically a wizard and this entire page will basically be based off some of the methodologies he talks about in his latest performance book. http://www.amazon.com/Systems-Performance-Enterprise-Brendan-Gregg/dp/0133390098

CPU Analysis

vmstat

vmstat (run per second) and see how much IDLE headroom there is, if less than 10% there might be an issue, or it might be time to upgrade the CPU.

vmstat 1

mpstat

mpstat will show you the current activity of each CPU. This can help to identify "hot" CPUs and will show if a process is only utilizing a single CPU, or if it is utilizing many CPUs.

mpstat -P ALL 1

perf

To record what is going on when a process runs, specifically where the most CPU time is being spent, you can run a command then start perf to record what is going on. An example would be to start sysbench at 1 thread, record the results, then start sysbench again at 32 threads to see if there is a threading issue.

### In one screen / terminal run:

sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=$db --mysql-user=sysbench --mysql-password=$password --max-time=120 --oltp-read-only=on --max-requests=0 --num-threads=1 run

### Right before / after you run the above command, run this to record the data

perf record -a sleep 120


### To view the data run:

perf report --stdio

You can also specify which command you want to record by running the following

perf record -g $command

### Example
perf record -g sysbench --test=cpu --cpu-max-prime=20000 --num-threads=64 run

To sort results by CPU:

perf report --stdio --sort=cpu


You can also view live information using perf top

perf top