Performance Troubleshooting Methodologies
Will be filling this in later. But for now, check out the link above to learn more. This guy is basically a wizard and this entire page will basically be based off some of the methodologies he talks about in his latest performance book. http://www.amazon.com/Systems-Performance-Enterprise-Brendan-Gregg/dp/0133390098
vmstat (run per second) and see how much IDLE headroom there is, if less than 10% there might be an issue, or it might be time to upgrade the CPU.
mpstat will show you the current activity of each CPU. This can help to identify "hot" CPUs and will show if a process is only utilizing a single CPU, or if it is utilizing many CPUs.
mpstat -P ALL 1
To record what is going on when a process runs, specifically where the most CPU time is being spent, you can run a command then start perf to record what is going on. An example would be to start sysbench at 1 thread, record the results, then start sysbench again at 32 threads to see if there is a threading issue.
### In one screen / terminal run: sysbench --test=oltp --db-driver=mysql --oltp-table-size=10000000 --mysql-db=$db --mysql-user=sysbench --mysql-password=$password --max-time=120 --oltp-read-only=on --max-requests=0 --num-threads=1 run ### Right before / after you run the above command, run this to record the data perf record -a sleep 120 ### To view the data run: perf report --stdio
You can also specify which command you want to record by running the following
perf record -g $command ### Example perf record -g sysbench --test=cpu --cpu-max-prime=20000 --num-threads=64 run
To sort results by CPU:
perf report --stdio --sort=cpu
You can also view live information using perf top