I have been working with many customers over last 6 months on fixing performance issues across different stacks in production deployments. One of the tool that always comes to the rescue is strace - a process diagnostic tool. You can install it using one of your favorite package manager if it's not there already.

Most of the times tuning performance on a live production machine (if you have some decent scale) is like dealing with a patient in emergency ward - you have to act fast and fix things quickly. During any troubleshooting, there will be too many components in the system or application and you need time to understand the impact of them before getting into "lets fix this" mode.

I generally get started with the first/last component in the chain (typically a web/app or db server) and try to identify what it is doing through strace utility - it helped me so many times to identify whether the issue was with apache/nginx/php-fpm/uwsgi/java processes or some other blocking for a resource like DB or accessing certain system call too many times due to bad code (imagine iterating over 1000's of records and accessing timezone locale every time instead of caching it) in the application.

The simplest way to get diagnostic information is to attach your busy process (high cpu or memory) to the strace and just watch the system calls - it will be immensly helpful to know  process execution trace.

Attach a process to strace for diagnostics : $ strace -p <processid>

To see all open and read calls of a process : $ strace -e trace=open, read, close, connect  -p <pid>

Capture strace output for a process : $ strace -p <pid> -o /file/path/debug.php-fpm.txt

What is taking time? : $ strace -c -p <pid>

And you can options like -s to specify the maximum size of the output string to more than the default 32. It is the most powerful tool for troubleshooting things in Linux environments.

Happy debugging and good luck with tuning systems - the best job at times:)