Every day we face issues with performance and it is not always clear what leads to bad results in the first place. In this post, I decided to put together possible steps on how to detect performance problems.
First of all, we need to detect possible causes of low performance. Usually, they are:
- CPU usage;
- GPU usage;
- I/O operations.
Check CPU usage
The time utility runs the specified command with the given arguments. When the command finishes, time outputs a message to standard error giving timing statistics about this command run. These statistics consist of the elapsed real time between invocation and termination, the user CPU time (the sum of the tms_utime and tms_cutime values in a struct tms), and the system CPU time (the sum of the tms_stime and tms_cstime values in a struct tms).
To check which part of the code caused problems, we will use profiling.
As we can see, that operation of I/O doesn’t take time (real-user+sys), almost all the time of the program is used by CPU.
The top utility provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel.
We can see on the screen (pid 7183), that command used 100% of a single kernel of CPU.
Htop is similar to top but allows you to scroll vertically and horizontally, so you can see all the processes running in the system, along with their full command lines.
We can see on the screen (pid 32145), that command does not use a single kernel of CPU.
Check GPU usage
The NVIDIA System Management Interface (nvidia-smi) is a command line utility, based on top of the NVIDIA Management Library (NVML), intended to aid in the management and monitoring of NVIDIA GPU devices.
In our case, almost all of the memory is used (10752 Mb). This GPU instance is fully utilized.
Iotop watches I/O usage information output by the Linux kernel and displays a table of current I/O usage by processes or threads in the system
According to the information from this post
SSD (Solid State Drive) Generally above 200 MB/s and up to 550 MB/s for cutting edge drives
HDD (Hard Disk Drive) The range can be anywhere from 50–120MB / s
This test was run on SSD disc so no issues with IO operations detected
cProfile is recommended for most users; it’s a C extension with reasonable overhead that makes it suitable for profiling long-running programs. Based on lsprof, contributed by Brett Rosen and Ted Czotter.
To run cProfile you can use instructions from the documentation or run it from the command line:
python2.7 -m cProfile -o profile.output io.py
-m mod: run library module as a script (terminates option list)
– o: file with the output
Cprofile generates results in an unreadable (binary) format. To interpret the results, there is several tools to use.
Pyinstrument with a call tree:
You can find usage recommendations in the post
Conclusion and Recommendations
So right now you have everything to detect the problem and understand the reason why it occurred. After that, you can decide on how to solve the issue with the detected bottleneck. You can either update your code or change environment configurations.
It is a good idea to run the test again after you fix the root cause, just to check if the problem was solved.
Hope that will help.