Performance issues detection. Python profiling
Every day we face issues with performance, and it is not always clear what leads to bad results in the first place. In this post, I decided to put together possible steps to detect performance problems.
First of all, we need to detect possible causes of low performance. Usually, they are:
- CPU usage;
- GPU usage;
- I/O operations.
Check CPU usage
Time utility
The time utility runs the specified command with the given arguments. When the command finishes, time outputs a message to standard error giving timing statistics about this command run. These statistics consist of the elapsed real time between invocation and termination, the user CPU time (the sum of the tms_utime and tms_cutime values in a struct tms), and the system CPU time (the sum of the tms_stime and tms_cstime values in a struct tms).
Run
To check which part of the code caused problems, we will use profiling.
Results
As we can see, the operation of I/O doesn’t take time (real-user+sys). Almost all the time, the program is used by the CPU.
Top utility
The top utility provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel.
Run
Results
We can see on the screen (pid 7183) that the command used 100% of a single kernel of CPU.
Htop utility
Htop is similar to top but allows you to scroll vertically and horizontally, so you can see all the processes running in the system, along with their full command lines.
Run
Results
We can see on the screen (pid 32145), that command does not use a single kernel of the CPU.
Check GPU usage
Nvidia-smi utility
The NVIDIA System Management Interface (nvidia-smi) is a command line utility based on top of the NVIDIA Management Library (NVML), intended to aid in the management and monitoring of NVIDIA GPU devices.
Run
Results
In our case, almost all of the memory is used (10752 Mb). This GPU instance is fully utilized.
Check I/O
Iotop utility
Iotop watches I/O usage information output by the Linux kernel and displays a table of current I/O usage by processes or threads in the system
Runs
Results
According to the information from this post
SSD (Solid State Drive) Generally above 200 MB/s and up to 550 MB/s for cutting-edge drives
HDD (Hard Disk Drive) The range can be anywhere from 50–120MB / s
This test was run on an SSD disc so no issues with IO operations detected
Profiling
cProfile
cProfile is recommended for most users; it’s a C extension with reasonable overhead that makes it suitable for profiling long-running programs. Based on lsprof, contributed by Brett Rosen and Ted Czotter.
To run cProfile you can use instructions from the documentation or run it from the command line:
python2.7 -m cProfile -o profile.output io.py-m mod: run library module as a script (terminates option list)
– o: file with the output
Cprofile generates results in an unreadable (binary) format. To interpret the results, there are several tools to use.
Tools
Pyinstrument with a call tree:
Tuna visualization:
Cpofilev visualization:
You can find usage recommendations in the post.
Conclusion and Recommendations
So now you have everything to detect the problem and understand why it occurred. After that, you can decide on how to solve the issue with the detected bottleneck. You can either update your code or change environment configurations.
It is a good idea to rerun the test after fixing the root cause just to check if the problem was solved.
Hope that will help. Reach out to Quantum if you have any questions or ideas to share.
