Python: How do you find CPU consumption for a piece of code?

Question

Background:

I have a django application, it works and responds well to low load, but at high load, for example, 100 users / sec, it consumes 100% of the processor, and then, due to the lack of a processor, it slows down.

Problem:

• Profiling an application gives me the time it takes to execute functions.
• This time increases under high load.
• Elapsed time can be caused by complex calculations or by waiting for the CPU.

, so how do you find the processor cycles consumed by a piece of code?

Since reducing CPU consumption increases response time.

• I may have written extremely efficient code and you need to add more processor power.

OR

• Can I have some kind of dumb code that takes a CPU and causes a slowdown?

Any help is appreciated!

Update:

• I use Jmeter to profile my webapp, it gives me a bandwidth of 2 requests / sec. [100 users]
• I get an average time of 36 seconds on 100 requests versus 1.25 s on 1 request.

Additional Information

• Nginx + Uwsgi configuration with 4 workers
• There is no database using responses from the REST API
• For the first time, the REST API response gets cached, so it doesn't matter.
• Using ujson to parse json.

Curious to know:

• Python-Django is used by many orgs for a large number of large sites, then there should be some tools for analyzing debugging / memory and the CPU.
• All I found were random code snippets that perform profiling.

Answer 1

You can try to set your test to slow, slow so that you can see how the processor gradually increases and then starts the profiler before you click on the high processor. There is no point in trying to profile the code when the CPU is at its maximum, because at that moment everything will be slow. In fact, you really only need a small load to get useful data from the profiler.

In addition, by gradually increasing the load, you can better understand whether there is a gradual increase in the number of processors (which indicates a processor bottleneck) or a sudden jump in the CPU (suppose this is another problem, it will not necessarily be solved by a large number of processors).

Try using something like Cosntant's bandwidth timer to speed up requests, this will prevent JMeter from getting into and overloading the system.

Answer 2

Checkout A new relic for some pretty nice analysts, they have a django-specific logging.