Bottleneck for testing downloads on nodes using Google Compute Engine

Question

I can’t understand what is the reason for the bottleneck on this site, the very poor response time when it reached about 400 users. The site is in the Google compute engine, using a group of instances, with network load balancing. We created a project using sailjs.
I did load testing using a google container using kubernetes by running the locust.py script.

The main results for one of the tests are:

RPS : 30 Spawn rate: 5 p/s TOTALS USERS: 1000 AVG(res time): 27500!! (27,5 seconds) 

The response time at first is long, below one second, but when it begins to reach about 400 users, the response time starts in hop-hop mode.

I tested the obvious factors that may affect this response time, the results are below:

Calculate instance instances (2 x standard-n2, 200gb disk, ram: 7.5gb per instance):

 Only about 20% cpu utilization used Outgoing network bytes: 340k bytes/sec Incoming network bytes: 190k bytes/sec Disk operations: 1 op/sec Memory: below 10% 

MySQL

 Max_used_connections : 41 (below total possible) Connection errors: 0 

All the other results for MySQL also seem to be fine, for no reason causing a bottleneck.

I tried the same test for a new project created by sailjs, and it did better, but still had terrible results, 5 seconds of res time for about 2000 users.

What else should I check? What could be the bottleneck?

Answer 1

Do you read / write files? This is a serious obstacle in node.js and always causes some problems. Caching read files or eliminating the need for such code should be done as much as possible. In my own experience, submitting files like images, css, js and my same node server can cause problems when increasing the number of simultaneous requests. The solution was to serve all this through a CDN.

Another example would be the mysql driver. We had some problems with the connection being closed incorrectly (not using sails.js, but I think they were using the same driver at the time I came across this), so they will cause problems on the mysql server, which will lead to long delays when retrieving data from the database. You should time / track the number of mysql queries and make sure they are delayed.

Finally, it could be some kind of special issue with sails.js and the Google Computing Engine. You must make sure that you have any open problems on the same problem that you are facing.