EC2 ELB Performance Issues

Question

Two questions about EC2 ELB:

First, how to run JMeter tests correctly. I found the following http://osdir.com/ml/jmeter-user.jakarta.apache.org/2010-04/msg00203.html , which basically says to set -Dsun.net.inetaddr.ttl = 0 at startup JMeter (which is easy), and the second point it does is that routing is done for ip not for every request. Therefore, in addition to starting the jmeter instance farm, I don't see how to get around this. Any ideas are welcome, or maybe I am reading the explanation incorrectly (?)

In addition, I have a web service that makes a server-side call to another web service in java (and both behind ELB), so I use HttpClient and this is MultiThreadedHttpConnectionManager, where I provide some big-ish routes to host the value in the connection manager. And I'm wondering if this ELB load balancing behavior will change because the connections are cached (and also that all requests come from the same computer). I can switch to using the new HttpClient every time (kind of lame), but this does not mean that all requests come from a small number of hosts.

Backstory: I do primary testing of the service using ELB on EC2, and the traffic does not spread evenly (most traffic is at 1-2 nodes, almost no traffic to 1 node, no traffic at all to 4th node). And so the above problems are the possible culprits that I have identified.

Answer 1

I had very simulated problems. It’s one thing that ELB does not scale well with packet load. Therefore, when you try to test it, it does not scale immediately. It takes a long time to move up. Another drawback is the fact that it uses CNAME to look up DNS. Only this will slow you down. There are more performance issues that you can investigate.

My recommendation is to use haproxy. You have much more control and enjoy performance. I was very pleased with this. I use heartbeat to set up a redundant server, and I'm good to go.

Also, if you plan to do SSL with ELB, you will suffer more, because I found that performance is below par.

I hope this helps. When it comes to this, AWS told me personally that ELB load testing really does not work, and if you plan to run under heavy load, you need to tell them so that they can scale you ahead of time.

Answer 2

You don’t say how many instances of jmeter you are using, but in my experience it should be about twice the number of AZ that you are scaling. Even then, you are likely to see unbalanced loads - it is very unusual to see that the scale scales exactly in your rear fleet.

You can help (a bit) by running jmeter instances in different regions.

Another factor is the duration of your test. ELB takes some time to scale - you can generally tell how many instances work by doing nslookup against the ELB name. Understand your scaling patterns and create tests around them. (So ​​if you want to add another instance to the ELB pool, enable a 25-30 minute warm-up test). You also get AWS to “preheat” the ELB pool, if necessary.

If your ELB pool size is sufficient for your test and can verify that the pool does not change during the test run, you can always try to run the tests directly against the ELB IP addresses, i.e. manually balance traffic.

I'm not sure what you expect from the second level of calls - if you open the connection and reuse it, it is obvious that there is no way to scale all instances without closing and reopening the connection. Do these calls work on the same set of servers or in a different set? You can create an internal ELB and use this endpoint to connect, but I'm not sure if this will help in the scenario you described.