Which is the better benchmarking strategy: local machine vs remote machine

 

today I had a debate which is the better benchmarking strategy: local machine vs remote machine. And I would like to know some other opinions on that topic. It is more general question, but in this case it was a java project.

The test was about how well some template engines perform. After a warming up phase of 30 seconds the benchmark runs 120 seconds with 100 threads.

The tested page has some static and dynamic parts. like random items in a table and list, but a static header for example (no database operations).

Senario 1 :

The webserver and the benchmarking tools runs on the same machine. The tests were made with ab and jmeter.

Pro arguments:

  • the production would also be not an ideal system.
  • other java process may also run on the production system.
  • even if the benchmarking tools slow down the result in average the results are still valid ( the better technology would be better on any variant)

    • Con arguments:

  • the same machine tries to beat itself
  • benchmarks slow down real performance

    • Senario 2 :

    A remote machine tests the other system:

    Pro arguments:

  • one system can try to reach the limits of the other
  • benchmarks (jmeter) does not run on the same jvm

    • Con arguments:

  • production system would not be an ideal environment
  • network may slow down the results

    • First I would like to know which is the better way to go. Second if some of the arguments are not valid. Or in the end it doesn't even matter. And I would like to know other arguments for any of both variants. I hope this question does not get closed, because I think it is not to general. One option must be the better one.

    Scenario 2, in almost all cases, is going to be closer to production behavior. I would argue that the cons listed for Scenario 1 effectively invalidate any of the results. However, as long as you are consistent with the approach you should be able to use results for comparisons even if they do not match what you expect to see from production.

    Addressing the cons of Scenario 2:

  • Production System would not be an ideal environment.

    • If you are about to load test production, something else is wrong to begin with.

  • Network may slow down results.

    • While this might be true, the networking for a distributed internal test environment is still better than your customers will have from external sources. In almost all cases you will want to do bandwidth network simulations to ensure production-like behavior.

    To accomplish this, look for the following line in the jmeter.properties file: 

    # Define characters per second > 0 to emulate slow connections
#httpclient.socket.http.cps=0
#httpclient.socket.https.cps=0

    This will limit the "bandwidth" of each JMeter thread. (quoted because the throttling does not happen at the network level) Do a simple study to see what CPS means in terms of your applications.

    The Short Answer:

    Always try to test in a distributed setup because you have more flexibility. Also, consider the scenario where you need multiple load generator machines to actually stress a target. In that case you have no choice at all, so I would argue "best practice" is to configure isolated test environments with separate load generator machines.

    Also, to effectively maximize the output of each load generator machine it is advised to customize open file descriptor limits (ulimits) and JVM memory settings for the load generator machines. In most cases those configurations are not acceptable for application servers.

    Hope that helps, please let me know if I can elaborate further, -Addled

    链接地址: http://www.djcxy.com/p/86526.html

    上一篇: 简单基准测试表现奇怪

    下一篇: 哪个更好的基准策略：本地机器与远程机器