06 August 2007 - 1.7.5 | home user-guide eclipse jbossws intellij netbeans maven 1.X/2.X PDF files forums bugs sourceforge

Home
Getting Started
User Guide

- Overview
  - Tabbed Desktop
  - License Management
  - Preferences
  - XPath Selection
  - Script Library
- Workspaces
- Projects
- Interfaces
  - Interface Viewer
- Operations
- Working with Requests
  - Form Editor
  - Outline Editor
  - Message Inspectors
  - Attachments/Inline Files
- Functional Testing
  - Getting Started
  - TestSuites
  - TestCases
    - Test Requests
      - Assertions
      - Editor Wizards
    - Property Transfers
    - Conditional Gotos
    - Groovy Scripts
      - Examples
      - GroovyUtils
    - Properties Step
    - Delay Step
    - Mock Response
    - DataSource
    - DataSource Loop
  - Property Expansion
- Load Testing
  - Getting Started
  - Limit & Strategies
  - Execution
  - Assertions
  - Diagrams
  - JMeter comparison
- Mocking
  - Getting Started
  - Mock Services
  - Mock Operations
  - Mock Responses
- Usage Scenarios
  - Data-Driven Testing
  - Template-Driven Testing
  - Interactive Testing
  - Surveillance Testing
- Tool Integrations
  - Code Generation
  - WSDL Generation
  - WS-I Tools
  - Apache Tcp-Mon
- CommandLine Tools
  - TestCaseRunner
  - LoadTestRunner
  - MockServiceRunner
  - ToolRunner
- Keyboard shortcuts

IDE/Tool Plugins
Search soapUI site
Project Documentation

- About soapUI
- Project Info
- Project Reports

## JMeter Comparison

This document aims mainly to compare the results obtained with the popular JMeter tool from apache with those obtained with soapUI and to show and discuss some of the differences in the results obtained with the 2 tools. If you think/know we are doing something wrong in our tests or conclusions, don't hesitate to contact us and we will try to sort things out :-)

A feature comparison will between the two tools will not be done in any detail, generally one could say that JMeter is much better at load-testing in general (ie HTTP, JDBC, JMS, etc, etc) while soapUI is better at load-testing web services specifically. Most things you can do with soapUI can probably be done in JMeter, but since JMeter has a more "generic" approach to almost everything, it won't always be as "intuitive" as it is (or at least should be) in soapUI. On the other hand, many things that can be done in JMeter are not possible in soapUI (for example distributed load testing).

In general one can say that Load-testing web services is extremely difficult since so many factors can affect the measured results; network latency, host-load, hardware, jvms, etc.. Both soapUI and JMeter will be a "victim" to this, although soapUI has slightly more options to control exactly what and how results are to be measured (in the TestCase options dialog).

### Comparison Setup

The setup for the comparison is a simple request to a web service running locally under an improved version of JBoss 4.0.3SP1 and running a slightly modified sample Web Service from the jbossws 4.0.3 samples. A simple request will be run 5000 times under 4 different loads; 1 thread, 5 threads, 10 threads and 25 threads all using the simple strategy with no delay, the TestCase Options are configured to include only request write times, and will be run with and without closing connections after each request.

This is in JMeter is configured equivalently using one ThreadGroup with no ramp-up and varying the loop count and number of threads accordingly; 1/5000, 5/1000, 10/500 and 25/200. The "WebService(SOAP) Request" sampler is used and configured as shown below. An "Aggregate Report" listener is configured for collecting the results.

My laptop is running WinXP SP2 on a Pentium M 1.8 with 1Gb of RAM. JRE 1.5.0_06 is used for running JBoss, JMeter and soapUI.

### Results

The table shows the results, for each test the average and TPS times are shown, (*) tests are run without closing connections

Threads | jmeter | soapUI | soapUI (*) | soapUI cmdline | soapUI cmdline (*)
--- | --- | --- | --- | --- | ---
1 | 8 ms, 105 TPS | 6.78 ms, 147 TPS | 10.7 ms, 94 TPS | 5.75 ms, 174 TPS | 10 ms, 99 TPS
5 | 43 ms, 110 TPS | 38.7 ms, 128 TPS | 23.7 ms, 211 TPS | 30.4 ms, 164 TPS | 24 ms, 210 TPS
10 | 86 ms, 112 TPS | 82 ms, 122 TPS | 46.5 ms, 215 TPS | 61 ms, 164 TPS | 38 ms, 262 TPS
25 | 214 ms, 114 TPS | 204 ms, 123 TPS | 124 ms, 202 TPS | 159 ms, 157 TPS | 95 ms, 263 TPS

This can be plotted as follows:

As can be seen in the graph and numbers, soapUI consistently gives a higher TPS value than JMeter, except when reusing connections and running only one thread. This may be a bit surprising, evidently it is faster for HttpClient and the JVM to create new connections than to pool existing ones.

The biggest difference between the two tools can be seen when switching to connection-reuse and running from the command-line. This is expected, connection reuse should have a big impact when running many threads, and running from the command-line uses no UI resources at all.

One could argue that closing connections after each request is a more "realistic" scenario for web services that serve a large number of different clients, for example an online weather service. Web services running internally in a SOA may be more likely to serve a smaller number of clients, so these could be tested with connection reuse enabled.

Looking through the JMeter sources, it seems as if JMeter uses HTTP 1.0 and does not maintain any internal connection pool for Web Service requests. Also, it uses `System.currentTimeMillis()` for measuring elapsed time instead of the `System.nanoTime()` feature available in Java 1.5 and used by soapUI.

### Further Analysis

Since the above results indicate an improved result when reusing connections, I ran a LoadTest in soapUI using the "Threads" strategy increasing the number of threads from 1 to 25 over a period of 250 seconds, both with and without connection reuse. The graphs below show the results:

First without connection reuse (click on image for full-size):

This confirms the previous results; TPS does not vary much when increasing the number of threads, the average request time "takes" the entire increased load.

And then with connection reuse (click on image for full-size):

The results here also confirmed those above; when reusing connections the TPS value increases steadily up to about 6 threads and then lies still. Also, one can see that the average request time starts increasing steadily first when the TPS has hit its peak, the average then "takes" all the increased load just as when not reusing connections at all.

### Conclusion

The values obtained with soapUI are almost consistently "higher", if this is better is a matter of discussion. One clear advantage of soapUI seems to be the possibility to reuse connections, which also makes a big difference when running many threads (ie simulating many clients).

Based on the above results my recommendation would therefore be the following:

- For behavioural and functional load testing of Web Services; start with soapUI, move to jmeter if you need its "high profile" features.
- For performance load testing of Web Services: use soapUI in conjunction with another tool (for example JMeter) and weigh the results!

### Files

If you want to run the tests yourself, the following files are available in comparison-files.zip:

File | Description
--- | ---
comparison-soapui-project.xml | soapUI project file containing all tests run in the comparison
soapui-compare.jmx | jmeter project file containing all tests run in the comparison
ws4ee-samples-server-ejb.jar | precompiled jar file containing tested Web Service, just drop this in the JBoss 4.0.3 deploy folder and the above tests should run as configured (you may need to change the endpoint)

You can download JBoss 4.0.3SP1 from here, but you can also use the 4.0.4RC2 release using the new JBossWS stack which gives slightly different results than those shown above (not shown here since this comparison is not meant to compare jboss versions)

Next: Web Service Mocking