Stress testing, Load testing, Performance testing, Capacity testing…they all are based on the same idea:

• Software systems are usually built to be used, and chances are that, at one point or another, you will want to know how your system reacts on heavy loads, and, in case the results are not satisfactory, you will want to improve the systems performance.

A first question arises:

Should it be done at some particular phase or should it be triggered by some specific requirement?

There is probably no reason why load testing should not be done since the very genesis of a software project, but in most cases, load testing is almost always considered to be one of the things to do “later”.

One reason for this is that load tests are considered to be useful only once a representative percentage of the system functionalities to develop have been produced.

This can be true to some point, but in most cases, load testing can probably be introduced in the build-and-release cycle at a very early stage.

Even at very early project stages, changes can have a big impact on the current and future system performance.

Another common reason for delaying load and performance testing is the non availability of “goal” load data, such as number of concurrent users, the number of concurrent users at peak time, number of transactions per unit of time, response times, message loads, etc…

Load testing can be performed without a goal, by stressing the application in order to obtain a benchmark.

In this manner, we can determine the maximum load for a certain deployment scenario in which metrics (such as average transaction processing time, average response times, etc) are still “acceptable”.

Another reason for delaying performance tests is priority and resources: other tasks, such as functional testing or development of new functionalities are normally considered to be more important, and establishing a load test platform can be costly.

Delaying performance and load testing should be considered a risk: fixing performance bottlenecks can have an impact on the project architecture, so the sooner the bottlenecks are detected, the better.

Load tests can be done after a nightly build for example, and results can be interpreted the next day, triggering necessary actions if needed.

On small scale systems, it is probably even feasible to have a load test available on demand at anytime for the developer, who can react immediately to performance impacting code changes.

All this said, care should be taken to not undermine simplicity, maintainability and other good things in the name of performance, pre-optimization can be evil !

Benchmarking vs. Optimization

Probably the first important question to ask is whether we want to benchmark our system, or we want to optimize its performance.

In general, benchmarking can be quite simpler than optimization.

One possibility is to benchmark in early stages of the project, monitoring results in order to not stray to far away from “good” performance, and then go on to more complex optimization hunting once certain goals have been established.

Benchmarking

In this case, the idea is to provide a certain metric (average response time, transactions per unit of time, etc…) for a certain load, test plan and deployment scenario.

A minimal set of information resulting from the benchmark could be for example:

• Test Plan Details:
  • What test cases were run?
  • How was the test plan run? (Think Time, Randomness…etc.)
  • Etc.
• Deployment Details:
  • What infrastructure was used?
  • What software was used (Application Servers, OS, etc.) ?
  • What software settings were used (JVM settings, etc.) ?
  • What hardware was used (CPU, RAM, etc.)?
  • What architecture was used (Load Balancing, Clustering, Persistent / Non Persistent Queues, etc.)?
• Number of Concurrent Users, Clients, etc.
• Average Response Time, Transactions processed / unit of time, etc.
• Deviation

Some examples could be:

• For the web application ‘X’, for 50 concurrent uses running test plan ‘A’, the system delivers an average response time per page of 2.7s with a deviation of 0.5s, with a non clustered, non load balanced deployment on a Ubuntu Server with ‘X’ RAM, etc.
• For the messaging system ‘X’, for a workload of test messages ‘X’, with non persistent queues, with remote-local queuing deployed on ‘X’ delivers a throughput of XXX transactions per second…

Optimization

In case of optimizing the system to attain a certain goal, the idea is to identify performance bottlenecks by collecting analysis data for all components involved in the system.

This can mean for example asking questions such as:

• Deployment-related:
  • Are software components up to date?
    (Such as data access drivers, third party libraries, application servers, web servers)
  • Are they scaled and configured for performance?
    (Adequate JVM heap memory settings, remove unused modules, etc.)
  • Etc.
• System Architecture-related:
  • Are performance scenarios such as load balancing, clustering considered?
  • Are there any external system dependencies that slow the global system down?
  • Is the global system architecture too complex?
  • Etc.
• Infrastructure-related:
  • Are memory, CPU…etc. resources sufficient?
  • Is network latency ok?
• Data-related:
  • Can data access be improved?
• Codebase-related:
  • Can the code be optimized?
    (Data retrieval optimization, Memory use improvement, HTTP request improvement, Thread use improvement, etc.)

To sum up, depending on the complexity of the system, there will be a lot of places were performance bottlenecks can occur.

The solution to the bottleneck may be quite simple (modification of a deployment setting for example) or quite complex (re-engineering of the global architecture for example).

Fortunately, there is a huge amount of ready available tools that can be used to monitor and inspect your system in order to answer these and other questions, and will do almost all the hard work for you.



Tools

Depending on the needs (benchmark or optimization) more or less tools will be needed, and they will have to offer a smaller or bigger feature set.

In the simplest benchmarking scenario, a single tool could be enough to record and replay scenarios, and monitor and deliver the system’s behavior.

In a more complex optimization scenario, a range of different tools will probably be needed, in order to stress and monitor the whole system.

General Criteria

Some general criteria for tool selection could be:

• Cost
• Number of Supported Platforms
• Tool Track Record (Proven? Widespread Use?)
• Documentation Quality
• Lightweightness
• Learning Curve and Ease of Use




Load Generation Criteria

Regarding load generation, the basic process will probably be:

• Record certain scenario(s)
• Store the scenarios
• Replay the scenarios in a certain manner in order to generate more or less load

Load testing tools will offer more or less features regarding these functionalities.

For example:

• If a large amount of concurrent clients must generate a huge amount of data, then some sort of coordinated load generation agents must be used due to hardware restrictions. If the amount of concurrent clients is low, this may not be needed.
• If scenarios are rather complex, Data Driven Testing (data separation from scenario) could be needed
  Required test execution features can vary (user think time, randomness, Etc)
• Etc.




Monitoring Criteria

Potentially all elements of the system can be source of a performance bottleneck, and thus, all elements should probably be monitored.

For example:

• Software Infrastructure logging can be a great source for detailed information of what is happening
• Almost all vendors offer some sort of monitoring tools for their products
• Server-side resource level monitoring where the application components are running (CPU starvation, Swapping,…etc)
• Profiling tools can be used to where precious CPU cycles are going
• If their really is no suitable tool for your system, perhaps creating one that suits your needs is the way to go
• Etc.




Good luck!



Improving performance is usually no easy task, but can be a great and rewarding challenge.

Finding resources on this topic should not be a problem, and these initial resources centered on web applications can be a good starting point:

• 15 free functionality and load testing tools for web applications
• Revisiting performance fundamentals by Joey Bernal




Related posts:

1. Web Application Load Balancing and In Memory Session Sharing with Tomcat and Apache
Tags: Capacity Testing, Load Testing, Performance Testing, Stress Testing