What to Look for in Load Test Reporting: Six Tips for Getting the Data you Need

Looking at graphs and test reports can be a befuddling and daunting task – Where should I begin? What should I be looking out for? How is this data useful or meaningful? Hence, here are some tips to steer you in the right direction when it comes to load testing result management.

For example, the graph (above) shows how the load times (blue) increase [1] as the service reaches its maximum bandwidth (red) limit [2], and subsequently how the load time increases even more as bandwidth drops [3]. The latter phenomenon occurs due to 100% CPU usage on the app servers.

When analyzing a load test report, here are the types of data to look for:

• What’s the user scenario design like? How much time should be allocated within the user scenario? Are they geographically spread?

• Test configuration settings: is it ramp-up only or are there different steps in the configuration?

• While looking at the tests results, do you get an exponential growing (x²) curve? Is it an initial downward trend that plateaus (linear/straight line) before diving downwards drastically?

• How does the bandwidth/requests-per-second look like?

• For custom reporting and post-test management, can you export your test results to CSV format for further data extraction and analysis?

Depending on the layout of your user scenarios, how much time should be spent within a particular user scenario for all actions (calculated by total amount of sleep time), and how the users are geographically spread, you will likely end up looking at different metrics. However, below are some general tips to ensure you’re getting and interpreting the data you need.

Tip #1: In cases of very long user scenarios, it would be better to look at a single page or object rather than the “user load time” (i.e. the time it takes to load all pages within a user scenario excluding sleep times).

Tip #2: Even though “User Load Time” is a good indicator for identifying problems, it is better to dig in deeper by looking at individual pages or objects (URL) to get a more precise indication of where things have gone wrong. It may also be helpful to filter by geographic location as load times may vary depending on where the traffic is generated from.

Tip #3: If you have a test-configuration with a constant ramp-up and during that test the load time suddenly shoots through the roof, this is a likely sign that the system got overloaded a bit earlier than the results show. In order to gain a better understanding of how your system behaves under a certain amount of load, apply different steps in the test configuration to allow the system to calm down for approximately 15 minutes. By doing so, you will be able to obtain more and higher quality samples for your statistics.

Tip #4: If you notice load times are increasing and then suddenly starting to drop, then your service might be delivering errors with “200-OK” responses, which would indicate that something may have crashed in your system.

Tip #5: If you get an exponential (x²) curve, you might want to check on the bandwidth or requests-per-second. If it’s decreasing or not increasing as quickly as expected, this would indicate that there are issues on the server side (e.g. front end/app servers are overloaded). Or if it’s increasing to a certain point and then plateaus, you probably ran out of bandwidth.

Tip #6: To easily identify the limiting factor(s) in your system, you can add a Server Metrics Agent which reports performance metrics data from your servers. Furthermore, you could possibly export or download the whole test data with information containing all the requests made during the tests, including the aggregated data, and then import and query via MySQL database, or whichever database you prefer.

In a nutshell, the ability to extrapolate information from load test reports allows you to understand and appreciate what is happening within your system. To reiterate, here are some key factors to bear in mind when analyzing load test results:

• Check Bandwidth

• Check load time for a single page rather than user load time

• Check load times for static objects vs. dynamic objects

• Check the failure rate

• For Server Metrics – check CPU and Memory usage status

……………….

 

This article was written by Alex Bergvall, Performance Tester and Consultant at Load Impact. Alex is a professional tester with extensive experience in performance testing and load testing. His specialities include automated testing, technical function testing, functional testing, creating test cases, accessibility testing , benchmark testing, manual testing, etc.

Twitter: @AlexBergvall

New Load Script APIs: JSON and XML Parsing, HTML Form Handling, and more!

Load scripts are used to program the behavior of simulated users in a load test. Apart from native functionality of the Lua language, load script programmers can also use Load Impact’s load script APIs to write their advanced load scripts.

Now you can script your user scenarios in the simple but powerful language Lua, using our programmer friendly IDE and new APIs such as: JSON and XML parsing, HTML form handling, Bit-fiddling, and more.

Configuring a load test with multiple user scenarios

We recently had a great question come in from one of our customers that we thought we would share.

Question: Planning to run a test with 10.000 concurrent users spanning 4 or 5 user scenarios.  How do I configure a test to run with, say 35% of the load running user scenario 1, 35% running user scenario 2, 10% running user scenario 3 etc?

And, when running multiple scenarios, where each scenario consists of 2 or more pages, how can we see the performance (load time) of each page in each scenario?

Answer: Assigning a certain amount of the simulated users to each user scenario is something you do in the “Test configuration” section.

Just scroll down the page to the section called “User scenarios”, then click the “Add scenario” button to add a new user scenario to the test. When you have all the scenarios you want added, you can fiddle with the percentages to get the exact load allocation for each scenario that you want.

 

The load time of each page in a user scenario can be collected if you use the –

http.page_start() and http.page_end() functions

– inside the user scenario script. Read more about that here and here.

Example: page metrics

-- Log page metric
 http.page_start("My page")
 responses = http.request_batch({
    { "GET", "http://loadimpact.com/" },
    { "GET", "http://loadimpact.com/style1.css" },
    { "GET", "http://loadimpact.com/image1.jpg" },
    { "GET", "http://loadimpact.com/image2.jpg" }
 })
http.page_end("My page")

Using the above script as a user scenario would result in a plot-able page load time metric for a page called “My page”. The name of the page can be changed to whatever you want.

The Load Impact Session Recorder – Now Available as a Chrome Extension!

Start a load test with just a few clicks. Record all HTTP traffic and use the recordings to simulate real user traffic under realistic load.

The Load Impact Chrome extension will capture everything – every single thing being loaded into the browser as you click – including ads, images, documents, etc., so you get a far more accurate read of what’s going on.

Just press “record”, start browsing and when complete, the script will automatically upload to your Load Impact account.

Here’s how it works:

With the help of our Chrome extension, you can run up to 10 different users scenarios in each load test and simulate up to 1.2 million concurrent users.  You can also run the multiple user scenarios simultaneously from up to 10 different geographic regions in a single test (powered by Amazon and Rackspace).

Until now our session recorder required developers to go to our website and manually change the proxy settings in the browser or operating system to perform a recording. That was a bit of a hassle, and the proxy solution sometimes caused problems with SSL certificates.

The extension now automates the entire process, from recording traffic in a specific browser tab, to stopping, saving and sending the scrip to your Load Impact account for future use.

The Chrome extension is available free of charge from the Google Chrome Web Store and is easily ported to the Safari and Opera browsers.  An extension for the Firefox browser is planned for release early next year.

To use the Chrome extension, you will need to register for a Load Impact account at loadimpact.com.