What to Look for in Load Test Reporting: Six Tips for Getting the Data you Need

Looking at graphs and test reports can be a befuddling and daunting task – Where should I begin? What should I be looking out for? How is this data useful or meaningful? Hence, here are some tips to steer you in the right direction when it comes to load testing result management.

For example, the graph (above) shows how the load times (blue) increase [1] as the service reaches its maximum bandwidth (red) limit [2], and subsequently how the load time increases even more as bandwidth drops [3]. The latter phenomenon occurs due to 100% CPU usage on the app servers.

When analyzing a load test report, here are the types of data to look for:

• What’s the user scenario design like? How much time should be allocated within the user scenario? Are they geographically spread?

• Test configuration settings: is it ramp-up only or are there different steps in the configuration?

• While looking at the tests results, do you get an exponential growing (x²) curve? Is it an initial downward trend that plateaus (linear/straight line) before diving downwards drastically?

• How does the bandwidth/requests-per-second look like?

• For custom reporting and post-test management, can you export your test results to CSV format for further data extraction and analysis?

Depending on the layout of your user scenarios, how much time should be spent within a particular user scenario for all actions (calculated by total amount of sleep time), and how the users are geographically spread, you will likely end up looking at different metrics. However, below are some general tips to ensure you’re getting and interpreting the data you need.

Tip #1: In cases of very long user scenarios, it would be better to look at a single page or object rather than the “user load time” (i.e. the time it takes to load all pages within a user scenario excluding sleep times).

Tip #2: Even though “User Load Time” is a good indicator for identifying problems, it is better to dig in deeper by looking at individual pages or objects (URL) to get a more precise indication of where things have gone wrong. It may also be helpful to filter by geographic location as load times may vary depending on where the traffic is generated from.

Tip #3: If you have a test-configuration with a constant ramp-up and during that test the load time suddenly shoots through the roof, this is a likely sign that the system got overloaded a bit earlier than the results show. In order to gain a better understanding of how your system behaves under a certain amount of load, apply different steps in the test configuration to allow the system to calm down for approximately 15 minutes. By doing so, you will be able to obtain more and higher quality samples for your statistics.

Tip #4: If you notice load times are increasing and then suddenly starting to drop, then your service might be delivering errors with “200-OK” responses, which would indicate that something may have crashed in your system.

Tip #5: If you get an exponential (x²) curve, you might want to check on the bandwidth or requests-per-second. If it’s decreasing or not increasing as quickly as expected, this would indicate that there are issues on the server side (e.g. front end/app servers are overloaded). Or if it’s increasing to a certain point and then plateaus, you probably ran out of bandwidth.

Tip #6: To easily identify the limiting factor(s) in your system, you can add a Server Metrics Agent which reports performance metrics data from your servers. Furthermore, you could possibly export or download the whole test data with information containing all the requests made during the tests, including the aggregated data, and then import and query via MySQL database, or whichever database you prefer.

In a nutshell, the ability to extrapolate information from load test reports allows you to understand and appreciate what is happening within your system. To reiterate, here are some key factors to bear in mind when analyzing load test results:

• Check Bandwidth

• Check load time for a single page rather than user load time

• Check load times for static objects vs. dynamic objects

• Check the failure rate

• For Server Metrics – check CPU and Memory usage status

……………….

 

This article was written by Alex Bergvall, Performance Tester and Consultant at Load Impact. Alex is a professional tester with extensive experience in performance testing and load testing. His specialities include automated testing, technical function testing, functional testing, creating test cases, accessibility testing , benchmark testing, manual testing, etc.

Twitter: @AlexBergvall

New Load Script APIs: JSON and XML Parsing, HTML Form Handling, and more!

Load scripts are used to program the behavior of simulated users in a load test. Apart from native functionality of the Lua language, load script programmers can also use Load Impact’s load script APIs to write their advanced load scripts.

Now you can script your user scenarios in the simple but powerful language Lua, using our programmer friendly IDE and new APIs such as: JSON and XML parsing, HTML form handling, Bit-fiddling, and more.

Automated Acceptance Testing with Load Impact and TeamCity (New Plugin)

As you know, Continuous Integration (CI) is used by software engineers to merge multiple developers’ work several times a day. And load testing is how companies make sure that code performs well under normal or heavy use.

So, naturally, we thought it wise to develop a plugin for one of the most widely used CI servers out there – TeamCity by JetBrains. TeamCity is used by developers at a diverse set of industry leaders around the world – from Apple, Twitter and Intel, to Boeing, Volkswagen and Bank of America. It’s pretty awesome!

The new plugin gives TeamCity users access to multi-source load testing from up to 12 geographically distributed locations worldwide, advanced scripting, a Chrome Extension  to easily create scenarios simulating multiple typical users, and Load Impact’s Server Metrics Agent (SMA) for correlating the server side impact of testing – like CPU, memory, disk space and network usage.

Using our plugin for TeamCity makes it incredibly easy for companies to add regular, automated load tests to their nightly test suites, and as a result, get continuous feedback on how their evolving code base is performing. Any performance degradation, or improvement is detected immediately when the code that causes it is checked in, which means developers always know if their recent changes were good or bad for performance – they’re guided to writing code that performs well.

 

Here’s how Load Impact fits in the TeamCity CI workflow:

 

 

Once you have the plugin installed, follow this guide for installing and configuring the Load Impact plugin for TeamCity. 

Guess when we will run our ONE MILLIONTH load test and win a Pebble Watch!

Updated: We hit a million at exactly 2014-02-23 22:36:18 UTC. The winner is the person who guessed – 2014-02-24 at 10:00 AM! Your Pebble watch is on the way sir 🙂

—-

In case you hadn’t noticed, we are about to hit our ONE MILLIONTH load test! Just kidding, we know nobody is watching the counter as closely as we are.

As we place bets internally about when we will actually reach this coveted number, we thought our customers might want to join in the fun too. Especially since there’s some nice swag to be won for having the closest guess.

All you have to do is leave a comment below with your guess of the DATE and TIME (to the closest hour) we will execute our one millionth load test. Or submit your answer via email to: support@loadimpact.com. 

The person with the closest guess will win a Pebble watch worth $249 USD.

The top 10 runner-ups, including the winner, will receive $100 of Load Impact test credits and a nifty Load Impact t-shirt.

Submit your guess in the comment section below or email support@loadimpact.com. We will reply to you directly if you’ve won.

Hurry, the counter is ticking….

Contest rules: No purchase necessary; contest starts 2014/02/19 and ends no later than 2014/06/30 or when Load Impact runs its one millionth load test (whichever comes first); participant must 18 or older and have a confirmed Load Impact account; method to enter is to leave a comment in the section below or email support@loadimpact.com; limited to one entry per person; value of the first prize is $350 USD; the winner will be selected by comparing their submission with the actual date and time Load Impact runs it’s one millionth load test. In the event of a tie, the entry submitted first will win.; The winner will be notified with a reply in the comment section below or a reply via email; in order to claim the prize the winner(s) must provide Load Impact a valid email address, telephone number and mailing address.

Contest administered by Load Impact, Götgatan 14, SE-118 46, Stockholm, Sweden.

Countdown of the Seven Most Memorable Website Crashes of 2013

Let this be a lesson to all of us in 2014. 

Just like every other year, 2013 had its fair share of website crashes. While there are many reasons why a website might fail, the most likely issue is the site’s inability to handle incoming traffic (i.e. load).

Let’s look at some of the most memorable website crashes of 2013 that were caused by traffic overload.

#7. My Bloody Valentine

February 2nd, obviously not so alternative shoegaze legends, My Bloody Valentine, decided to release their first album since 1991, and they decided to do so online. They crashed within 30 minutes.

In the end, most of their fans likely got hold of the new album within a day or two and the band, which clearly has a loyal fanbase, probably didn’t end up loosing any sales due to the crash.

#6. Mercedes F1 Team 

Mercedes F1 team came up with a fairly clever plan to promote their web content. In february, they told fans on Twitter that the faster they retweeted a certain message, the faster the team would reveal sneak preview images of their 2013 Formula One race car.

It worked a little too well. While waiting for the magic number of retweets to happen, F1 fans all over the world kept accessing the Mercedes F1 web page in hopes of being the first to see the new car. Naturally, they brought the website down.

“You guys are LITERALLY killing our website!” Mercedes F1 said via Twitter.

#5. NatWest / Royal Bank of Scotland

Mercedes F1 and My Bloody Valentine likely benefited from the PR created by their respective crashes, but there was certainly nothing positive to come out of the NatWest/RBS bank website crash. A crash which left customers without access to their money!

In December, NatWest/RBS saw the second website crash in a week when a DDOS attack took them down.

It’s not the first DDOS attack aimed at a bank and it’s probably not the last one either.

#4. Sachin Tendulkar crash

One of Indias most popular Cricketers, Sachin Tendulkar, also known as the “God of Cricket”, retired in 2013 with a bang! He did so by crashing local ticketing site, kyazoonga.com.

When tickets for his farewell game at Wankhede in Mumbai became available, kyazoonga.com saw a record breaking 19.7 million hits in the first hour, after which the website was promptly brought down.

Fans were screaming in rage on Twitter and hashtag #KyaZoonga made it to the top of the Twitter trending list.

#3. UN Women – White Ribbon campaign 

It may be unfair to say that this website crash could have been avoided, but it’s definitely memorable.

On November 25th – the International Day for the Elimination of Violence against Women – Google wanted to acknowledge the occasion by linking to the UN Women website from the search giant’s own front page.

As a result, the website started to see a lot more traffic than they’ve been designed for and started to load slowly, even crashing entirely.

Google had given the webmasters at unwomen.org a heads up and the webmasters did take action to beef up their capacity, but it was just too difficult to estimate how much traffic they would actually get.

In the end, the do-no-evil web giant and unwomen.org worked together and managed through the day, partly by redirecting the link to other UN Websites.

Jaya Jiwatram, the web officer for UN Women, called it a win. And frankly, that’s all that really matters when it comes to raising awareness for important matters.

#2. The 13 victims of Super Bowl  XLVII

Coca Cola, Axe, Sodastream, Calvin Klein had their hands full during Super Bowl XLVII. Not so much serving online visitors as running around looking for quick fixes for their crashed websites.

As reported by Yottaa.com, no fewer than 13 of the companies that ran ads during Super Bowl saw their websites crash just as they needed them the most.

If anything in this world is ever going to be predictable, a large spike of traffic when you show your ad to a Super Bowl audience must be one those things.

#1. healthcare.gov

The winner of this countdown shouldn’t come as a surprise to anyone. Healthcare.gov came crashing down before it was even launched.

It did recover quite nicely in the last weeks of 2013 and is now actually serving customers. If not exactly as intended, at least well enough for a total of 2 million americans to enroll.

But without hesitation, the technical and political debacle surrounding healthcare.gov makes it the most talked about and memorable website crash in 2013.

Our friends over at PointClick did a great summary of the Healthcare.gov crash. Download their ebook for the full recap: The Six Critical Mistakes Made with Healthcare.gov

There’s really nothing new or surprising about the website crashes of 2013. Websites have been developed this way for years – often with the same results. But there are now new methodologies and tools changing all that.

It isn’t like it used to be; performance testing isn’t hard, time consuming or expensive anymore. One just needs to recognize that load testing is something that needs to be done early and continuously throughout the development process. It’s not optional anymore. Unfortunately, it seems these sites found that out the hard way. A few of which will likely learn the lesson again in 2014.

Our prediction for 2014 is more of the same. However, mainstream adoption of developmental methodologies such as Continuous Integration and Delivery, which advocate for early and continuous performance testing, are quickly gaining speed.

A Google search trend report for the term, DevOps, clearly shows the trend. If the search trends are any indication of the importance being given to proactive performance testing by major brands, app makers and SaaS companies, we might only see half the number of super bowl advertiser site crashes in 2014 as we did last year.

Update following Superbowl XLVIII: According to GeekBeat, the Maserati website crashed after their ad featured their new Maserati Ghibli. And monitoring firm, OMREX, found two of the advertiser websites had uptime performance issues during the game – Coca-Cola and Dannon Oikos.

New Pay-Per-Test Credits ($1 = 1 Credit)

We recently made two big changes to our pricing.

1. We released monthly subscriptions

2. We converted our test credits to a $1=1 credit model.

If you already had credits in your account before this change, you will notice that the total amount of credits in your account has increased.

Don’t worry, we aren’t a failed state. Your new credit count buys you just as much load testing as you could do before.

We simply wanted to simplify the process of purchasing a single test. Now you can get an exact dollar price for the test you want to run. Easy-peasy-lemon-squeezy!

The price for a single test is based on two factors – load level and test duration.

Head of over to our pricing page to see how it works.

Unlimited Load Testing for Everybody (Subscription-Based Load Testing)

Ask and ye shall receive! You can now load test till your heart’s content using our new UNLIMITED subscriptions.

We know many of you want to make performance testing a regular, automated and continuous part of your development process. To load test DevOps-style.

That’s why we’ve launched unlimited monthly subscriptions. We want to help you achieve that goal in the best way we know how – by making our tool more accessible, easy-to-use and cost-effective.

In addition to our recently launched Continuous Integration plugin for Jenkins, as well as our open API and SDKs, our new subscription model is designed to help take the stress out of stress testing, the load-off of load testing.

Ok, no more lame word-play.

Basically, we hope our subscriptions will make it easier for you to guarantee the scalability of your system. To feel more confident in the performance of each and every release you make. And to do it all without the risk of overage fees.

If you’re not interested in a monthly subscription, you can still pay-per-test as well as enjoy our free subscription.

We’d love your feedback, please let us know what you think about our new subscription model.

Configuring a load test with multiple user scenarios

We recently had a great question come in from one of our customers that we thought we would share.

Question: Planning to run a test with 10.000 concurrent users spanning 4 or 5 user scenarios.  How do I configure a test to run with, say 35% of the load running user scenario 1, 35% running user scenario 2, 10% running user scenario 3 etc?

And, when running multiple scenarios, where each scenario consists of 2 or more pages, how can we see the performance (load time) of each page in each scenario?

Answer: Assigning a certain amount of the simulated users to each user scenario is something you do in the “Test configuration” section.

Just scroll down the page to the section called “User scenarios”, then click the “Add scenario” button to add a new user scenario to the test. When you have all the scenarios you want added, you can fiddle with the percentages to get the exact load allocation for each scenario that you want.

 

The load time of each page in a user scenario can be collected if you use the –

http.page_start() and http.page_end() functions

– inside the user scenario script. Read more about that here and here.

Example: page metrics

-- Log page metric
 http.page_start("My page")
 responses = http.request_batch({
    { "GET", "http://loadimpact.com/" },
    { "GET", "http://loadimpact.com/style1.css" },
    { "GET", "http://loadimpact.com/image1.jpg" },
    { "GET", "http://loadimpact.com/image2.jpg" }
 })
http.page_end("My page")

Using the above script as a user scenario would result in a plot-able page load time metric for a page called “My page”. The name of the page can be changed to whatever you want.

5 Websites That Crashed This Holiday That Could Have Been Avoided

T’was the season to deliver a seamless online user experience, to bring under two second response times to shoppers looking for the best pre and post Christmas sale. Except that it wasn’t. At least not for the following five companies.

Every Christmas, e-commerce, ticketing, flash-sale and other online businesses prepare themselves to meet the demands of expected visitor traffic. Most fair exceptionally well because they take the necessary precautions and run realistic load and performance tests well in advance.

Yet, as more and more traditionally offline services move online and consumer demand for faster response times increases, the post-mortem on websites that crash during the holiday rush draw ever more media attention.

The increasing media attention is also due in part to the fact that innovation in performance testing has dramatically reduced the cost of doing so and the proliferation of cloud-based tools make testing services accessible to every website owner within just a few minutes. Basically, there is really no excuse for crashing.

Here’s a recap of some of the websites that crashed on our watch this holiday. We definitely didn’t catch all of them, so please do share your stories in the comment section below. Moreover, as we are a Swedish based company, many examples are from Sweden. Feel free to share examples from your countries.

1. December 4th, Wal-Mart.com:

Wal-Mart went down for a brief period, about an hour, on December 4th. Admittedly, they did claim to have had over 1 billion views between Thanksgiving and Cyber Monday and to have had the best online sales day ever on Cyber Monday.

So, despite the brief downtime, we’ll give it to Wal-Mart. They did have a pretty massive load to bare and if anyone can take it and recover that quickly, it’s probably them.

Read more 

2. December 16th, Myer.com.au:

On boxing day, Australia’s largest department store group, Myer, suffered technical difficulties that prevented online purchases during the biggest shopping day of the season.

According to the media, Myer has pumped tens of millions of dollars into improving its website over the years. Despite boosting its technology, this isn’t their first crash during peak shopping periods. They also crashed in June when heavy customer traffic triggered a website failure half an hour after the start of the annual stocktaking sale.

Although Myer is pushing an omni-channel strategy and hoping to boost its online sales in the long-term, the website is only responsible for about 1% of the company’s business today.

Although online sales may not make up a significant part of business today, it would be wise not to deny the impact these constant crashes probably have on the successful implementation of an omni-channel strategy. Yet this is how Myer CEO, Mr. Brookes,  seems to be behaving when he made this odd statement about the recent boxing day crash.

“There will be no impact at all on our profitability or our overall sales”

Sure Mr. Brookes, if you say so.

Read more

3. December 25th, Siba.se:

The day after Christmas, Siba – one of Sweden’s largest electronic’s dealers –  crashed due to overwhelming visitor traffic. This in turn led to a social media storm of customers complaining that the site was down.

As a curtesy to those who were not able to access the site,  Siba directed visitors to its sales catalogue saying: “Oops, at the moment there is a lot of traffic on the site, but you can still read our latest catalogue and stay up to date through our Facebook page”.

Thanks Siba, reading about the sales I’m missing out on is totally the same as taking advantage of them.

4. December 29th, SF.se 

In the period between Christmas and New Year’s,  SF  – Sweden’s largest movie theatre chain – suffered continuous day long crashes and delays. This left many people unable to fill those long cold days, when not much else is going on, with a cozy few hours at the cinema. In fact, these “mellandagarna” (days between Christmas and New Year’s) are the busiest movie going days of the entire year.

Needless to say, people were very frustrated. Particularly because SF has a monopoly and if they go down there is pretty much no where else to turn to get your cinema fix.

Read more

5. January 1st, Onlinepizza.se:

For the third new year’s day in a row, Onlinepizza.se crashed due to heavy user load. This may seem trivial to some, but to Swedes it’s devastating. That’s because on new year’s day, Swedes eat pizza. It’s just what they do.

So, despite the nearly  30,000 or so pizzas sold that day through Onlinepizza.se, many hungry swedes were forced to brave the cold and wind and buy their pizza the old fashion way – in a pizzeria.

Read more

Some of the holiday website crashes described above are bearable; most of us can go without buying another electronic device or pair shoes for at least a few days. But not being able to cozy up in a warm cinema on days when it’s to cold to go outside and nothing else in the city is open is just disappointing. As is not getting a home delivered pizza when you just simply can’t stuff another left-over Swedish meatball down your throat.

The Load Impact Session Recorder – Now Available as a Chrome Extension!

Start a load test with just a few clicks. Record all HTTP traffic and use the recordings to simulate real user traffic under realistic load.

The Load Impact Chrome extension will capture everything – every single thing being loaded into the browser as you click – including ads, images, documents, etc., so you get a far more accurate read of what’s going on.

Just press “record”, start browsing and when complete, the script will automatically upload to your Load Impact account.

Here’s how it works:

With the help of our Chrome extension, you can run up to 10 different users scenarios in each load test and simulate up to 1.2 million concurrent users.  You can also run the multiple user scenarios simultaneously from up to 10 different geographic regions in a single test (powered by Amazon and Rackspace).

Until now our session recorder required developers to go to our website and manually change the proxy settings in the browser or operating system to perform a recording. That was a bit of a hassle, and the proxy solution sometimes caused problems with SSL certificates.

The extension now automates the entire process, from recording traffic in a specific browser tab, to stopping, saving and sending the scrip to your Load Impact account for future use.

The Chrome extension is available free of charge from the Google Chrome Web Store and is easily ported to the Safari and Opera browsers.  An extension for the Firefox browser is planned for release early next year.

To use the Chrome extension, you will need to register for a Load Impact account at loadimpact.com.