Load Testing Validates Performance Benefits of CDN – 400% Improvement (CASE STUDY)

Ushahidi used Load Impact to greatly improve the performance of its software. Through comparing “before” and “after” test results it was possible to see the performance impact of optimization efforts – like the use of a CDN. 

Ushahidi is a non-profit tech company that specializes in developing free and open source software for information collection, visualization and interactive mapping. Such software is deployed during disasters so that real time information can be shared across the web. Like WordPress, the software can be self hosted or hosted on the company’s server.

Case:

Ushahidi software is generally used for crisis and disaster situations so optimization is absolutely crucial. An earthquake reporting site based on Ushahidi software (http://www.sinsai.info/) received a spike in traffic after the earthquake and tsunami in Japan and it went down several times, causing service outage at the time the service was needed the most.

Ushahidi were interested in using a load testing tool to test the performance of their software before and after optimization efforts, to determine what effect the optimizatons had had.

Test setup:

There were four load tests run on two different versions of the Ushahidi software. The software was hosted on Ushahidi’s servers. The first two test runs used ramp-up configurations up to 500 concurrent users on the test sites to test performance differences between Ushahidi 2.0.1 and Ushahidi 2.1. The results were revealing, showing performance graphs that were practically identical. There hadn’t been any change in performance from 2.0.1 to 2.1.

From these tests, it was also found out that the theoretical total number of concurrent users for Ushahidi on a typical webserver is about 330 clients but may be lower, depending on configuration. Load times at the 330-client level were very high, however, and defining the largest acceptable page load time to be 10 seconds meant that a more realistic figure would be 100 concurrent users on the typical webserver.

Finally, Ushahidi wanted to measure the potential performance gain when using a CDN (content delivery network). The Ushahidi 2.1 software was modified so that static resources were loaded from Rackspace’s CDN service instead of the Ushahidi server, then the previous load test was executed again.

The result was a major increase in the number of concurrent users the system could handle. Where previous tests had shown a significant slowdown after 60-100 concurrent users, and an absolute max limit of about 330 concurrent users, the CDN-enabled site could handle more than 300 concurrent users before even starting to slow down. To find out the extreme limit of the site with CDN enabled, a final test was run with even higher load levels, and it was found that the server now managed to serve content at load levels up to 1,500 concurrent users, although with the same high load times as in the 330-client case with no CDN.

Service environment:

• Apache
• PHP
• MySQL
• Linux (CentOS 5.0)

Challenges:

• Find load limits for 2 different software versions
• Find load limits with/without CDN enabled for static files
• Detect potential problems in the infrastructure or web app before they affect customers

Solution:

• Run ramp-up tests with identical configurations on the 2.01 and the 2.1 software. See which one performs better or worse
• Run ramp-up tests with identical configurations on the 2.1 software with CDN enabled, and without CDNenabled. See which performs better or worse.
• Run final, large-volume ramp-up test for the CDN-enabled software, to find out its theoretical maximum concurrent user limit.

Results:

• Ushahidi found out that there was a significant performance gain when using CDN to serve their static files.
• Load test measured that performance increased by 300% – 400% when using the CDN
• Load times started to increase only after 334 concurrent users when using the CDN, and the server timed out at around 1500 concurrent users.
• Faster time to verify CDN deployment. Test also quantified % increase in performance which leads to justification for additional cost of CDN service.
• Test showed no changes in load time between version 2.01 and 2.10.

Is Your Application as Mobile and Global as You Claim it is? – Prove it!

Your application has been localized, your website is responsive, you’ve even built a mobile app – how about your performance?! 

It takes more than a mobile app, responsive design and localization to stay ahead of the game, make sure your performance can also meet the demands of an increasingly mobile and global user-base.

Regardless of whether your applications are in a highly virtual, cloud based environment or a self-hosted single datacenter, realistic performance testing must take into account all the complexities that exist between applications and end users. In today’s highly mobile world, users can literally be anywhere in the world coming across connections that vary widely in quality and speed.

A successful application deployment must take into account factors that influence this Quality of Experience (QX) and integrate continuous testing that best simulates a wide variety of situations.

Not long ago, load testing was a simple and typically one-time test done to size hardware before a roll-out. Testing was nearly always done in-house and did not take into effect what the end user experience was like and how those variables could significantly affect not only user experience but server resources as well.

Gone are the days of users only using your application from a desktop, connected to their DSL at home, and located within the same national borders as your business. Depending on who you ask, by 2020 75% of commercial transactions and 50% of consumer spend will be mobile.

Already today, mobile accounts for 25% of all web usage globally – and 80% for China only. With internet penetration soaring in countries like China, Indonesia and Brazil, its no surprise that nearly all big US-based internet properties are seeing a larger portion of their traffic and users coming from abroad.

The 2014 Mary Meeker Internet Trends report revealed that 6 of the top 10 US-based internet properties that have global operations have more than 86% of their users coming from outside the US.

This shouldn’t come as a major shock to most applications teams, those who now know they must design either a mobile responsive page or a mobile app in addition to the traditional desktop browser to stay competitive, let alone make sure that a users’ experience is consistent regardless of geographic location.

So if applications teams are so focused on designing around an increasing mobile and global user base, wouldn’t it make sense to performance test in that mode as well – using geographically distributed load, simulating mobile networks, browsers and connections?

Here are a few key considerations and benefits of what a global/mobile approach will bring:

 1.  Browser Simulation

Users interact with applications from a wide variety of desktop and mobile browsers (or apps) today and there are very real differences in how each use case impacts scale.  It’s not good enough to simply assume every browser will follow caching and compression directives the same and that TCP connections issues will be consistent across the whole user base.

Additionally you have to take into account iPhone and Android OS types and multiple browsers on each platform.  Bottom line here is to use multiple user scenarios that include different browsers and platforms mixed in!

A realistic testing platform should simulate both desktop & mobile browsers

2.  Network Connections

One thing that’s for sure these days is an inconsistency when it comes to how users connect to an application.  Some users will have super low latency, google fiber connections (one can dream) that probably eclipse your datacenter circuit performance and others will be on a roaming 3G connection with tons of packet loss.

Even more challenging is what happens when a mobile user hands off from cellular data to WiFi and what that means to server resources (think  lots of FIN & WAIT TCP states) and experience.  A realistic test should include simulations for multiple connection types – DSL, 3G, LTE, unlimited, etc.  Even better would be a system that can introduce jitter and packet loss to mobile connections for the ultimate in realism and impact to server resources.

Being able to simulate different connection types and associated connection quality is also important

3.  Geo-Distributed Users

Users are going to be geographically distributed for just about any application these days, even intra-net only corporate applications. And they should expect a great user experience regardless of where they are.  At a bare minimum, testing within the continent where 80% of users will be located is recommended – going global is even better.  Being able to test from multiple geographies simultaneously during a single test is very valuable since you can then see exactly the differences in performance and user experience with the only variable being the user location.

If users are primarily US based then test multiple locations within the US – at least

However if users (or company execs) frequently travel abroad then test it!

A great user experience (sub 1-sec load times for example) is great but if that performance drops off a cliff as you move away from the datacenter then looking into a CDN (or a better CDN) may become a high priority.  If you are using distributed server resources and a complex CDN strategy, this is a great way to validate that all is working properly and you are getting the best value from the provider of choice.

The bane of most Ops teams’ existence is the “the app is slow” ticket, and the last thing a user will want to hear from a support reply is “not from here it’s not!”  A great way to identify early potential performance issues on a geographic basis is to continually test (ok maybe hourly or daily) and automate that process.

If a baseline is created then when performance numbers well outside of that reference range occur you can be proactive and not reactive.  If performance is slow from users in the UK but no where else and you have a quantitative analysis in hand, discussions with hosting and CDN providers takes on a much more productive tone.  Think of all the unnecessarily steps and level-1 troubleshooting that can be eliminated, all potentially before the first support ticket is opened for the UK slowness that you already were working on.

Consistently slower page load times from Australia might mean it’s time for a new hosting resources or a CDN upgrade

With the tools available today, application teams have the ability to continuously test load and performance with highly realistic and sophisticated test scenarios. Performing this testing using a cloud based test platform removes on-premise test tool cost and deployment hassles and allows teams to test at every phase of a deployment including after the app goes live.

This type of approach can also help evaluate different hosting and CDN offerings well before the application goes live, and determine which providers offer the best value in the regions of the country or world you care most about. Taking a pro-active approach to monitoring the performance of applications, especially mobile applications where you are certain to face TCP connection issues, roaming from 4G to WiFI and a host of other mobile-centric challenges will go a long way to ensuring deployment success in a Dev-Ops fashion.

 

 

———–

This post was written by Peter Cannell. Peter has been a sales and engineering professional in the IT industry for over 15 years. His experience spans multiple disciplines including Networking, Security, Virtualization and Applications. He enjoys writing about technology and offering a practical perspective to new technologies and how they can be deployed. Follow Peter on his blog or connect with him on Linkedin.

Don’t miss Peter’s next post, subscribe to the Load Impact blog by clicking the “follow” button below. 

[Case Study] How One Digital Agency Guaranteed Performance BEFORE a Big Release

JWT, one of the largest advertising agencies in the United States and the fourth-largest in the world, used Load Impact to perform load tests to verify that their new campaign site could handle up to 120,000 visitors/hour.

Background:

According to an independent global research study undertaken by Vanson Bourne, even minor delays to website response times can have a sizable impact on customer satisfaction, page views, conversion rates and site abandonment. Despite this, an astonishing 68% of website owners experienced performance or stability problems and 32% of organizations do not know if their website is monitored on a 24×7 basis*. To make matters worse, 47% of PC visitors, 69% of tablet visitors and 34% of smartphone visitors expect response times equal to or below 2 seconds**. 

In an effort to ensure quality performance of a new campaign website built for a client in the pharmaceutical industry, Load Impact was commissioned to establish that the backend for the website could handle the expected traffic – 120,000 visitors per hour – while exhibiting load times that were within acceptable limits.

The campaign-site was built with a user signup/login procedure and offers an interactive online game. The backend for the service is hosted in Texas, and all static content is distributed through a CDN which makes calls to the backend servers. There is also an external partner which handles the user database including registration, signup etc.

Test setup:

For the purpose of testing the backend only, a set of specific user actions were defined, such as “user registration”, “user sign-in”, and other actions where the backend systems had to be involved. These actions were activated through requesting certain URLs, one for each type of action, that were created specifically for the load test. In practice it meant that a simple API was created only for running the load test.

The simulated users in the test were configured to perform a series of these predefined user actions, resulting in corresponding calls to the backend systems. The static content, normally served through CDN operators, was ignored in the test.

The test was run as a series of 5 minute ramp-up tests (simulating approximately 8.33 clients per second), individually tuned depending on results of previous tests and designed to find out the breaking point of the target system.  

Service environment:

• Apache
• PHP
• MySQL
• Linux

The tested environment consisted of an HTTP load balancer plus web-, application- and database servers.

Challenges:

There were numerous challenges that the test was designed to detect. First of all, there was a need to validate that the system could handle the expected amount of traffic and establish a performance baseline. The test was also set-up to detect potential problems in the infrastructure and web app. 

• Validate that the system could handle the expected amount of traffic
• Detect potential problems in the infrastructure or web app
• Establish a performance baseline

Solution:

The solution agreed upon was to generate and measure load from multiple geographic locations as well as to measure response times, throughput and customer experience.  

• Load generation and measurements from multiple geographic locations
• Application response time, throughput and customer experience analysis provided for JWT

Results:

The results of the load test revealed that the campaign website could withstand the expected traffic and there were no specific performance problems with the site. Therefore, a baseline was established at about the required level of 120k visitors/hour.

The external service provider of user registration and sign-in functionality had no problems and their response times remained constant during the tests while the other backend services exhibited response times that were stable until just over the required level of 120,000 visitors/hour, after which response times started to increase rapidly and exponentially.

Specifically, the response times for the start page were under 1 second for up to 2,000 concurrent visitors. Response times for the configured tests, which included the set of specific user actions, were under 2 seconds for up to 2,000 concurrent visitors. Considering that the average response time for similar campaigns of this size is above 4 seconds*, these results were impressive. 

The campaign site was launched successfully on YouTube.

*Source: State of Web Readiness Report, Load Impact, 2013

**Source: How To Deliver Fast, Engaging Responsive Web Design Sites, Akamai, 2012