iPhone 5 – Apple was ready for it this time

Apple handled the load during iPhone 5-launch

Load problems is almost a tradition when Apple launches new iPhones. When the iPhone 4s was launched the whole website went down and when Apple started taking preorders for iPhone 4 the Apple Store went down.

But no such problems when iPhone 5 was released together with new iPods, EarPods and iOS6. According to our measurements www.apple.com loaded just as fast as usual. Maybe Apple did their homework this time?

The average response time for www.apple.com in our measurements is between 30 and 100 milliseconds on a normal day. During the launch of the iPhone 5 there were no detectable differences in response time. As usual the Apple Store was closed during the event.

We’ll see what happens with reponse time when Apple Store opens up for preorders of the new iPhone 5 on September 14.

If you want to see the response time graphs for http://www.apple.com and Apple Store, check out the previous blog entry

iPhone 5 – is Apple ready for it, part 2

It is now about 2 hours after the iPhone 5 was revealed. Apple Store has reopened after being taken offline to get updated with new content.

We have not seen any major problems with the sites this time. Well, apart from Apple Store being completely inaccessible for a few hours while being updated of course (do they really have to do that, or is it some marketing stunt?  Seems a bit primitive to have to take the whole thing offline just to update some content).

The main website – http://www.apple.com – seems to have worked fine during the whole event. It remains to be seen if Apple Store works as well now after the event as it did before it, or if it will suffer from huge numbers of iPhone 5-shoppers checking out details and availability of the new phone.

Here are a couple of graphs showing response time statistics during the launch and a short while afterwards.

 

iPhone 5 – is Apple ready for it?

Most people seem to believe the new iPhone will be released on the 12th of September. For some time now we at Load Impact have been monitoring how fast Apple serves content to visitors at its website and Apple Store. It will be interesting to see if Apple has learned from the two latest iPhone releases and are able to keep the web sites up and responding quickly this time.

Apple normally closes its online store http://store.apple.com shortly before a new major release, but during the launch of the iPhone 4s Apples main website www.apple.com went down due to the sheer number of users accessing it simultaneously – http://www.computerworld.com/s/article/9220525/Apple.com_crashes_as_iPhone_4S_is_unveiled

Something similar happened when the iPhone 4 was launched, with Apple Store crashing in both the US and UK.
During this launch we will periodically be updating response time statistics for both www.apple.com and store.apple.com, showing if and how the increasing number of users affect the sites. As this event might be one of the biggest product releases ever, it will surely be interesting to see what happens.

Below you can find some graphs showing the current response times for Apple store and http://www.apple.com, as measured from two different locations (Sweden and Netherlands).

New updates

The world’s best load testing service gets even better!

Yesterday, on Aug 2, we pushed a small update that included the following features and changes:

  • Ramp-up restrictions removed
    You can now ramp up or down the number of simulated users in a test as quickly as you like (well, almost – a ramp operation can be done in as little as 1 minute now).
  • Ramp steps can be any length
    Previously, a single ramp-up/down step could have a duration of max 120 minutes. This limit has been removed. There is now just a single limit for the whole test schedule, which can be max 24 hours (1440 minutes).
  • Changed default values for tests
    All defaults for test configurations, are now to ramp up to 50 VU during 10 minutes (previously, an automatically created test configuration would ramp up to 50 VU during 15 minutes, while a new test configuration created by a user would by default ramp up to 25 VU during 10 minutes).
  • New load script API functionality: auto_cookie_handling
    There is now a new boolean option that can be set with http.set_option() – “auto_cookie_handling”. It is set to true by default, but if set to false it will turn off the automatic handling of cookies between requests, allowing the script programmer to design his/her own cookie management.
  • Load generator bug fix
    Fixed an intermittent bug caused by issuing sleep statements with sleep time set to zero.
On June 20, we pushed another small update that included these changes:
  • Several proxy recorder bug fixes:
    • Fixed problem with injected toolbar appearing in the wrong place
    • Fixed problem with extra HTML added
    • Fixed problem with proxy sometimes generating extra CRLF’s to requests
  • New page on site: The state of web readiness 2012
    http://loadimpact.com/readiness
  • Company address on invoices
    You now get your company’s address on your receipts/invoices, viewable online at loadimpact.com
On June 8, we released Load Impact 2.4 that contained the following fixes and improvements:
  • New load script API functionality
    In this release we introduced a range of new API functions:

    • client.get_user_scenario_name() – returns name of currently executing user scenario
    • client.get_time_since_start() – returns elapsed time since the simulated user started executing
    • client.get_source_ip() – returns the source IP address seen in network traffic generated by this user
    • test.get_name() – returns the name of the currently running load test
    • test.get_time_since_start() – returns elapsed time since start of test execution
    • util.unique() – returns a string guaranteed to be unique across simulated clients in a test
  • Extra IP addresses
    You can now configure your load test to use more source IP addresses when generating traffic. This comes at an extra cost as it requires more infrastructure (cloud) resources, but can be very useful for e.g. spreading traffic evenly if you have a load balancer.
  • Small UI changes
    Several minor UI tweaks & fixes:

    • Changed “Test title” to “Test name”, for consistency
    • Fixed inconsistent naming of load zones. Load zones are now named as: CITY, COUNTRY CODE (PROVIDER)
      E.g. “Ashburn, US (Amazon)”
  • Bugfixes
    • Fixed broken resend email activation link
    • Fixed bug allowing tests to be scheduled up to 1 hour in the past
    • Fixed pagination bug in URL table on test results page
    • Fixed deployment bug affecting graphical editor user scenarios containing Unicode characters
    • Fixed bug causing screen to gray out in certain cases when selecting script editor for new user scenario

The State of Web Readiness 2012

For the moment we are attending the O’Reilly Velocity Conference in Santa Clara, where we have launched our brand new report “The State of Web Readiness 2012”.  In short the report covers how robust sites are based on 8,522 load tests executed in 132 countries. We found out that the average site was load tested at up to 3.4 times the actual capacity. What does that mean? Well the short summary is that a large part of the websites in the world might not stand up to what site owners expect of them.

This is actual data from actual load tests conducted with our own cloud-based online load test tool and frankly, we were a bit concerned with the findings of our study. Not that we are surprised that websites go down when we need them the most. Even though web sites have been a mainstream occurrence for over 15 years, we don’t lift an eyebrow when Apple Store crashes when a new iPhone-model is released. And if even the largest company in the world isn’t able to provide a premium sales channel that performs reliably, then who is, right? It almost seems unavoidable that websites go down. Like a natural disaster you can’t prepare for.

Our analysis indicates something else. After going through 8,522 actual tests we believe that you can be prepared with the right knowledge. The analysis shows that an important factor in the unreliable web is simply overconfidence about how many visitors websites can really handle. If you haven’t done the tests and you still think your website will continue to work unaffected during a hot product launch, a seasonal peak in interest or if you are luckily beeing “slashdotted”, think again!

Have a look at our report here. And we’re looking forward to hear more about what you think about the state of web readiness.

New partner in Benelux

Today we’re proud to announce that we have signed a premium partnership agreement with Systemation in Benelux, one of the leading distributors in integration testing and quality assurance in northern Europe. 
As you might already know, we provide a full-scaled partner program in priority markets in the U.S., Europe and Asia. Together with partners like Systemation we can reach out much faster to clients in need of efficient load testing. As a Premium Partner, Systemation will actively represent, sell and implement customer projects for the Benelux market.
Systemation acts as distributor in the Benelux countries for several leading international software companies like Load Impact. Together with the software solutions Systemation provides their clients with a comprehensive suite of professional services, including implementation, local technical support and software maintenance.
Jaap Franse, Managing Director of Systemation, states the following concerning our partnership:
“We are proud to add the world class Load Impact solution to our portfolio of services. This marks a significant step in supporting our customers when executing performance tests for their business critical web applications.”
So if you’re active in the Benelux market and in need for performance testing, contact Systemation directly.

Monthly visits and concurrent users

This post has been updated. It’s now much easier to calculate concurrent users using your Google Analytics data. Follow this 3 steps guide.

———

To determine how much traffic (how many visitors) your site can handle, you usually run a load test that simulates a certain number of users accessing your site at the same time, then reports how fast your site serves web pages to those simulated users. Many people encounter a problem when they want to do this; Most, if not all, load testing systems want you to specify how many concurrent simulated users should be used in the load test, but most people only know how many visitors their website has per day or per month. The question that comes up is then “how to convert visits per month to concurrent users?”

If we start with the source of information about site visits, Google Analytics is very popular as a means of keeping track of website traffic. It shows you lots of interesting information about your visitors and what they do on your site, and is used by a lot of site administrators today. Because of this, I will use Google Analytics in my examples below.

One thing you can see on the Google Analytics dashboard is the number of visits your site has had the last month (“Visits”). This is of course interesting as it gives you an idea of how much traffic your webserver has to handle in a month. However, it doesn’t tell you the peak traffic load your server has to handle at some specific point during the month. Most people will want to make sure their server can handle the peak traffic load (and more) so as not to risk losing business because of slow page load times.

Peak traffic can vary a lot between days, depending on many different circumstances. To view the number of visits per day in Google Analytics, you just click the “Visits” link on the dashboard (circled on the image above). For our own site –loadimpact.com – we see that high-traffic days (for us typically tuesday-thursday) see 100-200% more traffic than low-traffic days (typically saturday and sunday). Your mileage may vary, of course. It all depends on the type of site and the type of users. Below we have selected a 7-day period and clicked on the “Visits” link for our example site.

We can see in this example that on monday, january 26, the site had a traffic peak with 622 visits.

If you find a single day where you have a lot of traffic, you can then proceed to find out something that is even more interesting – namely what the peak traffic is for any single hour that day. This is a bit more complex in Google Analytics, however – you have to create a custom report. The custom report interface is available through a link in the left hand menu:

When you’re in the custom report interface you have to create a new custom report, then select what things you want your report to contain. You can specify different Metrics and Dimensions to use in the report. We will specify a single Metric and a single Dimenion in our report. The Metrics and Dimensions are grouped in different categories. The categories we will use are “Site Usage” and “Visitors”. You can see them circled on the image below.

First, we click on the Metric category “Site Usage”, which brings up a range of metrics we can use. Scroll down until you find the “Visits” metric, then drag it over to the empty metric box to the right.

Then close the “Site Usage” metric category and instead open the “Visitors” dimension category. Under that category you’ll find a dimension called “Hour of the day”. Drag this dimension over to the empty dimension box to the right.

Now you have a report that can show you the number of visits distributed over the different hours of the day, for the time period you have selected. Save the report and try viewing it.

Generating this report for the day you found that had the highest number of visitors, will likely show what the peak number of visits per hour your site has. Maybe you will see that at 3pm on that day, you had 1,000 visits. This value, plus a little margin, is very possibly the target traffic level you want your site to be able to support (while still responding reasonably fast, so users don’t have to wait for pages to load).

On the screenshot above we can see that on the day selected, we got the most visits at 1600 hours (4:00 pm to 4:59 pm). This is the traffic for the peak hour of the peak day, so it is a fairly good approximation of the maximum traffic our site has seen.

When you have determined what number of visits per hour you want your site to be able to handle, you should of course test if your site can handle that level of traffic (with reasonably fast page load times). To do this, you have to run a load test. The problem then is that most, if not all, load testing systems will want you to specify how many concurrent (“simultaneous”) users you want to subject your site to. They don’t talk about users or visits per monthper day, or even per hour. So how do we translate those figures to concurrent users?

Converting visits per hour to concurrent users:

concurrent_users = (hourly_visits * time_on_site) / 3600

So, if you have 1,000 visits per hour, and each visitor stays on the average 3 minutes (180 seconds) on your site, that means you would have (1000 * 180) / 3600 = 50 concurrent users.

Note that you need to use “visits”, and not “unique visitors”, when calculating the number of concurrent users. A single physical person may visit a site twice during an hour, which will of course cause twice the load on the web server. So we want to count the number of times the site has been visited, and not the number of unique persons that visit the site.

Time on site

The time on site parameter is the average time a user spends on the site. This is also something Google Analytics will tell you. It is the “Avg. Time on Site” value shown on the dashboard.

To translate visits per month to concurrent users:

concurrent_users = (monthly_visits * time_on_site) / (3600 * 24 * 30)

So, you just divide by the number of seconds in a month, rather than as before the number of seconds in an hour.

Now, when you know how many concurrent users you want your site to be able to handle, you can set up your load test. If, for example, you calculate that your site experiences a maximum of 50 concurrent users, but you want to test that the site can handle occasional peaks of, say 50% more traffic, then you want to verify that your site can handle up to 75 concurrent users. A reasonable load test setup might then be a ramp-up load test that tests four different load levels, starting at 25 concurrent simulated users, ramping up to 50, 75 and finally 100 concurrent users. That test would show you what response times (page load times) users would experience in low (25 users) and high (50 users) traffic conditions. The load levels 75 and 100 simulated users would also show what happens when the site/service grows or you see an exceptional burst of traffic due to some external event (maybe a big news blog writing an article about your site, generating a lot of extra traffic).

Thanks to Ditlev at VPS.NET who gave us the idea for this article. He actually wrote one of his own also, that you should look at if nothing of this made any sense. It is available here

About Load Impact

Load Impact is the leading cloud-based load testing software trusted by over 123,000 website, mobile app and API developers worldwide.

Companies like JWT, NASDAQ, The European Space Agency and ServiceNow have used Load Impact to detect, predict, and analyze performance problems.
 
Load Impact requires no download or installation, is completely free to try, and users can start a test with just one click.
 
Test your website, app or API at loadimpact.com

Enter your email address to follow this blog and receive notifications of new posts by email.