Load Testing Mobile Web Applications

With all of the recent interest in mobile devices – including all kinds of smartphones and hand-held tablets – we decided to provide readers with a short description of eValid's capabilities in "mobile device testing".

• Technology
  The basis for the eValid approach is to simulate ANY mobile device in a realistic way. Test runs done by eValid, when it is imitating or simulating a particular mobile device, appear to the supporting web application server as if the actual device is being used. Server response times to the device being exercised in a test session – mainly intra-browser timings – are very accurate (1 millisecond resolution). The big advantage is that you can test multiple devices from a single solution platform. This makes comparing different devices' performance very easy. All you do is change the simulation specification and re-run your functional test.
• A Typical Mobile-Device Loading Experiment
  As a use-case example, we did an experiment simulating an iPhone navigating to a popular website that delivers transportation system departure times, based on the requester's location, and retrieved from a database that tracks all of the available vehicles.
  1. Using a short self-synchronizing test script that asked a very simple and standard question from the database: "When is the next vehicle departure?" The script recorded a single data point: the time the database takes to retrieve the data to the user. For this application this is a Key Performance Indicator (KPI).
  2. We ramped the load from 1 Browser-User (BU) to 1,000 BUs by adding BUs at 1-second intervals. It took about 16 minutes to reach the 1,000 level. Once there, we held the imposed load at 1,000 BUs, all repeating the same test script as rapidly as possible for another 15 minutes.
  3. We used an Amazon cloud-based "Quadruple Extra Large" machine as the test driver. During the experiment the driver machine had only about 15% CPU usage and < 5% network usage at the full 1,000 BUs level. Overall, we collected data from over 5,900 KPI timings during the entire run.
  4. Here's the chart that shows the KPI vs. imposed load: 1,000 BUs Chart You can see from the chart that response time is constant up to about 200 users and then grows linearly with the increasing load.
• Detailed Example Explanation
  Here is the detailed Experiment Description that describes all of the above in complete detail, and also includes the actual eValid test script used.
• Webinar YouTube Movie: "Load Testing Mobile Web Apps"
  Here is the webinar movie associated with this experiment: Mobile Testing Web Apps [Webinar, 39:07].

You may also want to see the earlier eValid Blog item that deals with related aspects of testing of mobile web applications: Testing Mobile Device Web Applications.

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.

•  What is "Real User Monitoring" (RUM)? -- How monitoring works when you drive your site with a simulated real user? 
• How do I convert the old QTP tests? -- Considering how much more complex QTP is vs. eValid, it's probably earier to... 
• Modify our JavaScript to make our application more testable -- Simple rules to make life easier for testers. 
• Adaptive Playback -- How this important feature extends the life of test scripts. 
• eValid browser cache -- Details on how eValid handles caching. 

SalesForce Example Runs At Full Speed

We have heard from several users and potential users asking whether the SalesForce example script we used in our SalesForce Example 1 writeup could be played back faster? And if so, how much faster?

The recording from life that you see played back in the YouTube movie is running in "real time" -- every delay made during the recording is faithfully reproduced. The total script playback time is 310.1 seconds; the duty cycle of that playback is 20%, meaning that 80% of that time is spent waiting (expending "think time").

Can it be run faster? The answer is yes!

By adjusting the Playback Wait Time Multiplier to 0 the playback runs at a 95% duty cycle, with a total playback time of 59.8 seconds. There's no unnecessary waiting at all by the playback engine; all of the waiting time is for the SalesForce application and website to respond.

Here is the Original YouTube Movie and here is the adjusted Full Speed YouTube Movie. Viewer warning: the full speed movie goes by very fast, but rest assured that every test step is there. For a play by play sequence of screenshots you can review the complete Screen Image sequence, which is the same for either version.

Standard Speed Playback

Full Speed Playback

SalesForce Example Illustrates eValid Operation

The www.SalesForce.com application, which represents one of the earliest "software as a service" offerings, is well known for its very sophisticated use of JavaScript. Because of its internal complexity, and also in part because the application is under constant modification and update, successful functional testing of the SalesForce application has always been challenging.

Test Script Creation and Playback
The test plan for this example is to log into the account, create a new unique user, add some data to that user's record, confirm that the new user data is present, and then log off. The completed Example 1 Test Script recording was made "from life" using the AJAX Recording Protocol. Here is the Complete Example 1 Explanation.

Playback
Here is the sequence of Screen Images taken after each step in the test plan. You can watch a movie of this eValid test scrip playing back (click to view):

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.

•  Some data keeps being loaded into the cache? -- What is and isn't added to the Cache and when? 
• Where a server will reject requests? -- If it doesn't like you, actually your browser, the server can decline to play! 
• Measure response times -- More about time measurement at the client side. 
• What license limitations? -- A simple, practical limit on EVAL and DEMO licenses, explained. 
• The future of FLASH/Flex and SilverLight -- No comment, but consider the implications of HTML5!

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.

•  Get detailed timings from a monitoring test -- Details of measuring playback timings in GWOS/eValid monitoring. 
• Does every command eValid record play a part in playback? -- Some of the actual groundrules for EventLog messages. 
• What is the highest synchronization check rate? -- Why eValid's maximum sync rate is 100 Hz. 
• Have you ever tested some of GWT's more exotic features? -- How to handle GWT's exotic/troublesome "features"! 
• About synchronization of AJAX application test playback -- It's important! Without sync you're sure to have a test fail eventually.