Monday, April 29, 2013

Load Testing Mobile Web Applications

With all of the recent interest in mobile devices – including all kinds of smartphones and hand-held tablets – we decided to provide readers with a short description of eValid's capabilities in "mobile device testing".
  • Technology
    The basis for the eValid approach is to simulate ANY mobile device in a realistic way. Test runs done by eValid, when it is imitating or simulating a particular mobile device, appear to the supporting web application server as if the actual device is being used. Server response times to the device being exercised in a test session – mainly intra-browser timings – are very accurate (1 millisecond resolution). The big advantage is that you can test multiple devices from a single solution platform. This makes comparing different devices' performance very easy. All you do is change the simulation specification and re-run your functional test.
  • A Typical Mobile-Device Loading Experiment
    As a use-case example, we did an experiment simulating an iPhone navigating to a popular website that delivers transportation system departure times, based on the requester's location, and retrieved from a database that tracks all of the available vehicles.
    1. Using a short self-synchronizing test script that asked a very simple and standard question from the database: "When is the next vehicle departure?" The script recorded a single data point: the time the database takes to retrieve the data to the user. For this application this is a Key Performance Indicator (KPI).
    2. We ramped the load from 1 Browser-User (BU) to 1,000 BUs by adding BUs at 1-second intervals. It took about 16 minutes to reach the 1,000 level. Once there, we held the imposed load at 1,000 BUs, all repeating the same test script as rapidly as possible for another 15 minutes.
    3. We used an Amazon cloud-based "Quadruple Extra Large" machine as the test driver. During the experiment the driver machine had only about 15% CPU usage and < 5% network usage at the full 1,000 BUs level. Overall, we collected data from over 5,900 KPI timings during the entire run.
    4. Here's the chart that shows the KPI vs. imposed load: 1,000 BUs Chart You can see from the chart that response time is constant up to about 200 users and then grows linearly with the increasing load.
  • Detailed Example Explanation
    Here is the detailed Experiment Description that describes all of the above in complete detail, and also includes the actual eValid test script used.
  • Webinar YouTube Movie: "Load Testing Mobile Web Apps"
    Here is the webinar movie associated with this experiment: Mobile Testing Web Apps [Webinar, 39:07].
You may also want to see the earlier eValid Blog item that deals with related aspects of testing of mobile web applications: Testing Mobile Device Web Applications.

Tuesday, April 23, 2013

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.

Friday, April 12, 2013

SalesForce Example Runs At Full Speed

We have heard from several users and potential users asking whether the SalesForce example script we used in our SalesForce Example 1 writeup could be played back faster? And if so, how much faster?

The recording from life that you see played back in the YouTube movie is running in "real time" -- every delay made during the recording is faithfully reproduced. The total script playback time is 310.1 seconds; the duty cycle of that playback is 20%, meaning that 80% of that time is spent waiting (expending "think time").

Can it be run faster? The answer is yes!

By adjusting the Playback Wait Time Multiplier to 0 the playback runs at a 95% duty cycle, with a total playback time of 59.8 seconds. There's no unnecessary waiting at all by the playback engine; all of the waiting time is for the SalesForce application and website to respond.

Here is the Original YouTube Movie and here is the adjusted Full Speed YouTube Movie. Viewer warning: the full speed movie goes by very fast, but rest assured that every test step is there. For a play by play sequence of screenshots you can review the complete Screen Image sequence, which is the same for either version.

Standard Speed Playback
  Full Speed Playback

Tuesday, April 9, 2013

SalesForce Example Illustrates eValid Operation

The application, which represents one of the earliest "software as a service" offerings, is well known for its very sophisticated use of JavaScript. Because of its internal complexity, and also in part because the application is under constant modification and update, successful functional testing of the SalesForce application has always been challenging.

Test Script Creation and Playback
The test plan for this example is to log into the account, create a new unique user, add some data to that user's record, confirm that the new user data is present, and then log off. The completed Example 1 Test Script recording was made "from life" using the AJAX Recording Protocol. Here is the Complete Example 1 Explanation.

Here is the sequence of Screen Images taken after each step in the test plan. You can watch a movie of this eValid test scrip playing back (click to view):

Monday, April 8, 2013

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.

Monday, April 1, 2013

Selected Recent Forum Posts

Here is a selection of some of the posts that we think would be of general interest.