Mobile Device Testing

There has been a lot of interest recently in how to handle quality issues (content, performance, server capacity) for web applications that run on mobile devices. From the eValid perspective the main issues work out to these:

• Device Simulation: Using the eValid SetUserAgent command to set the User Agent String (UAS) so that the subsequent browser activity is reported to the server as if it was being requested from the specified device. eValid does this without needing the actual hardware and software! The server responses to client-side user requests made in this way behave as if it was talking to the mobile device.
  

• Content Validation: Once a script is running, you need to validate that the eValid rendered output matches what you expect in the actual device. Here is a Rendering Comparison for Common Devices Vs. Device Emulator. As you can see, other than the screen size, the rendering by eValid is identical to that for the mobile device. This confirms that evalid is rendering your mobile application correctly.
  

• Functional Testing: To illustrate how this works we ran an experiment that applied a simple test to 100 Mobile Devices, and recorded download sizes and times for each cases. A sample of that data is shown here, with some Selected Screenshots of how the server content varied as a function of the different devices and software versions simulated (with download byte counts for each device shown).
  
  

  The most interesting thing we saw was that the asymmetric ratio of Download Bytes vs. Download Time, which illustrates that some servers are not prepared to deliver mobile content to every kind of device.

• Server Loading: Functional tests lift easily to a server loading context, with multiple eValid instances (we call them Browser Users or BUs) working in parallel to create the load. This is illustrated in the experiment that we ran to drive a mobile application from a simulated smartphone up to 1,000 BUs -- up to 1,000 simulated users. The resulting
  Chart of Derived Internal Response Times shows that this particular server began to have a problem beyond ~200 simultaneous users.
  

The thing to remember is that eValid can effectively drive mobile devices that support web applications perfectly well, and can impose realistic load levels sufficient to identify bottlenecks.