Tuesday, July 20, 2010

Current User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

Monday, July 19, 2010

Structural Testing: Making Web Tests More Durable

We often hear users complaining that their web pages changed and the tests that used to PASS yesterday now FAIL. Is there anything that can be done to have tests endure page changes that don't affect functionality, they ask?

The answer lies in the use of the Structural/Algorithmic Testing testing approach. Instead of basing test on properties of the page, when a particular feature is likely to move around, the test is based on (pivots on) that feature. The result (as the table outlines) is a little bit more work in exchange for a much higher level of durability -- without any loss in test quality (no increase in false-negative results).

A simple technique that we recommend, a Manual Script Creation Process, takes the mystery out of converting a "from life" recording into a highly durable validation test. Once the key page elements are identified, you modify the script using the Structural Testing Commands.

P.S. If this structural testing approach still isn't strong enough for your needs, with some additional programming effort you can exploit the eValid Programmatic Interface [EPI] to create a C++ program that runs your test at any level of durability you want.

Saturday, July 10, 2010

User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

Wednesday, July 7, 2010

User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

  • What's The Impact Of This On The "Reality" Of The Playbacks? -- The cache is a unitary resource, and eValid instances must share it during multiple-BU playbacks, but, this turns out to be a conservative assumption.
  • Making A Recording Versus Scripting A Test -- The eValid approach makes creating the script so easy, our recommended approach is to "record first, script later".
  • Why is having a lot of BUs running such a big deal? -- For many modern web applications -- especially those which use AJAX methods -- having an actual user do the test playback assures 100% realism.
  • Structural Testing For Web Apps -- There are special eValid "structural testing" commands that allow a script to be completely independent of inconsequential page changes.
  • A Sequence of HTTP References -- There is a special facility in eValid to generate a sequence of HTTP Get's that correspond to the actual sequence of page components that comprise a full page download.
  • Microsoft SkyDrive Applications -- The Microsoft SkyDrive based applications -- common office utilities like a word processor or a spreadsheet -- actually test quite well with eValid, but they are among the more challenging applications to test.
  • Link Checking -- There's a special eValid dynamic link check command that automatically visits every link on a page, even when the page is generated dynamically.
  • Constant ID Tags -- When a web page has permanently-assigned ID tags the theses can "pivot" on those internal identifies, but even when the ID tags are dynamic it is still possible to obtain reliable tests.
  • Running A Very Large Scan -- When the eValid site analysis "spider" is scanning a website you have to remember that to do its work properly eValid has to download every page. If the page it scans to is "bad" for some reason then that may stop the scan.

Friday, July 2, 2010

About The Browser User Controversy

About The Browser User Controversy

Key Idea: In creating realistic load for performance testing of AJAX applications it's critical to simulate users realistically, and for this reason the trend is toward using Browser Users (BUs). Why is this so important? And how many BUs can you get on one machine? On 10 machines? On 100 machines?

The question often comes up: What is all this ruckus about "browser users"? Why is it such an important?

Introduction

The general discussion you're probably referring to is about server loading methods. The goal in such projects is to simulate large numbers of users to do "web performance testing" in general. And, more specifically, to do "AJAX applications performance testing" in particular. The claim is made that the usual methods -- which typcially are based on simulating HTTP/S traffic -- don't work with AJAX.

There are [at least] two dimensions to this discussion:

  1. The simulation of users needs to be realistic if you are to believe the results; and,
  2. It is very difficult to get good scaling factors so that your realistic sessions impose realistic load.

About AJAX Applications

AJAX applications are NOT stateful and typically involve complex interactions between the JavaScript (JScript) programs in the browser and various programs running in the server that cooperate with each other. And, because it is "A"JAX it is "A"synchronous by nature.

Consequently, the big issue with testing an AJAX application is timing.

Now, the use of HTTP/S protocol traffic simulation is insufficient for AJAX applications because HTTP/S is "memoryless" -- whereas AJAX applications do have internal state. A "memoryless" simulation can't remember any state, and therefore can't wait for download work to be completed.

The way most HTTP/S loading approaches overcome that is to put in user "think time" or "waits" -- so that the traffic is slow enough to account for typical delays. This is good practice, but...what happens when the server slows down?

No matter how long a Wait you put in, there is ALWAYS a load level at which the Wait times are "not long enough" and the test fails. When your AJAX tests "fail to sync" you are out of luck. But an eValid script playback, given that it has been adapted to be self-synchronizing, won't have these timing issues. You'll never "fail to sync" -- because the script has programming in it that directs the eValid browser to wait in just the right ways.

Creating Realistic Load
There are two main ways to create load on a server:

  1. Virtual Users (VUs) -- This methods uses simulated HTTP/S protocol traffic to produce the effect of a user running a specific application.

    This approach scales well -- because all that is needed is to generate HTTP/S traffic, and this can be done with simple HTTP Get commands. On the other hand, the use of a stateless protocol disables end-user reality when you have an AJAX application under scrutiny

  2. Browser Users (BUs) -- Running a test playback with an actual browser instance, which is what eValid does.

    The scaling issue here is more complex, because to run 10, or 100, or 1,000 BUs requires quite a lot of compute power. Typically, with eValid BUs, we find that you will run out of RAM first, and so on a typical XP box you will max out at about 100 BUs per user. (Interestingly, our experience is that you run out of RAM first and we don't usually have problems with I/O capacity with higher loading levels).

    To get to the 1,000 BU level -- or to even higher BU counts -- you will need to have a fairly strong machine (RAM is the scarce resource) and use multiple user accounts.

At the end of the day, if you imagine a 100-BUs loading scenario, what you have is a group of eValid instances each running a separate playback script and each one independently acting like an AJAX application user. Any question about realism is moot. You're actually running real browsers. And, your load, performance, and testing results show you accurately and easily what the end-users are experiencing.