Current User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

• Sync On The Same Document Object Property -- When two scripts synchronize on the same object property there is no interference because there are separate processes involved.
• The Value Of "sourceIndex" And "elementValue" -- How to use the values of these two critical DOM-based variables inside your current script.
• How Do You Do Step Functions In Load?" -- What's the best way to ramp up load when running multiple eValid instances, and how to arrange for a "step function" in that ramp schedule.
• Can Two eValid's Be Running At One Time? -- eValid technology allows multiple instances to run in parallel and not interfere with each other at all.
• Does eValid use HTTP Traffic in Simulating User Load? -- The way eValid generates load exploits the browser's ability to download pages, but there is an alternative utility that can drive simple non-brower HTTP traffic if you wish.

Structural Testing: Making Web Tests More Durable

We often hear users complaining that their web pages changed and the tests that used to PASS yesterday now FAIL. Is there anything that can be done to have tests endure page changes that don't affect functionality, they ask?

The answer lies in the use of the Structural/Algorithmic Testing testing approach. Instead of basing test on properties of the page, when a particular feature is likely to move around, the test is based on (pivots on) that feature. The result (as the table outlines) is a little bit more work in exchange for a much higher level of durability -- without any loss in test quality (no increase in false-negative results).

A simple technique that we recommend, a Manual Script Creation Process, takes the mystery out of converting a "from life" recording into a highly durable validation test. Once the key page elements are identified, you modify the script using the Structural Testing Commands.

P.S. If this structural testing approach still isn't strong enough for your needs, with some additional programming effort you can exploit the eValid Programmatic Interface [EPI] to create a C++ program that runs your test at any level of durability you want.

User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

• Encountered Script Error... -- What does a "script error" mean when you see this message in the eValid EventLog? What are the implications?
• Automatically Snap A Screenshot -- How do you set up eValid to automatically capture a screenshot when there is a FAIL status?
• How long does a section in the middle of my script take? -- Explains how to extract timing and other performance details from the middle of an eValid playback sequence using the SaveRecord command.
• The Internal State Of The Browser -- A discussion about the meaning -- and the value -- of a fully state-preserving test system. Plus, some information about how eValid is architected.
• The New Microsoft Coded UI Test System -- How does eValid compare with Microsoft's latest functional test offering? Key areas to study and important factors to consider.

User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

• What's The Impact Of This On The "Reality" Of The Playbacks? -- The cache is a unitary resource, and eValid instances must share it during multiple-BU playbacks, but, this turns out to be a conservative assumption.
• Making A Recording Versus Scripting A Test -- The eValid approach makes creating the script so easy, our recommended approach is to "record first, script later".
• Why is having a lot of BUs running such a big deal? -- For many modern web applications -- especially those which use AJAX methods -- having an actual user do the test playback assures 100% realism.
• Structural Testing For Web Apps -- There are special eValid "structural testing" commands that allow a script to be completely independent of inconsequential page changes.
• A Sequence of HTTP References -- There is a special facility in eValid to generate a sequence of HTTP Get's that correspond to the actual sequence of page components that comprise a full page download.
• Microsoft SkyDrive Applications -- The Microsoft SkyDrive based applications -- common office utilities like a word processor or a spreadsheet -- actually test quite well with eValid, but they are among the more challenging applications to test.
• Link Checking -- There's a special eValid dynamic link check command that automatically visits every link on a page, even when the page is generated dynamically.
• Constant ID Tags -- When a web page has permanently-assigned ID tags the theses can "pivot" on those internal identifies, but even when the ID tags are dynamic it is still possible to obtain reliable tests.
• Running A Very Large Scan -- When the eValid site analysis "spider" is scanning a website you have to remember that to do its work properly eValid has to download every page. If the page it scans to is "bad" for some reason then that may stop the scan.

About The Browser User Controversy

About The Browser User Controversy

Key Idea: In creating realistic load for performance testing of AJAX applications it's critical to simulate users realistically, and for this reason the trend is toward using Browser Users (BUs). Why is this so important? And how many BUs can you get on one machine? On 10 machines? On 100 machines?

The question often comes up: What is all this ruckus about "browser users"? Why is it such an important?

Introduction

The general discussion you're probably referring to is about server loading methods. The goal in such projects is to simulate large numbers of users to do "web performance testing" in general. And, more specifically, to do "AJAX applications performance testing" in particular. The claim is made that the usual methods -- which typcially are based on simulating HTTP/S traffic -- don't work with AJAX.

There are [at least] two dimensions to this discussion:

1. The simulation of users needs to be realistic if you are to believe the results; and,
2. It is very difficult to get good scaling factors so that your realistic sessions impose realistic load.

About AJAX Applications

AJAX applications are NOT stateful and typically involve complex interactions between the JavaScript (JScript) programs in the browser and various programs running in the server that cooperate with each other. And, because it is "A"JAX it is "A"synchronous by nature.

Consequently, the big issue with testing an AJAX application is timing.

Now, the use of HTTP/S protocol traffic simulation is insufficient for AJAX applications because HTTP/S is "memoryless" -- whereas AJAX applications do have internal state. A "memoryless" simulation can't remember any state, and therefore can't wait for download work to be completed.

The way most HTTP/S loading approaches overcome that is to put in user "think time" or "waits" -- so that the traffic is slow enough to account for typical delays. This is good practice, but...what happens when the server slows down?

No matter how long a Wait you put in, there is ALWAYS a load level at which the Wait times are "not long enough" and the test fails. When your AJAX tests "fail to sync" you are out of luck. But an eValid script playback, given that it has been adapted to be self-synchronizing, won't have these timing issues. You'll never "fail to sync" -- because the script has programming in it that directs the eValid browser to wait in just the right ways.

Creating Realistic Load
There are two main ways to create load on a server:

1. Virtual Users (VUs) -- This methods uses simulated HTTP/S protocol traffic to produce the effect of a user running a specific application.

   This approach scales well -- because all that is needed is to generate HTTP/S traffic, and this can be done with simple HTTP Get commands. On the other hand, the use of a stateless protocol disables end-user reality when you have an AJAX application under scrutiny

2. Browser Users (BUs) -- Running a test playback with an actual browser instance, which is what eValid does.

   The scaling issue here is more complex, because to run 10, or 100, or 1,000 BUs requires quite a lot of compute power. Typically, with eValid BUs, we find that you will run out of RAM first, and so on a typical XP box you will max out at about 100 BUs per user. (Interestingly, our experience is that you run out of RAM first and we don't usually have problems with I/O capacity with higher loading levels).

   To get to the 1,000 BU level -- or to even higher BU counts -- you will need to have a fairly strong machine (RAM is the scarce resource) and use multiple user accounts.

At the end of the day, if you imagine a 100-BUs loading scenario, what you have is a group of eValid instances each running a separate playback script and each one independently acting like an AJAX application user. Any question about realism is moot. You're actually running real browsers. And, your load, performance, and testing results show you accurately and easily what the end-users are experiencing.