Drive tests using JavaScript?

An eValid user wrote:

Does evalid drive tests using JavaScript? If not, why not?

There are many reasons why evalid does NOT "drive tests using JavaScript" and some of them go back to the very beginnings of eValid.

A main goal of the eValid technology development was to provide a test bed or test driver that could work efficiently and easily with web browser enabled applications. In that beginning period -- going back to over 10 years ago -- we had completed internal "proof of concept" implementations of several kinds of testbeds. We tried out "wrappers," we tried out on-board programs using JavaScript passages, we tried out various "browser editors"...in short, we tried all of the combinations that we could think of.

What we discovered in our early engineering work was that there was a vast difference in the overall performance -- and a vast difference in how much each testbed trial solution interacted with the behavior of the web application. It was the potential for "bad interaction" which pointed the way for the team, because the underlying assumption was that, to be a good test driver, the test drive could not get in the way of the thing being tested.

Of all of the alternatives that were considered, we leaded strongly on the one prototype product that was built with an open-source browser. Having that kind of direct, immediate, "intimate" control of all of the signals and events and transfers and other activity while the browser gets the page components, assembles them into the internal document object model, and finally renders them to the user -- that level of interaction, we felt, was "perfect" for what we envisioned eValid to do.

We sill had to deal with the question of "overhead." No matter how efficiently you build a test driver, it is software so it takes some fraction of the CPU to execute. We settled on a target of not more than 0.1% interference with the application. And, the solution we ultimately chose, and which has become the eValid architecture, has met that goal in every regard.

So, the REAL reason that eValid does not use JavaScript to drive the test is that it is (a) too messy, (b) has too high an overhead, (c) imposes too large a footprint for the test engine, and (d) interfers with the normal operation of a web page. With the growing importance of AJAX this choice is more than justified -- eValid's architecture says "hands off" to the co-executing JavaScript passages that implement AJAX -- and this means that eValid tests are far more accurate and far more reliable than would have been any of the other architectures which we studied, and discarded as wanting.

__________________
eValid Tech Support

Do you have a standard pricing structure for building an eValid test? What are the limits on size and complexity?

An eValid user wrote:

Do you have a standard pricing structure for building an eValid test? What are the limits on size and complexity?

Yes, in addition to providing licenses to the eValid product so you can build your own tests, we also provide services that generate tests according to you own specification.

We have (as you might expect) a great deal of experience in building functional tests using eValid, and we generally are able to deliver test scripts to you that are 100% self-synchronizing and also 100% load-test safe.

Self-synchronizing means that you can make any adjustment to the Wait Time Multiplier you like -- including eliminating all delays, a Wait Time Multiplier of zero -- and the test will still run successfully. This is done with eValid structural commands and with eValid DOM-synchronization commands.

Load-test safe means that the tests can be run in multiple eValid instances on the same machine, without interfering with each other. That way, such a test can run 100's of browser users on one regular machine -- and 1,000's of browser users on a beefy, multi-user computer (like the ones we deploy when doing cloud-based loading work for customers).

We are so confident that we can complete such test script development projects on a tight budget -- and who isn't interested in having tight control of costs these days? -- that we offer such test creation mini-projects for a typical fixed-price fee $1,495 per script. And we take all of the engineering risks if it takes us longer than our nominal time budget. We have a 100% deliver success ratio on such mini-projects.

There are size constraints and limits of course, but for this very modest price we generally deliver a test script that has all of the above properties but in addition typically has coverage for up to 200 test steps (test plan imperatives like "click this..." or "select that..."), and can play back as long as about 5 minutes with no wait times.

You might think that 5 minutes is not a long time nor a very comlicated script, but when eValid scripts run with zero delay times, we find that the resulting test script, which may be upwards of 400 lines or more, will cover a very great deal of "functional ground."

All we need to get going on a mini-project like this is a step-by-step outline of your test, and appropriate access credentials for your application. If you don't want to write your test plan down, we can even include doing that step with a phone interview.

Driving a complex AJAX APP

An eValid user wrote:

Driving a complex AJAX APP...Tell Me How

This is a good question and it raises an interesting point about how modern web browser enabled applications that involve large amounts of AJAX actually operate.

If you recall, when AJAX is used there are JavaScript programs that run in the JavaScript engine (inside the browser) that do all kinds of things, but mainly control communication and data transfers between the browser and the server, based (in part) of user inputs. Think in terms of what you see on your screen when you type strings into a search engine that has implemented some kind of "autocomplete" feature. In that case there is a LOT of back-and-forth to/from the server.

The amount of processing time for that used to be very small, almost "invisible" to you as a tester in the scale of time that relates to your test cases.

But, as the total volume of JavaScript has grown the total time that running those (admittedly, wonderful programs) also grows. Accordingly, the amount of time it takes for the browser to settle down and reliably accept an input from the test engine grows.

That's where the problem arises.

eValid's structural commands are done separately from that JavaScript (this is an important architectural feature of eValid's implementation -- the non-interference with JavaScript). Even so, we have seen cases in which the eValid structural commands are "too fast" for the browser...we have heard of cases when, due to rapid-fire issuance of structural commands that update the DOM, the browser actually aborts execution!

So, what to do?

What we recommend is to insert a Delay 10 command, or perhaps a Delay 100 command after major structural command sequences to give the browser time to stabilize after the changes. For some cases, we have even had to use Delay 1000 -- that is, delay an entire second -- to prevent failure.

Or, duh, you could get a faster client computer (but you didn't want to hear that, did you!).

_____________________
eValid Tech Support

Do you have any recommended maximum length of a test?

An eValid user wrote:

Do you have any recommended maximum length of a test?

Another way to ask this might be, beyond how many test steps is the state of the browser no longer going to be repeatable.

There's no imposed limit on the length of time an eValid script can play back, and we have seen some scripts that take 15+ minutes the execute.

One script we worked with recently had about 250 separate steps (about 500 commands overall) and took about 10 minutes to play. That's quite long!

That is quite long, and the elapsed playback time, even when accellerated by reducing waits to the minimum through synchronization, imposes delay when you are trying to modify or debug a script.

Obviously there is no general rule, but we would argue that a playback time above 5 minutes (300 seconds) is going to run into either state-loss or tester-patience-loss issues before long.

____________________
eValid Tech Support

What happens to an evalid site analysis run if my website changes in the middle of the run?

An eValid user wrote:

What happens to an evalid site analysis run if my website changes in the middle of the run?

Here is the way the site analysis engine works.

It starts (at the page you specify) and makes a list of all of the links on that page and then, one by one, it attempts to visit every page on that list.

On each new page, it ADDs to the "work list" by enumerating all of the links on that page.

No page is visited a second time, and the scan stops when the "work list" is empty.

Now, if one of the pages changes during the scan, then the NEW information will be used as the basis for adding items to the "work list". If the new page has left out a link that was present at the time the scan began, then that page will not land in the "work list" and will never be visited.

So, bottom line, if the site changes there is a risk that some pages that were present at the time the scan started will not be visited, and there is a corresponding risk that some pages that were not present at the time the scan started will be visited.

Practically speaking, if you are running a scan regularly, ultimately all of the pages' actual [stable] content will be visited, so this effect may not be an issue for you. But a one-off scan may be in error.

It's best, of course, to run scans on stable sites that are not undergoing maintenance and/or upgrade and/or change.

_________________
eValid Tech Support

Testing With DOM Interactions -- Guide to Examples (Part 2)

This article presents eValid example scripts that illustrate use of the index/motion (structural testing) DOM manipulation commands. These scripts were developed using the available DOM Technology Resources to accomplish the results demonstrated in each "snippet".

In most cases the index/motion (structural testing) scripts are extremely robust in that these test script passages do not depend on specific page or application properties, but only on objects and effects that can change location and visual detail.

Tests built using these techniques survive the effect of dynamic page changes, including inconsequential changes that don't affect the underlying functionality. They also address basic test playback synchronization issues and can still provide fine-detail client-side performance data extraction suitable for large-scale loading experiments.

Link	Short Example Explanation
Third Item in Dropdown List	Find the third item in a dropdown list and click on it.
Nth Item in a Select Box	Choose the Nth entry in a select box and click on it.
Select Two Elements in a CheckBox	Select two elements of a checkbox.
Find and Follow a Specified Link	Find a link based on knowing something about it and then follow it.
Find A Dynamic Tag And Click It	Locate a dynamic element tag and then click on it.
Text Entry Into a JavaScript Editor Field	Find a particular input field that is handled by JavaScript and type specific text into it.
Validate That One Item Matches Another	Extract two values from different spots on a page and compare the extracted values.
Using Regular Expressions	Using the IndexFindElementEX command to find using multiple "name=value" pairs.

Testing With DOM Interactions -- Guide to Examples (Part 1)

This article presents eValid example scripts that illustrate use of the index/motion (structural testing) DOM manipulation commands. These scripts were developed using the available DOM Technology Resources to accomplish the results demonstrated in each "snippet".

In most cases the index/motion (structural testing) scripts are extremely robust in that these test script passages do not depend on specific page or application properties, but only on objects and effects that can change location and visual detail.

Tests built using these techniques survive the effect of dynamic page changes, including inconsequential changes that don't affect the underlying functionality. They also address basic test playback synchronization issues and can still provide fine-detail client-side performance data extraction suitable for large-scale loading experiments.

Link	Short Example Explanation
Navigation to URL	Shows how to navigate to a particular URL.
HTML Push Button on a FORM	Locate and click on a push button on a FORM.
Radio Button On A FORM	Locate and push a radio button on a FORM.
Check Box On A FORM	Locate and check a particular box on a FORM.
Single Form Text Input Field	Find a particular text field in a FORM and type something into it.
Multiple Form Text Input Fields	Type in multiple files in a FORM.
First Item in Dropdown List	Click on the first item in a dropdown list.

Selected User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

• Can multiple browsers run from multiple data streams? -- Feeding data into each browser separately.
• Random wait time between events -- How to use the WaitRandomUniform command.
• Login Account Name -- Details of handling logins and internal states.
• Can eValid click on a hidden button? -- Doing the impossible: processing something that isn't there!
• Testing AJAX Applications For Load -- How to handle loading of AJAX apps.

Selected User Forum Posts

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

• AJAX Page is updated continuously with feed from my website -- How evalid handles dynamic pages.
• How do you coordinate between two or more browsers? -- Multiple-browser locking.
• What good is "geometric test"? -- More about making good geometric tests.
• How does eValid keep track of frames? -- Details of frame processing.
• Analyze DOM values in a current page -- Reading and analyzing specifc DOM values.

Can eValid create a non-visible-text validation from the GUI?

An eValid user wrote:

Can eValid create a non-visible-text validation from the GUI?

Yes, there is a programmed-from-the-GUI optional sequence that will automatically construct a command to validate a selected object property. This command works from the PageMap and allows you to validate ANY property that is present in ANY object anywhere on the page.

A word of caution: this command, and the GUI-sequence that leads to it. is somewhat more technical that the vast majority of eValid validations. We put it in there because it was appropriate for completeness -- to have a least one command that digs into the DOM and constructs something in your recorded script for you.

As you can imagine, there are many ways that the GUI sequences can lead you to construction of a script -- "from the GUI" and thereby somewhat automated -- and you might ask, "Why doesn't eValid have a richer set of such GUI sequences?"

The answer lies in this observation: No matter how complex you create the from-the-GUI operations, there will always be some case that you haven't thought of, and then you're stuck. Instead, with eValid our idea was to provide the mimimum amount of GUI-based help needed, consistent with keeping the GUI simple and intuitive. (We have seen some other offerings that boggle the mind with the complexity of the GUI sequences involved!)

So, instead -- to give the user the power needed for the "outside the GUI" case -- we created a simple set of "structural testing commands" that are easy to learn, few in number, and ultimaly can do anything and everything that you need. Here is a link to the manual page for that:

Structural Testing Commands

The set of primitive operations you have with these commands -- get, set, put, find, move, etc. -- are able to handle ANY situation you can imagine.

In effect, rather than try to guess what you might need (and know that there will always be a case in which it "won't work"), we have provided a simple programming tool -- actually a kind of "program the browser" tool -- that is FAR more efficient in terms of your time and effort than an elaborate and ultimately ineffective GUI sequence.

We hope you agree with our simple is better approach to eValid design and operation.

_____________
eValid Support