Conversation With a Test Manager (Part III)

In Part I and Part II we related how eValid dealt with concerns about eValid technology and about reliance on the record/play paragigm. Here we finish up looking at issues of operational complexity and overall field costs.

• "Effective Use of eValid Will Be Obscured By The Complexity Of The Scripting..."
  Actually, it's the opposite. eValid's approach is to AVOID scripting as much as possible and to resort to manual script modifications only as a last resort. It's cheaper that way.

  We have found that 99.99% of the time the "from life" recorded script is adequate as it stands, mainly because Adaptive Playback (defaulted ON) softens the "brittle nature" of the script/application correspondence.

  In practice, users record scripts from life and then modify and/or augment the generated script by manually adding commands (e.g. AJAX synchronizations) as required. You don't have to remember any script syntax -- eValid does that for you. This process is enormously more efficient that hand-coding tests from first principles -- which eValid users normally never need to do.

  The Testing the GWT "Kitchen Sink" example scripts show some of these modifications, e.g. when extra validation steps were added to confirm that the playback arrived at the correct end state.

  An even better example of this is a sample test of Gmail, which shows a number of hand-modified advanced-usages of AJAX synchronization command is our Testing Google Gmail with Index/Motion Commands.

• "But, Hey, That's Great But I'm Really Concerned About Productivity Costs..."
  eValid is first and foremost a productivity tool. For web applications eValid's automated script recording, augmented with a few hand modification of scripts based on some knowledge of the available "extra" commands, and in a context that includes AdaptivePlayback, easily trumps every other approach in terms of productivity.

  The famous testing notion "...if it ain't broke, you're not trying hard enough!" is only going to be effective if you can try things out in sufficient quantity AND quality. Think functional coverage here. A script that costs $5,000 to develop and that tests one feature brilliantly is never going to have the quality enhancement effect of 100 scripts that cost $50 each to develop, because these 100 tests may cover 100 different functions. The more different things you test the better off you are.

We hope you enjoyed our interview!

Corporate e-Communications Award (CECA)

eValid, our South Africa business partner BlueRiverStone, and a number of additional sponsorsing organizations, are mounting the Corporate e-Communication Award for 2008. Awards for this study are expected to be announced in mid-October 2008.

eValid's role in this competitive analysis is to perform the detailed technical evaluations of the candidate companies' websites. After we complete the scans we publish the information collected to each organization so they can have the opportunity to update their website to overcome any anomalies difficulties that the eValid scan identified.

In prior years a similar eValid methodology was used, to very good effect, for several similar website quality competitions and contests, some reported in the Financial Mail, and some published independently by BlueRiverStone. Here are several of the reports.

• South Africa Travel and Tourism Industry (OTTI) Award
• Who's Hot, Who's Not (BCW 2005)
• The Best, And The Rest (BCW 2004)
• Financial Times FT-150 Website Comparison, and also FM Top 100 Websites That Save You Time & Money

In all of these studies, the eValid site analysis scans were used as the basis of the the technical analysis scores, using data from runs we made from our facility here in San Francisco. Kudos to Ian Kruger for his great work in exploiting the data derived from this super eValid application!

We'll bring you more details about how the contest works out in future entires.

Conversation With a Test Manager (Part II)

In Part I of our conversation with the high-level big-company test manager we looked at how eValid technology works in some detail. Our conversation then turned to criticism of our use of the "record/play" paradigm. He believed that record/play is a throw-back to an earlier era, and doesn't apply to web based applications.

• But eValid's only a Record/Playback Tool, and Everyone Knows That's Passe..
  Actually, the record/playback method is actually very powerful if two conditions are met: (i) you need a strong recording engine and (ii) you need a strong playback engine. Without both we kind of agree with you -- you might be wasting your time.

  We were able to achieve both goals because all of the record/playback functionality is built inside the browser, where there is "perfect" information about the user's actions, about the current page, and about state changes an action causes.

  • Recording: The recording engine (the technically most difficult part of eValid) operates with full browser state knowledge. The result is a compact and reliable test recorder that creates scripts that play back very accurately. Generated commands are shown in the script window during recording virtually simultaneously with user actions. It's easy to see what is working and when Alternative Recording Modes need to be used.
  • Playback: Brittle tests are indeed problematic. In our view, eValid's Adaptive Playback feature overcomes that by taking advantage of a playback engine that is fully informed about all the details in the current page. It can do the "right thing" and prevent loss of valuable script data.
• "eValid Users Will Require Intimate Knowledge of How User Inputs Are Interpreted..."
  The eValid recording engine senses user actions in the DOM context. It uses browser-internal access to these signals and events, watching for them to happen (recording) and/or causing them to happen (playback). The automatic recording/scripting often fills in for needing detailed application knowledge.

  It is worth noting that the rather longish scripts in our Testing the GWT "Kitchen Sink" example were created 100% by recording -- they certainly were not scripted by hand (God forbid!). The longest one, if you check the recording start/end times, required about 90 seconds to actually record from life -- and that includes the recorded think times!

  The user behaves normally, and eValid automatically acquires the intimate knowledge of how user inputs are interpreted, and writes the commands accordingly. Very little training on how to do this is ever involved. (AJAX applications may be an exception, but you would expect that.)

• "But What About Actions That Are Interpreted by On-board JavaScript..."
  eValid is an IE clone; it implements the full IE JavaScript engine. JavaScript events are sensed internally during recording, and JavaScript events are triggered directly in the eValid browser to accomplish playback effects.

  It is VERY important to note that -- unlike systems like Selenium and several others -- eValid playback is NOT done with JavaScript. Script playback in eValid is via direct interaction with the browser -- inside the browser.

  This architecture has some neat benefits: eValid never encounters problems with the "same-origin policy", security zones and authentications, the "can't write to a file" constraint, cross-site scripting limits, or any of the other limits intrinsic to JavaScript-based test playback solutions.

  These are just a few of the intrinsic limits of JavaScript based playback solutions. The eValid advantage, in NOT being a JavaScript based solution but instead being based on use of a full-featured browser, becomes even more apparent when you need to synchronize an AJAX application's test playback. eValid does AJAX synchronizations natively, by repeatedly interrogating the DOM internally to wait for a DOM element attribute to take on a required value, withOUT interfering with the concurrent execution of JavaScript that supports the AJAX application.

  Because of these limitations we believe JavaScript-based playback methods are a technical dead end.

That covers the main points about eValid's architecture. In Part III we'll see how eValid shapes up as a practical system for delivering tests.

About Warnings Vs. Errors

Many eValid users don't understand what Warning messages mean. Here's a quick rundown:

• If you are using the Playback Data Dialog, you'll only see a Green Screen when the playback runs to completion without issuing any playback flags at all, i.e. no Warning, Error, Timeout, Alarm, or Failure flags.
• You'll see a Red screen on any Error, Timeout, Alarm or Failure flags.
• You'll see a Yellow screen when there are any Warning flags. The most common reason for a Yellow screen is that the Adaptive Playback system had to adjust the playback behavior to overcome minor changes (regressions) in the web page.

Here's a full explanation of eValid's Error Reporting Process. The key notion here is this: Warnings aren't Errors. Unless they are.

Conversation With a Test Manager (Part I: The Basics)

A while ago we had a long conversation with a senior test manager at a big company about eValid, about eValid's technology, and about how this might all play out in his environment of tester/programmers. It was quite a long conversation, and we didn't make a recording (only notes), but here are some of the main issues he raised and how eValid answers them.

But eValid Only Runs On Windows, And Only On IE...
Yes, that's true right now, but eValid technology is currently being implemented in the FireFox build tree, using the current product as the design base. In the meantime, eValid already can "set user agent string" so that it can pretend to be ANY browser. We even have a fully worked
Illustration of User Agent String Modification.

But, Hey, eValid's Scripting Language is Non-Standard...
Actually, the eValid script language is more an automatically generated "command language" than it is a pure scripting (programming) language. We worked up a view of the eValid script language as a abstraction that implements a user controlled browser. The Abstract Model of eValid: Script Language View is the result. All of the eValid commands easily map into simple function calls (method invocations), following the internal Script Principles description.

But We Are Used to Writing Very Complex Testing Programs...
For applications where it is required, the EPI (eValid Programmatic Interface) provides users with full programming language support. The EPI feature is currently available for C++; versions for Java, VB, and PERL are being implemented.

Do You Expect Us To Guess the Properties of Page Elements...
Not at all. Access to DOM objects and all their detailed properties is done with the PageMap feature. Information from the PageMap display is bi-directional: you can click on the tree and/or click on the page, and you see the DOM specifics or page elements immediately.

So much for eValid technology and some of the main features. In Part II we'll look at the issues surrounding eValid's use of the "record/play" paradigm.

Control Mode Transition Diagram

During a test recording eValid can be in a number of states, and the current state affects how user actions are recorded in the script. The Control Mode Diagram indicates how these modes interact. Here is a quick summary of the available control mode toggles:

* F11: Absolute clicks

* Ctrl-F11: Absolute mouseovers

* Alt-F11: Element mouseovers

* F12: Application Mode

* Ctrl-F12: Desktop Mode

In practice when you are making a recording you may need to toggle into and out of these modes based on what you see eValid recording in the script window. It's a good bet that if you take an action and nothing happens in the script window that you will either (a) need to use one of the above-defined options or (b) mark that part of your script as needing manual modification.

One way or the other, you can always arrive at a script that plays back reliably. 99.99% of the time that's just by recording "from life".

EPI C++ Example Programs

We have developed some simple examples that show off how eValid's Programmatic Interface (EPI) works.

Example #1: This very short example involves just a few evalid commands, an InitLink and a GotoLink. It's main purpose is to illustrate how the EPI interface is structured from the programmers' point of view. You can see both the CPP program and the corresponding non-EPI eValid script for comparison.

Example #2: Similar to the above, this example also is 1:1 with a corresponding eValid script. This script involves running an IndexFind command that identifies [to an internal variable] a particular field. The same command in the EPI interface performs the same way but in this case the actual index value is returned into the CPP program. That index is available to be used for other purposes in the CPP code.

Example #3: Here is a CPP program that performs a particular kind of search that can only be done easily with regular eValid command sequences of some complexity. The program uses a search engine to search for a particular search term, and then computes [by iterating and counting] the first search-results page that includes both the searched-for string and a target URL.

Example #4: Here there is a regular recorded eValid script that does a number of different kinds of things, and the EPI CPP program which duplicates the script actions exactly.

Example #5: The program in this example does not have a direct eValid script equivalent. The program navigates to a user-specified URL and then, within that page, finds the specific index of the [first] element in that page for which a specified attribute NAME has a specified VALUE. This read is done from the DOM of the page as it currently exists, which may or may not be what was sent down from the server, depending on what kind of AJAX-type operations were performed on it.

Example #6: The program in this example does not have a direct eValid script equivalent. Similar to Example #5, this program changes the value of attribute NAME at a specified index on the page to the specified VALUE. This update action is done to the DOM of the page as it currently exists, without permission of the browser or the server. This is a powerful feature and user caution is advised.

The bottom line for this set of examples is this: EPI for C++/CPP is available today and lets you do tests in an efficient and powerful way.

User Agent String Modification

We get a lot of questions about where eValid stands in relation to implementation on non-IE browsers, and in particular, when we'll have a Firefox version available. While we're not quite ready to announce any dates on that, you can rest assured that the Firefox version is on the product roadmap!

In the meantime, what can eValid offer to someone who wants to see if a particular application works (or perhaps equally as important, doesn't work) on a particular browser. It turns out that eValid has this nifty "SetUserAgent" command that may be helpful in that kind of testing.

The SetUserAgent command lets eValid "pretend" to be any browser you wish it to be, during a particular test session. This feature can be used to validate server behavior that involves alternative responses based on browser type (including, as the example will illustrate, denial of support to specific browsers). The User Agent string manipulation feature is also available at the command line interface.

Here is the Illustration of User Agent String Modification example. Let us know how you like it?