Conversation With a Test Manager (Part II)

In Part I of our conversation with the high-level big-company test manager we looked at how eValid technology works in some detail. Our conversation then turned to criticism of our use of the "record/play" paradigm. He believed that record/play is a throw-back to an earlier era, and doesn't apply to web based applications.

• But eValid's only a Record/Playback Tool, and Everyone Knows That's Passe..
  Actually, the record/playback method is actually very powerful if two conditions are met: (i) you need a strong recording engine and (ii) you need a strong playback engine. Without both we kind of agree with you -- you might be wasting your time.

  We were able to achieve both goals because all of the record/playback functionality is built inside the browser, where there is "perfect" information about the user's actions, about the current page, and about state changes an action causes.

  • Recording: The recording engine (the technically most difficult part of eValid) operates with full browser state knowledge. The result is a compact and reliable test recorder that creates scripts that play back very accurately. Generated commands are shown in the script window during recording virtually simultaneously with user actions. It's easy to see what is working and when Alternative Recording Modes need to be used.
  • Playback: Brittle tests are indeed problematic. In our view, eValid's Adaptive Playback feature overcomes that by taking advantage of a playback engine that is fully informed about all the details in the current page. It can do the "right thing" and prevent loss of valuable script data.
• "eValid Users Will Require Intimate Knowledge of How User Inputs Are Interpreted..."
  The eValid recording engine senses user actions in the DOM context. It uses browser-internal access to these signals and events, watching for them to happen (recording) and/or causing them to happen (playback). The automatic recording/scripting often fills in for needing detailed application knowledge.

  It is worth noting that the rather longish scripts in our Testing the GWT "Kitchen Sink" example were created 100% by recording -- they certainly were not scripted by hand (God forbid!). The longest one, if you check the recording start/end times, required about 90 seconds to actually record from life -- and that includes the recorded think times!

  The user behaves normally, and eValid automatically acquires the intimate knowledge of how user inputs are interpreted, and writes the commands accordingly. Very little training on how to do this is ever involved. (AJAX applications may be an exception, but you would expect that.)

• "But What About Actions That Are Interpreted by On-board JavaScript..."
  eValid is an IE clone; it implements the full IE JavaScript engine. JavaScript events are sensed internally during recording, and JavaScript events are triggered directly in the eValid browser to accomplish playback effects.

  It is VERY important to note that -- unlike systems like Selenium and several others -- eValid playback is NOT done with JavaScript. Script playback in eValid is via direct interaction with the browser -- inside the browser.

  This architecture has some neat benefits: eValid never encounters problems with the "same-origin policy", security zones and authentications, the "can't write to a file" constraint, cross-site scripting limits, or any of the other limits intrinsic to JavaScript-based test playback solutions.

  These are just a few of the intrinsic limits of JavaScript based playback solutions. The eValid advantage, in NOT being a JavaScript based solution but instead being based on use of a full-featured browser, becomes even more apparent when you need to synchronize an AJAX application's test playback. eValid does AJAX synchronizations natively, by repeatedly interrogating the DOM internally to wait for a DOM element attribute to take on a required value, withOUT interfering with the concurrent execution of JavaScript that supports the AJAX application.

  Because of these limitations we believe JavaScript-based playback methods are a technical dead end.

That covers the main points about eValid's architecture. In Part III we'll see how eValid shapes up as a practical system for delivering tests.