Friday, August 22, 2008

Conversation With a Test Manager (Part II)

In Part I of our conversation with the high-level big-company test manager we looked at how eValid technology works in some detail. Our conversation then turned to criticism of our use of the "record/play" paradigm. He believed that record/play is a throw-back to an earlier era, and doesn't apply to web based applications.

  • But eValid's only a Record/Playback Tool, and Everyone Knows That's Passe..
    Actually, the record/playback method is actually very powerful if two conditions are met: (i) you need a strong recording engine and (ii) you need a strong playback engine. Without both we kind of agree with you -- you might be wasting your time.

    We were able to achieve both goals because all of the record/playback functionality is built inside the browser, where there is "perfect" information about the user's actions, about the current page, and about state changes an action causes.

    • Recording: The recording engine (the technically most difficult part of eValid) operates with full browser state knowledge. The result is a compact and reliable test recorder that creates scripts that play back very accurately. Generated commands are shown in the script window during recording virtually simultaneously with user actions. It's easy to see what is working and when Alternative Recording Modes need to be used.
    • Playback: Brittle tests are indeed problematic. In our view, eValid's Adaptive Playback feature overcomes that by taking advantage of a playback engine that is fully informed about all the details in the current page. It can do the "right thing" and prevent loss of valuable script data.
  • "eValid Users Will Require Intimate Knowledge of How User Inputs Are Interpreted..."
    The eValid recording engine senses user actions in the DOM context. It uses browser-internal access to these signals and events, watching for them to happen (recording) and/or causing them to happen (playback). The automatic recording/scripting often fills in for needing detailed application knowledge.

    It is worth noting that the rather longish scripts in our Testing the GWT "Kitchen Sink" example were created 100% by recording -- they certainly were not scripted by hand (God forbid!). The longest one, if you check the recording start/end times, required about 90 seconds to actually record from life -- and that includes the recorded think times!

    The user behaves normally, and eValid automatically acquires the intimate knowledge of how user inputs are interpreted, and writes the commands accordingly. Very little training on how to do this is ever involved. (AJAX applications may be an exception, but you would expect that.)

  • "But What About Actions That Are Interpreted by On-board JavaScript..."
    eValid is an IE clone; it implements the full IE JavaScript engine. JavaScript events are sensed internally during recording, and JavaScript events are triggered directly in the eValid browser to accomplish playback effects.

    It is VERY important to note that -- unlike systems like Selenium and several others -- eValid playback is NOT done with JavaScript. Script playback in eValid is via direct interaction with the browser -- inside the browser.

    This architecture has some neat benefits: eValid never encounters problems with the "same-origin policy", security zones and authentications, the "can't write to a file" constraint, cross-site scripting limits, or any of the other limits intrinsic to JavaScript-based test playback solutions.

    These are just a few of the intrinsic limits of JavaScript based playback solutions. The eValid advantage, in NOT being a JavaScript based solution but instead being based on use of a full-featured browser, becomes even more apparent when you need to synchronize an AJAX application's test playback. eValid does AJAX synchronizations natively, by repeatedly interrogating the DOM internally to wait for a DOM element attribute to take on a required value, withOUT interfering with the concurrent execution of JavaScript that supports the AJAX application.

    Because of these limitations we believe JavaScript-based playback methods are a technical dead end.

That covers the main points about eValid's architecture. In Part III we'll see how eValid shapes up as a practical system for delivering tests.

6 comments:

stanleyxu (2nd) said...
This comment has been removed by the author.
stanleyxu (2nd) said...

"Because of these limitations we believe JavaScript-based playback methods are a technical dead end."

Hello, I do not think these technical limitations will sentence javascript-based playback methods to death. Once popular browsers provide a super privilege for javascript-based testing tools, everything will change. Actually selenium solved some problems already.

No doubt, that eValid (tools implemented as a browser extension) provides a great solution for testing jobs. But please note that such kind of tools are browser and platform dependent. In principle you can finish the testing job on certain versions of IE very quick, but there is no prove, that the web application works /exactly/ so well in other browsers (other rendering engines).

Different rendering engines have different fault-tolerant. For instance:
Unclosed tag might be understand by some browsers but not all of them.

So I think, for browser-extension-based playback methods, the biggest challenge is, how to test all (popular) browsers (rendering engines)

eValid said...

Just a small point, but if JavaScript engines did have "super privaleges" that would allow a very large security hole, and it is in part for this very reason that we believe that JavaScript methods are inferior to "in the browser" methods.

stanleyxu (2nd) said...

Thanks your reply, but it does not hit the point. I do not mean, that browsers should give JavaScript super privileges by default. But it seems to be totally harmless, if browsers could allow user to elevate the privileges with a confirmation. Actually MSIE and Firefox provide such an option already.

I am not questioning the power of eValid toolkit on testing webapps in MSIE. But if your team want to extend the test coverage, you have to make double (, triple or more) efforts on implementing toolkit for other browser rendering engines.

So could you please tell me, do you think that to test webapps in MSIE using eValid is significant enough?

eValid said...

Thanks for the comments; much appreciated.

You raised a number of points and we'll address them in
order.

(1) On the question of future use of JavaScript the
issue relative to simulating user behavior is that
JavaScrip eats resources, and is single-threaded to
boot, and comes with a lot of constraints (the same
origin policy, for example).

Doing test playback natively in the eValid browser is
very efficient and non-interfering with anything else
the browser is doing. Plus there are some kinds of
actions eValid can do that are problematic with
JavaScript, for example, DOM polling for playback
synchronization on an AJAX app.

(2) Just to be clear, eValid is not a browser extension
at all. eValid *is* a complete browser; the test
recording, playback, and analysis functions are built
in CPP right alongside the browser functionality. This
is neat, and very efficient (low eValid system
overhead).

(3) "...finish a testing job very quick" is a very
important property to have. eValid is a productivity
tool, so being able to accomplish the same amount of
testing in less time is seen as a good thing to
support.

(4) eValid's purpose is to test web applications, and
the rendering engine used is IE. eValid is not
supposed to be used to test whether delivered pages are
compliant in other browsers. That's cross-browser
testing and there are other ways to do that. eValid is
intended to test web application functionality and
performance.

(5) eValid has a command that can be used to set the
user agent string to any value. This command makes it
possible to cause eValid to report to a server that it
is ANY kind of browser, a feature that is useful for
validation of web applications beyond the IE context
that eValid supports.

(6) Users report being able to test their applications
very thoroughly in the MSIE context, apart from minor
rendering issues. So yes, it IS adequate to test
webapps fully with eValid.

stanleyxu (2nd) said...

Thanks again for your reply. I fully understood what you mean and have a fair image of eValid. It is a nice tool. ^^)

I wrongly classified eValid a browser extension. Actually I mean it is a shell browser based on Trident rendering engine.