Friday, June 27, 2008
Doing that piece of abstraction analysis is a pretty tall order, and we are not finished with it. But we did recently complete a description of how eValid represents an abstract model that is the based on viewing eValid from its script language perspective.
As our Abstract Model of eValid: Script Language View points out, you can understand eValid quite well as a controllable browser upon which you can take particular actions to produce particular effects. We'll publish more of this set of descriptions as they become available.
Monday, June 23, 2008
- To overcome the expense of creating functional tests it makes sense to have a good test recorder, which takes away most of the pain of creating tests.
- Tests can be brittle, so you should have some kind of functional test playback feature than automatically adapts the bahavior of the playback engine to overcome inconsequential changes in the pages tested.
- To split out tests of individual features you need some kind of CallScript capability that lets you organize your tests easily into groups of related functionality.
Interesting post, and yes we certainly agree that AJAX web applications can be quite difficult to test. But the picture you seem to paint is rather bleak, and it need not be.
The situation for testing an AJAX application is actually easier than you may think, provide that you have the right tools and by using their underlying technology base can "do the right things" at the right time.
But you are dealing with AJAX -- and it seems that you left out the most important aspect of all: test synchronization. Tests of AJAX type applications scream out for playback synchronization -- but not by adding Waits or Delays or Sleeps that never work in practice. There has to be a way built into your playback engine to interact with the DOM of the page and sense when it is OK to continue playback.
We believe if you have the right facilities in the test engine the functional testing job for an AJAX application can produce very good results.
We hope that will get a good discussion going. And, meanwhile, we invite AJAX application testers to send us an example AJAX application that they'd like us to test with eValid.
Thursday, June 19, 2008
We've heard from many sources that Google's Gmail application is a very difficult to test reliably. This recently appearing newsgroup posting is one example.
Yes, we agree. The latest Gmail version certainly is a tricky one. And, dropping your testing into the older, pure-HTML version as a way out is certainly not a real solution.
So we figured, if eValid could do the Gmail application well then that would bode well for eValid use on other, similarly complex web applications.
Our sample Gmail test was to sign on to an existing Gmail account, compose an email message, send it to the same Gmail account, observe that it arrives, then delete the message just recieved. This test sequence is "idempotent" (i.e. cleans up after doing its work and returns everything to the original state), it exercises the main Gmail functions, and it allows us to illustrate a variety of uses of eValid's DOM Element Manipulation/Motion commands.
We put this script through some very stressful playback checkouts. One test was to set up a 100-run sequence using a delay multiplier of 0.1 and that produced normal results. (Lower delay multipliers on a fast driver machine actually played ahead of the Gmail system response time.)
Here is the complete explanation of the Testing Google Gmail With Index/Motion Commands example. There's an important cautionary caveat: In this script if the round-trip test message is sent to the Trash area rather than to the InBox area where it is expected, eValid will -- quite correctly -- see this as an error, a true regression.
Tuesday, June 17, 2008
For a very large permutation count you will never try every combination. But over time, as the total number of runs increases, and the total number of combinations tried increases, you will be able to accumulate sufficiently many successful runs to generate some confidence in the integrity of your application.
Here is how this could work in practice.
- Create your test starting with an eValid recording of your web application behavior.
- Parameterize that script with variables whose values are to be assigned at the start of each test.
- Construct a table of possible values for each parameter.
- Use eVgenRand (an eValid suite component) to create the values for each instance of playback using the "random selection" option, and specify a large number of tries to make.
- Make sure errors are recorded somehow for later study.
If tests each take one minute, after a week you have about 10,000 tries and some number, N, of tests that fail. Then:
- If N continues to be zero, then you can increased confidence in the quality of you web application. The more you run the better you feel.
- For each of the N failures you have hard evidence of a problem that needs to be fixed.
This may not be as efficient as proposed schemes from the R&D community that attempt to reduce combinatoric complexity from the outset. But this solution has the advantage of easy implementation, and immediate availability. The relevant eValid commands involved are described in the eVgen Example: Yahoo! Random Search example.
Monday, June 9, 2008
You can read the complete interview at Book of Testing: Thoughts From A Braidy Tester.
Friday, June 6, 2008
Wednesday, June 4, 2008
This command is part of the set of DOM Element Motion/Manipulation commands that are used to construct position/structure independent scripts that operate entirely at the DOM level. Such scripts are particularly attractive because of their ability to tolerate a very wide range of page variability without losing their meaning and effect.