eValid Blog

Webinar: Performance Testing Mobile Web Apps

2012-01-17T15:42:00.000-08:00

Run 1,000's of Mobile Browser Users
Full AJAX Realism for Any Mobile Device

Easily Identify Performance Bottlenecks

Wednesday, 25 January 2012
2:00 PM Eastern Time / 11:00 AM Pacific Time

QA/Testing/Tuning/Performance projects need to qualify performance of complex AJAX web applications -- within strict budget and time constraints -- to make sure their server-stack setups can meet the load.

The traditional methods of ramping up load, based on using HTTP/S simulations or "VUs", don't always work when asynchronous AJAX applications are involved. VU's don't do AJAX. You need a browser.

eValid server loading methods offer both quick-to-create, realistic, and fully synchronized AJAX functional tests. Plus you can lift those tests into performance/loading scenarios that can involve 100's or 1,000's or 10,000's of Browser Users ("BUs") per machine.

In this webinar you'll learn: how special eValid commands overcome problems with variable-time playback dependency; how to create full-reality AJAX tests quickly; how to adjust tests to be totally self-synchronizing under stressed AJAX conditions; how to incorporate tests in an eValid LoadTest scenario; how to launch 100's or 1,000's or 10,000's of Browser User (BU) instances; and, how to analyze consolidated performance summary data to identify server-stack bottlenecks.

This unique approach demonstrates how eValid becomes a genuine force multiplier in your web application performance testing efforts.

Webinar Topic Summary:

eValid Architecture and Structure: How eValid functional and performance testing works.
Functional Testing: How to make reliable recordings of AJAX applications.
Making AJAX Tests "LoadTest Safe": How to augment tests for complete AJAX synchronization.
Creating LoadTest Scenarios: How to use the LoadTest scenario editor to organize realistic LoadTest control scripts.
Running LoadTests: How to launch single and multiple-instance runs using "cloud computing" resources.
Finding Bottlenecks: How to read the LoadTest and other raw data to help spot server-stack issues.

You are cordially invited to attend this free Webinar.

Mobile Testing FAQs

2012-01-04T16:58:00.000-08:00

As eValid users know, eValid is a single-point solution for testing the functional quality, performance, and supporting server capacity for all kinds of web applications -- including Mobile Devices that run web applications.

We've prepared a Mobile Testing Technology -- Frequently Asked Questions (FAQs) that answers some of the most common questions about how eValid fills this role. Please take a look and don't hesitate to respond or ask questions.

Happy New Year!

2012-01-01T16:28:00.000-08:00

We would like to wish everyone in the eValid community -- users, resellers, and visitors and potential customers alike -- the very best of health, happiness and success in the coming year!

As more and more applications move into the "mobile" domain, you will see more and more challenges in testing these complex applications. As some of our technical work in the past several months has shown, the quality and relative performance of mobile applications can be indeed be tested reliably and effectively. In coming months you can expect to see more and more about testing complex mobile web applications from the eValid team.

HAPPY NEW YEAR!

Selected User Forum Posts

2011-12-20T15:13:00.000-08:00

Beginning in mid-2010 we have directed all technical support questions to the eValid User Forum. We have learned that when one user has an issue, all users can profit from the answer.

Here is an additional selection of some of the posts that we think would be of general interest.

Can I get page component timings from eValid? -- Page component performance data details.
We want to include eValid in our own product -- Possibilities for OEMing eValid are discussed.
Can your SQL interface be used in monitoring? -- Using the SQL feature in eValid.
Multi-user Simulations of REAL AJAX Interactions -- Why multi-user AJAX apps run OK.
eValid operating with Web 2.0 and AJAX applications -- Pointers to business white paper about AJAX testing.
Has eValid technology been patented? -- eValid patents detailed.
Do you think eValid is ahead of its time? -- Prognostications of a sort.
Why is the pagemap so slow sometimes? -- Performance of PageMap on very large pages questioned.
How does eValid treat JavaScript errors? -- Internal JScript error detection and reporting.
Does adaptive playback work when the ID tag is blank? -- Adaptive playback without ID tags may not work.

Side-by-Side Viewing of Mobile Devices

2011-12-19T17:42:00.000-08:00

One of the things we've found in our work with testing various mobile devices is that it is often very convenient to be able to see the same page delivered to two different devices in a side-by-side comparison. The visual impact of the changes can be enlightening.

It turns out that this is relatively simple to do, using a special launch page and custom playback script. In the script the UserAgentString is set to a device and a sub-window opens emulating that device. After the two [it could be more, if you have the need] sub-windows are opened eValid pauses playback, leaving both sub-windows open and live.

In this state, you can manipulate either of the two emulated devices in any way you wish, and see what the server renders in response. For backup, you can see the server response for the same page request in the parent screen. Here is a screenshot of Two Smartphones (iPhone and Samsung Galaxy) and of Two Tablets (iPad and Samsung Galaxy Tab). Both screenshots show the same page all three ways. And you can see the UserAgentStrings directly.

Here is the full writeup of how this is done, including a sample script and a typical parameter setting file: Side-by-Side Comparison of Two Devices. Sample screen images are shown in Side-by-Side Sample Screens.

Selected User Forum Posts

2011-12-14T10:28:00.000-08:00

Can your structural testing test "anything"? -- The limits of structural testing are detailed.
A High BU Count During Monitoring -- A discussion of the impact on timing accuracy with heavy CPU usage.
Can eValid collect data similar to Google Page Speed? -- Comparing client-side and server-side timing data.
Synchronize Very Complex Pages -- The art of AJAX synchronization explained.
Frames With A Lot Of AJAX/JavaScript -- How and shy eValid handles AJAX frame-intensive pages.
"Mouseover Recording" Turned On and Off -- Pros and cons of the mouseover recording option.
Tell me about how eValid can suppress popup modal dialogs? -- Popup handling explained.
What is the difference between MatchString and SyncOnText? -- Face validation versus internal DOM synchronization discussion.
Can you tell me more about eValid's architecture? -- Details about how eValid is built.
My recording of an AJAX application -- Discussion of AJAX test script durability.

Mobile Device Testing

2011-12-08T17:37:00.000-08:00

There has been a lot of interest recently in how to handle quality issues (content, performance, server capacity) for web applications that run on mobile devices. From the eValid perspective the main issues work out to these:

Device Simulation: Using the eValid SetUserAgent command to set the User Agent String (UAS) so that the subsequent browser activity is reported to the server as if it was being requested from the specified device. eValid does this without needing the actual hardware and software! The server responses to client-side user requests made in this way behave as if it was talking to the mobile device.
Content Validation: Once a script is running, you need to validate that the eValid rendered output matches what you expect in the actual device. Here is a Rendering Comparison for Common Devices Vs. Device Emulator. As you can see, other than the screen size, the rendering by eValid is identical to that for the mobile device. This confirms that evalid is rendering your mobile application correctly.
Functional Testing: To illustrate how this works we ran an experiment that applied a simple test to 100 Mobile Devices, and recorded download sizes and times for each cases. A sample of that data is shown here, with some Selected Screenshots of how the server content varied as a function of the different devices and software versions simulated (with download byte counts for each device shown).

The most interesting thing we saw was that the asymmetric ratio of Download Bytes vs. Download Time, which illustrates that some servers are not prepared to deliver mobile content to every kind of device.
Server Loading: Functional tests lift easily to a server loading context, with multiple eValid instances (we call them Browser Users or BUs) working in parallel to create the load. This is illustrated in the experiment that we ran to drive a mobile application from a simulated smartphone up to 1,000 BUs -- up to 1,000 simulated users. The resulting
Chart of Derived Internal Response Times shows that this particular server began to have a problem beyond ~200 simultaneous users.

The thing to remember is that eValid can effectively drive mobile devices that support web applications perfectly well, and can impose realistic load levels sufficient to identify bottlenecks.

Selected User Forum Posts

2011-11-22T15:38:00.001-08:00

The settings for sourceIndex and elementValue -- Rules about preserving internal variable values.
Why doesn't eValid support XPATH? -- eValid's internal XPATH capabilities explained.
The value of sourceIndex -- Ways to retrieve the current value of internal parameters.
Synchronize a playback on an invisible property value. -- Unusual ways of achieving synchronization.
My website changes in the middle of the run -- What to look out for when web pages change rapidly.
The minimum "settling time" -- How much time is spend "rendering" and "settling" a page?
Text Synchronization -- Details of how to do synchronization on page text.
How do I turn on Application mode or Desktop mode? -- Internal mode change hints.
How well does eValid scale? -- Details on scaling eValid to high repetition counts.
A Dynamic String -- Methods of manipulating a dynamic string.

Mobile Device Rendering Validation Experiment

2011-11-21T14:31:00.001-08:00

We had had some people asking whether the eValid rendering of a mobile web application REALLY matches up with what you actually see in the real device?

To illustrate that it does, we constructed a demo of four four popular smartphone and how they rendered a page compared with the eValid renderingwhen eValid is imitating the same device.The Rendering Comparison for Common Devices results speak for themselves.

As you can see in a side-by-side comparison (click th images and move them so theyare side by side) the HTML that is delivered by the server to the suitably disguised eValid browser is renders identically to what you see in the smartphone.

Webinar: Load Testing Mobile Web Apps

2011-11-07T16:01:00.000-08:00

Simulate 1,000's of Mobile Browser Users Full AJAX Realism on All Mobile Devices Quick Identification of Bottlenecks

Wednesday, 16 November 2011 2:00 PM Eastern Time / 11:00 AM Pacific Time
QA/Testing/Tuning/Performance projects need to qualify performance of complex AJAX web applications -- within strict budget and time constraints -- to make sure their server-stack setups can meet the load. The traditional methods of ramping up load, based on using HTTP/S simulations or "VUs", don't always work when asynchronous AJAX applications are involved. VU's don't do AJAX. You need a browser. eValid server loading methods offer both quick-to-create, realistic, and fully synchronized AJAX functional tests. Plus you can lift those tests into performance/loading scenarios that can involve 100's or 1,000's or 10,000's of Browser Users ("BUs") per machine. In this webinar you'll learn: how special eValid commands overcome problems with variable-time playback dependency; how to create full-reality AJAX tests quickly; how to adjust tests to be totally self-synchronizing under stressed AJAX conditions; how to incorporate tests in an eValid LoadTest scenario; how to launch 100's or 1,000's or 10,000's of Browser User (BU) instances; and, how to analyze consolidated performance summary data to identify server-stack bottlenecks. This unique approach demonstrates how eValid becomes a genuine force multiplier in your web application performance testing efforts.

Webinar
Topic
Summary

eValid Architecture and Structure: How eValid functional and performance testing works.
Functional Testing: How to make reliable recordings of AJAX applications.
Making AJAX Tests "LoadTest Safe": How to augment tests for complete AJAX synchronization.
Creating LoadTest Scenarios: How to use the LoadTest scenario editor to organize realistic LoadTest control scripts.
Running LoadTests: How to launch single and multiple-instance runs using "cloud computing" resources.
Finding Bottlenecks: How to read the LoadTest and other raw data to help spot server-stack issues.

You are cordially invited to attend this free Webinar.

Mobile Device Loading Experiment

2011-11-03T16:58:00.000-07:00

Using eValid's ability to mimic being a mobile device, we engineered a simple demonstration of a 1,000 browser user (1,000 BU) test of a public transportation system's schedule information page, which is intended to be delivered to users of mobile devices. You're in the station and you want to know when the next train for your destination is? But you don't want it to take too long to show you the results, or you might miss your connection.

We set eValid up to imitate an iPhone, and configured the LoadTest scenario to launch successive eValid instances at 1 second intervals. The test script takes about 15 seconds to navigate to the website and download the key schedule data page. We put the playback in "repeat mode" so that, after all 1,000 BUs were launched there would be the equivalent of 1,000 users driving the mobile device.

Finally, the single data point taken from each and every repetition — some 5,587 data points in total — was the time that the actual data page took to download. Here's the chart of the results as a function ot time within the scenario:

Chart of Derived Internal Response Times (Click to See Full Image)

The payoff information here can be seen at the beginning of the chart, when the download time was pretty much independent of the applied load -- up to about 200 BUs. After that, the more users requesting that page, the longer it took, so that at the end, with 1,000 BUs asking the download time was as long as 5 minutes. It looks like someone'll be missing the bus!

Here is the link to the full solution description: Mobile Device Loading Experiment – 1,000 BUs

Testing Mobile Web Apps

2011-10-28T15:18:00.000-07:00

An eValid user wrote:

Can you please explain in more detail how eValid simulates web applications on mobile devices? Also, you claim that your tests done by eValid when simulating mobile devices are functionally equivalent to using the mobile device? Can you elaborate?

Background
First, some background information and details... The underlying question here is, what does it mean to test a mobile web application?

It is important to make sure of the distinctions between how a mobile web application responds and how a mobile device responds. In other words, you have to ask yourself: are you testing a mobile device; or, are you testing an application that runs on a mobile device.

If it is the latter — testing a mobile device — then eValid is not well suited to that type of work. eValid is not a device tester — it is a web application tester.

But if it is the former then the question really reduces to, "what does it mean to test a mobile web application?"

eValid Solution
Being primarily a web application testing system the main focus for eValid is, from a user's perspective (that is, from inside a browser), how does the application behave? The key word is "behave," and to eValid that means the kind of content is delivered (is it correct or not?) and how fast is it delivered (does it meet performance thresholds).

When you use the eValid SetUserAgent command to set up eValid to act like a mobile device, the data eValid generates in response to requests from the server is no different from what the actual device generates from the server. In other words, if you change the "identity" of the browser as a test driver, you change the response from the server, and you effectively test the server's ability to support the "identity" of the browser.

And the key facts about quality — validation of response data and quantification of response times — are perfectly measurable from the eValid desktop. In most instances, mobile device download speeds are nominally in the same range as those for most desktop applications. The data delivered to a mobile device passes through a wireless gateway of some kind before reaching the device, whereas the same data has to pass through some kind of internet service provided before it reaches your desktop.

In other words, you don't need the actual device to stimulate the server responses, and you don't need the actual device to validate content and to measure performance timing data.

_________________

eValid Support

Recent eValid Technology Questions

2011-10-27T13:52:00.000-07:00

Recently there have been a number of eValid User Forum postings that raise interesting questions about eValid's architecture and underlying technology.

Can eValid analyze any web page? -- Are there limits to what kinds of pages eValid can analyze?
Does evalid have an API? What is it like? -- Details about eValid's Programmatic Interface
eValid operating with Web 2.0 and AJAX applications -- Pointer to the Business White Paper that sumarizes eValid's development history.
Any Kinds Of Web Applications -- What eValid can and can't test?
IE Browser -- eValid's dependence on the IE browser: how it works!
Drive tests using JavaScript -- Why eValid is built to never interfere with JavaScript operations.
Why doesn't eValid work at the transport level? -- The advantages of eValid's high-level approach are explained.
Can eValid test IPhones? -- Specifics on how to test mobile devices.

Using evalid With An AJAX Autocomplete Functionality

2011-10-24T15:18:00.000-07:00

This scripting example investigates how eValid can be used to validate an AJAX autocomplete function. The two variants of the script shown below operate as follows:

Introduce a partial string into the text area, for example evali.
Capture the contents of the DOM after the AJAX activity has provided some suggested completions.
Confirm that a required string, for example evalid, is present within the available suggestions presented by the autocomplete feature.

The goal of the script is to confirm that, among all of the possible suggested completions, that the target evalid string is among the options presented. If successful, this process effectively validates operation of the AJAX autocomplete function by confirming its results with known (expected) data.

Here is the link to the full solution description: Checking AJAX Autocomplete Functionality

Manipulating the Target Window

2011-10-21T09:16:00.000-07:00

How do you use the eValid DOM manipulation commands to modify an anchor-tag's attributes to deliver the page to a different specified new window than the one given in the HTML passage.

In this typical HTML passage if you click on the link the new page is opened in a new window named "NEW-WINDOW":

<A HREF=../URL TARGET=NEW-WINDOW>Title Of Page</A>

The key to modifying this link target dynamically is to recognize that any anchor tag that has a specified target will have the name of the target stored as one of the attributes in the defining DOM element. All you need to do is find the right element, and then put in a new string for the target page.

Here is the link to the solution description: Change the Target of an Anchor Tag Link To A New Window

Modifying DOM Elements Dynamically

2011-10-19T13:07:00.000-07:00

How do you use the DOM manipulation commands to modify a specific DOM element dynamically?

In this illustrative solution an eValid script demonstrates the ability to modify the DOM dynamically by using eValid DOM manipulation commands to change the background color.

Navigates to a page
Changes the background color for one element to COLOR1.
Waits 5 seconds (so you can see the effect).
Changes the background color for one element to COLOR2
Waits 5 seconds (so you can see the effect).
Changes the background color back to the original background color.
Waits 5 seconds (so you can see the effect).

Here is the link to the full solution description: Modify a DOM Element Dynamically

Update on Mobile Application Performance

2011-10-17T15:22:00.000-07:00

Some months ago we ran a small experiment on the Beta site of a mobile application in which we ran a small 100-BU (Browser User) test to see how performance of the application changed as the load increased, linearly, from 1 BU to 100 BUs.

As a followup, we repeated the experiment on the same application -- which now has become a full-scale "non-Beta" application -- with the results shown here:

Further Comparative Improvement of Performance Under Load

As you can see from the charts, things didn't necessarily get better, and you could say that there was some deterioration. But remember, that later data -- from over four months ago when the tests were first run -- is now from a live application. Right now it appears this application is "good" for up to about 25 simultaneous users, but performance drops off quickly after that -- as much as a 5x degradation beyond 50 simultaneous users.

Selected User Forum Posts

2011-10-14T15:48:00.000-07:00

Can eValid test applications using GWT and Vaadim? -- Tricks in testing a specialzed application
Complex Synchronizations -- Is there a resource problem?
Keep Separate Cookies -- More about saving your cookies!
Can eValid detect graphic differences in my app's GUI? -- Methods to confirm graphics are described.
Use of the cache during server load test -- Details on cache locking.
Can you guys run a monitor on a smart phone app? -- Mobile monitoring.
Run a batch mode scan? -- Background site analysis.
Why does eValid base its operation on IE? -- Justifications for IE dependence.
Can eValid test IPhones? -- How to spoof iPhones.
Pretending to be a mobile client -- More about mobile client emulation.

Webinar: Structural Testing for Mobile/AJAX/Web 2.0

2011-10-06T12:06:00.000-07:00

Run Functional/Regression Tests on Complex, Dynamic AJAX/Web 2.0/Mobile Applications Extremely Robust Tests Are Based on Structural Page Features

Wednesday, 12 October 2011 2:00 PM Eastern Time / 11:00 AM Pacific Time
When web applications often change in subtle ways, and your functional/loading/monitoring tests may begin to fail if your test scripts are "brittle". eValid's advanced structural testing commands overcomes this problem to help you achieve tests that survive page changes -- but still validate performance realistically. The eValid structural testing command set makes it possible organize tests entirely around structural properties of a page that don't change over time. A test done with structural testing methods can handle any AJAX/Web 2.0/Mobile application reliably and efficiently. Even when page structure and details (but not essential functional intent and effect) -- change substantially. Learn how to achieve greater productivity with smaller budgets, maximize your IT investments, and get more work done in less time with less energy. Build reliable, robust AJAX/Web 2.0/Mobile tests once -- and you won't have to worry about them again.

Webinar Outline

Introduction to eValid's Architecture.
Functional Test Creation: "What you see and do, is what eValid records, is what eValid reproduces".
Methodology Overview: Record From Life, Adapt From Page Facts
Index/Motion (Algorithmic/Structural) Command Summary
Typical Script Passages: "Recorded From Life" and Structural Versions
Script Enhancement for AJAX/Web 2.0: Bullet-Proofing Your Playback
Practical Experience & Recommendations


You are cordially invited to attend this free Webinar. REGISTER NOW!

Can you tell me more about eValid's architecture?

2011-09-02T17:32:00.000-07:00

An eValid user wrote:

Can you tell me more about eValid's architecture? Is it based on the Trident Rendering Engine?

Yes, the browsing part of eValid is based on use of the MSHTML (Trident Rendering Engine) libraries.

Although this makes eValid a complete browser (which emulates behavior of whatever version of IE you have installed on your machine), there are a few differences in the way eValid interacts with MSHTML. The main difference is that eValid has direct, immediate, "in-memory" access to the browser DOM and this unique capability makes much of what eValid does possible.

That architecture -- incorporating direct DOM access in a free-standing executable -- also is the reason why we can use eValid in server loading work by running multiple (100's or 1,000's of) instances to impose completely realistic load.

Also -- and this fact may be just as important -- the eValid engine does not make use of the JavaScript interpreter that is part of the browser. That is important because it means that applications that rely on JavaScript like modern RIA's that use AJAX do not experience any interference in their operation from the test engine itself. (You would not want a test engine to interfere with it is supposed to test, would you?)

How to record synchronization points from the GUI?

2011-08-29T13:06:00.000-07:00

An eValid user wrote:

I need to make a good recording of an AJAX app, but I need to know how to record synchronization points from the GUI. I don't want to have to manipulate the script at all, please!

That's actually very easy.

The most common synchronization point is "Validate & Synchronize on Selected Text." This is 99.99% reliable even for AJAX applications (there are always 0.01% where you have to resort to "structural commands").

The protocol is simple to follow:

On each page, as you make your recording, select a piece of visible text with the left mouse button so that it is highlighed. Then click eValid: Record > Validate > & Synchronize > Text String, or simply press Ctrl-Y.

You'll see eValid record a command in the Script Window.

Now, knowing that the synchronization logic will wait for the highlighted text that you chose to show up on the screen at playback time, you can then take the next action. Click on a button, click on a link.

Then, wait for the application to settle down and repeat the sequence.

Simple enough. Just remember: Wait --> Sync --> Act, [Wait --> Sync --> Act, --> ...].

Advanced Structural Testing Webinar Movie Available

2011-08-25T12:40:00.000-07:00

The "movie" recording of the eValid Webinar, Advanced Structural Testing of Mobile/AJAX/Web 2.0 , that was given on 17 August 2011 by Dr. Edward Miller & Ms. Rita Bral, is now available. To view the movie simply click Request.

Selected User Forum Posts

2011-08-24T16:30:00.000-07:00

The most-common bottlenecks -- Where best to look for problems?
The nice banding of times in your demo scripts -- How to set up intra-script measurement timings.
Regular expression matching -- Relative advantage of regular expression (XPath) matching.
Does eValid have a script debugger? -- Available support for script checkout and problem resolution.
How do you test an auto-complete function? -- Methods for testing a common AJAX supported feature.
The right contents of a property -- Digging deep into the DOM for the right answer.
How do I keep certain pages from NOT being scanned? -- The role of the Exclude URLs file in site analysis.
Is eValid a plugin for IE? Or what? -- Resolving issues about eValid's architecture.
Why is AJAX test synchronization so important? -- Helping keep browser state to insure test realism.
Can eValid test a mobile device? -- Tricks for testing mobile device application responses.

How long do you think it will take me to learn to do a very complex AJAX application, starting from scratch with eValid?

2011-08-17T11:28:00.000-07:00

An eValid user wrote:

How long do you think it will take me to learn to do a very complex AJAX application, starting from scratch with eValid?

The answer depends on what kind of web application you are testing.

(1) If you are driving a "conventional" (non-AJAX, see below) website, then eValid orginarily has sufficient internal automatic playback synchronizations built in so that you don't need any extra steps. Just type and click and eValid will record what you've done and will play it back, no problem.

Standard advisory note: If you record logging in to an application and then play back that script withOUT having logged out, then it's likely the playback will fail. You're still logged in! Of course it will fail. Good practice says, either record after having logged in and don't log out, or record the login and make sure you end with a logout.

(2) For AJAX applications the situation is more difficult. Because of AJAX activity in the browser, you may not get an accurate playback because the time that AJAX takes varies with web load and other factors. The "out of the box" recording with no special steps may or may not fully synchronize at playback time.

However, there is a very simple recording protocol that we recommend to use when you are dealing with an AJAX application. The basic idea is to use the eValid: Record > Validate > & Synchronize > Text String sequence from the eValid GUI to create synchronization logic as you make the recording. The sequence is simple: Act, Wait, Sync, [Act, Wait, Sync]...

The steps are simple enough: (a) navigate to the next page in the test plan, (b) wait until it is fully loaded, (c) highlight a piece of text on the page, then (d) execute the above command sequence (or use the Ctrl-Y shortcut). The resulting script then has a synchronization step that has to be satisfied before every action -- and this will assure fully synchronized playback.

Additional Note: In certain rare cases for AJAX applications you may need to resort to other, more powerful, DOM-based synchronizations if the recording protocol described above doesn't work. Just look up "Synchronization" in the eValid User Manual for complete details on this.

Selected User Forum Posts

2011-08-10T13:02:00.000-07:00

How close is eValid to being identical to IE? -- eValid underlying technology explained. [01Apr11]
What makes eValid a "test enabled web browser" anyway? -- Some details on eValid's architecture.
Can you run eValid in monitoring mode from cloud resources? -- Guidlines for cloud-based monitoring.
Absolute Modes with MS-Notepad -- Creating scripts with eValid driving Notepad.
Can eValid handle changes in frame names? -- How to handle dynamc frame names.
Can you set up site analysis in a monitoring mode? -- How ro provision nightly site analysis runs.
eValid's capability to analyze DOM changes -- Input/output commands for DOM property values.
How does eValid support "CI"? -- Jumping into the Continuous Integration controversy.
About setUser agent command in Evalid -- Details on how to spoof non-IE browsers.
Why can't your desktop synchroninzation be used for AJAX? -- Pros and cons of alternative synchronization methods.

What is the best way to record a test that you know will run when you play it back?

2011-08-08T13:23:00.000-07:00

An eValid user wrote:

What is the best way to record a test that you know will run when you play it back?

Webinar: Advanced Structural Testing for Mobile/AJAX/Web 2.0

2011-08-03T16:35:00.000-07:00

Run Functional/Regression Tests on Complex, Dynamic AJAX/Web 2.0/Mobile Applications Extremely Robust Tests Are Based on Structural Page Features

Wednesay, 17 August 2011 2:00 PM Eastern Time / 11:00 AM Pacific Time
When web applications change in subtle ways, your functional/loading/monitoring tests may begin to fail if your test scripts are "brittle" -- i.e. break easily. eValid's advanced structural testing commands overcomes the "brittle test" issue to help you achieve tests that survive page changes -- but still validate performance. The eValid structural testing command set makes it possible organize tests entirely around structural properties of a page that don't change over time. A test done with structural testing methods can handle any AJAX/Web 2.0/Mobil application reliably and efficiently. Even when page structure and details (but not essential functional intent and effect) -- change substantially. Learn how to achieve greater productivity with smaller budgets, maximize your IT investments, and get more work done in less time with less energy. Build reliable, robust AJAX/Web 2.0/Mobile tests once -- and you won't have to worry about them again.

Webinar Outline

Introduction to eValid's Architecture.
Functional Test Creation: "What you see and do, is what eValid records, is what eValid reproduces".
Methodology Overview: Record From Life, Adapt From Page Facts
Index/Motion (Algorithmic/Structural) Command Summary
Typical Script Passages: "Recorded From Life" and Structural Versions
Script Enhancement for AJAX/Web 2.0: Bullet-Proofing Your Playback
Practical Experience & Recommendations

You are cordially invited to attend this free Webinar.
REGISTER NOW!

Page Performance Optimization Methods

2011-08-02T23:20:00.000-07:00

There has been a great deal of interest in the web community recently about how to speed up web page delivery. At least some of this discussion is prompted by the increasing sizes of web pages, particularly for AJAX and Web 2.0 applications. For example, a few years ago a 100 KByte page would be considered "large," but now we see some pages that download 1 MByte, and these are considered "normal". Some website, e.g. the CNN site, regularly download 1.5-2.5 MBytes -- and that's not counting videos. In earlier times web page component counts could be in the low tens, now the component counts are 100+. On a recent day, CNN's page had 113 individual URLs involved.

To improve average download performance you have to buy your customers more bandwidth (this could be expensive!), or you need to figure out how to reduce the size of your pages (tedious, but very effective). But to fix something you have to understand it, and that's where a couple of eValid tricks come into play.

Page Metrics Popup

This is an eValid feature that runs along with your browsing activity.

As you navigate through a website, after each page is fully downloaded you see a simple popup like the one at the right (click on the image to see the full picture).

The data extracted by eValid includes the size of the base page, the total number of URLs involved, and all of the details about the internal structure of the page including the total size of the download.

Here are the details on eValid's Page Metrics Popup feature.

PageSpeed Process

The eValid PageSpeed Process uses the "detailed page timings" feature to collect page component byte-count and download times, in the order in which each component is recovered and rendered.

The resulting "stack chart" reveals detailed the download time and size of each page component, by simply hovering your mouse over the display bars. This way, you know which element is the bottleneck.

Bottom Line: eValid gives you all the details on what's what about your page. It's up to you to fix the problems eValid analytics reveals. That's a lot easier to do if you know what's wrong.

Drive tests using JavaScript?

2011-07-25T15:12:00.000-07:00

An eValid user wrote:

Does evalid drive tests using JavaScript? If not, why not?

There are many reasons why evalid does NOT "drive tests using JavaScript" and some of them go back to the very beginnings of eValid.

A main goal of the eValid technology development was to provide a test bed or test driver that could work efficiently and easily with web browser enabled applications. In that beginning period -- going back to over 10 years ago -- we had completed internal "proof of concept" implementations of several kinds of testbeds. We tried out "wrappers," we tried out on-board programs using JavaScript passages, we tried out various "browser editors"...in short, we tried all of the combinations that we could think of.

What we discovered in our early engineering work was that there was a vast difference in the overall performance -- and a vast difference in how much each testbed trial solution interacted with the behavior of the web application. It was the potential for "bad interaction" which pointed the way for the team, because the underlying assumption was that, to be a good test driver, the test drive could not get in the way of the thing being tested.

Of all of the alternatives that were considered, we leaded strongly on the one prototype product that was built with an open-source browser. Having that kind of direct, immediate, "intimate" control of all of the signals and events and transfers and other activity while the browser gets the page components, assembles them into the internal document object model, and finally renders them to the user -- that level of interaction, we felt, was "perfect" for what we envisioned eValid to do.

We sill had to deal with the question of "overhead." No matter how efficiently you build a test driver, it is software so it takes some fraction of the CPU to execute. We settled on a target of not more than 0.1% interference with the application. And, the solution we ultimately chose, and which has become the eValid architecture, has met that goal in every regard.

So, the REAL reason that eValid does not use JavaScript to drive the test is that it is (a) too messy, (b) has too high an overhead, (c) imposes too large a footprint for the test engine, and (d) interfers with the normal operation of a web page. With the growing importance of AJAX this choice is more than justified -- eValid's architecture says "hands off" to the co-executing JavaScript passages that implement AJAX -- and this means that eValid tests are far more accurate and far more reliable than would have been any of the other architectures which we studied, and discarded as wanting.

__________________
eValid Tech Support

Do you have a standard pricing structure for building an eValid test? What are the limits on size and complexity?

2011-07-22T15:41:00.000-07:00

An eValid user wrote:

Do you have a standard pricing structure for building an eValid test? What are the limits on size and complexity?

Yes, in addition to providing licenses to the eValid product so you can build your own tests, we also provide services that generate tests according to you own specification.

We have (as you might expect) a great deal of experience in building functional tests using eValid, and we generally are able to deliver test scripts to you that are 100% self-synchronizing and also 100% load-test safe.

Self-synchronizing means that you can make any adjustment to the Wait Time Multiplier you like -- including eliminating all delays, a Wait Time Multiplier of zero -- and the test will still run successfully. This is done with eValid structural commands and with eValid DOM-synchronization commands.

Load-test safe means that the tests can be run in multiple eValid instances on the same machine, without interfering with each other. That way, such a test can run 100's of browser users on one regular machine -- and 1,000's of browser users on a beefy, multi-user computer (like the ones we deploy when doing cloud-based loading work for customers).

We are so confident that we can complete such test script development projects on a tight budget -- and who isn't interested in having tight control of costs these days? -- that we offer such test creation mini-projects for a typical fixed-price fee $1,495 per script. And we take all of the engineering risks if it takes us longer than our nominal time budget. We have a 100% deliver success ratio on such mini-projects.

There are size constraints and limits of course, but for this very modest price we generally deliver a test script that has all of the above properties but in addition typically has coverage for up to 200 test steps (test plan imperatives like "click this..." or "select that..."), and can play back as long as about 5 minutes with no wait times.

You might think that 5 minutes is not a long time nor a very comlicated script, but when eValid scripts run with zero delay times, we find that the resulting test script, which may be upwards of 400 lines or more, will cover a very great deal of "functional ground."

All we need to get going on a mini-project like this is a step-by-step outline of your test, and appropriate access credentials for your application. If you don't want to write your test plan down, we can even include doing that step with a phone interview.

Driving a complex AJAX APP

2011-07-21T11:56:00.000-07:00

An eValid user wrote:

Driving a complex AJAX APP...Tell Me How

This is a good question and it raises an interesting point about how modern web browser enabled applications that involve large amounts of AJAX actually operate.

If you recall, when AJAX is used there are JavaScript programs that run in the JavaScript engine (inside the browser) that do all kinds of things, but mainly control communication and data transfers between the browser and the server, based (in part) of user inputs. Think in terms of what you see on your screen when you type strings into a search engine that has implemented some kind of "autocomplete" feature. In that case there is a LOT of back-and-forth to/from the server.

The amount of processing time for that used to be very small, almost "invisible" to you as a tester in the scale of time that relates to your test cases.

But, as the total volume of JavaScript has grown the total time that running those (admittedly, wonderful programs) also grows. Accordingly, the amount of time it takes for the browser to settle down and reliably accept an input from the test engine grows.

That's where the problem arises.

eValid's structural commands are done separately from that JavaScript (this is an important architectural feature of eValid's implementation -- the non-interference with JavaScript). Even so, we have seen cases in which the eValid structural commands are "too fast" for the browser...we have heard of cases when, due to rapid-fire issuance of structural commands that update the DOM, the browser actually aborts execution!

So, what to do?

What we recommend is to insert a Delay 10 command, or perhaps a Delay 100 command after major structural command sequences to give the browser time to stabilize after the changes. For some cases, we have even had to use Delay 1000 -- that is, delay an entire second -- to prevent failure.

Or, duh, you could get a faster client computer (but you didn't want to hear that, did you!).

_____________________
eValid Tech Support

Do you have any recommended maximum length of a test?

2011-07-18T13:22:00.000-07:00

An eValid user wrote:

Do you have any recommended maximum length of a test?

Another way to ask this might be, beyond how many test steps is the state of the browser no longer going to be repeatable.

There's no imposed limit on the length of time an eValid script can play back, and we have seen some scripts that take 15+ minutes the execute.

One script we worked with recently had about 250 separate steps (about 500 commands overall) and took about 10 minutes to play. That's quite long!

That is quite long, and the elapsed playback time, even when accellerated by reducing waits to the minimum through synchronization, imposes delay when you are trying to modify or debug a script.

Obviously there is no general rule, but we would argue that a playback time above 5 minutes (300 seconds) is going to run into either state-loss or tester-patience-loss issues before long.

____________________
eValid Tech Support

What happens to an evalid site analysis run if my website changes in the middle of the run?

2011-07-16T21:33:00.000-07:00

An eValid user wrote:

What happens to an evalid site analysis run if my website changes in the middle of the run?

Here is the way the site analysis engine works.

It starts (at the page you specify) and makes a list of all of the links on that page and then, one by one, it attempts to visit every page on that list.

On each new page, it ADDs to the "work list" by enumerating all of the links on that page.

No page is visited a second time, and the scan stops when the "work list" is empty.

Now, if one of the pages changes during the scan, then the NEW information will be used as the basis for adding items to the "work list". If the new page has left out a link that was present at the time the scan began, then that page will not land in the "work list" and will never be visited.

So, bottom line, if the site changes there is a risk that some pages that were present at the time the scan started will not be visited, and there is a corresponding risk that some pages that were not present at the time the scan started will be visited.

Practically speaking, if you are running a scan regularly, ultimately all of the pages' actual [stable] content will be visited, so this effect may not be an issue for you. But a one-off scan may be in error.

It's best, of course, to run scans on stable sites that are not undergoing maintenance and/or upgrade and/or change.

_________________
eValid Tech Support

Testing With DOM Interactions -- Guide to Examples (Part 2)

2011-07-12T10:29:00.000-07:00

This article presents eValid example scripts that illustrate use of the index/motion (structural testing) DOM manipulation commands. These scripts were developed using the available DOM Technology Resources to accomplish the results demonstrated in each "snippet".

In most cases the index/motion (structural testing) scripts are extremely robust in that these test script passages do not depend on specific page or application properties, but only on objects and effects that can change location and visual detail.

Tests built using these techniques survive the effect of dynamic page changes, including inconsequential changes that don't affect the underlying functionality. They also address basic test playback synchronization issues and can still provide fine-detail client-side performance data extraction suitable for large-scale loading experiments.

Link	Short Example Explanation
Third Item in Dropdown List	Find the third item in a dropdown list and click on it.
Nth Item in a Select Box	Choose the Nth entry in a select box and click on it.
Select Two Elements in a CheckBox	Select two elements of a checkbox.
Find and Follow a Specified Link	Find a link based on knowing something about it and then follow it.
Find A Dynamic Tag And Click It	Locate a dynamic element tag and then click on it.
Text Entry Into a JavaScript Editor Field	Find a particular input field that is handled by JavaScript and type specific text into it.
Validate That One Item Matches Another	Extract two values from different spots on a page and compare the extracted values.
Using Regular Expressions	Using the IndexFindElementEX command to find using multiple "name=value" pairs.

Testing With DOM Interactions -- Guide to Examples (Part 1)

2011-07-11T12:05:00.000-07:00

Link	Short Example Explanation
Navigation to URL	Shows how to navigate to a particular URL.
HTML Push Button on a FORM	Locate and click on a push button on a FORM.
Radio Button On A FORM	Locate and push a radio button on a FORM.
Check Box On A FORM	Locate and check a particular box on a FORM.
Single Form Text Input Field	Find a particular text field in a FORM and type something into it.
Multiple Form Text Input Fields	Type in multiple files in a FORM.
First Item in Dropdown List	Click on the first item in a dropdown list.

Selected User Forum Posts

2011-07-08T17:19:00.000-07:00

Can multiple browsers run from multiple data streams? -- Feeding data into each browser separately.
Random wait time between events -- How to use the WaitRandomUniform command.
Login Account Name -- Details of handling logins and internal states.
Can eValid click on a hidden button? -- Doing the impossible: processing something that isn't there!
Testing AJAX Applications For Load -- How to handle loading of AJAX apps.

Selected User Forum Posts

2011-07-07T13:31:00.000-07:00

AJAX Page is updated continuously with feed from my website -- How evalid handles dynamic pages.
How do you coordinate between two or more browsers? -- Multiple-browser locking.
What good is "geometric test"? -- More about making good geometric tests.
How does eValid keep track of frames? -- Details of frame processing.
Analyze DOM values in a current page -- Reading and analyzing specifc DOM values.

Can eValid create a non-visible-text validation from the GUI?

2011-07-06T13:52:00.000-07:00

An eValid user wrote:

Can eValid create a non-visible-text validation from the GUI?

Yes, there is a programmed-from-the-GUI optional sequence that will automatically construct a command to validate a selected object property. This command works from the PageMap and allows you to validate ANY property that is present in ANY object anywhere on the page.

A word of caution: this command, and the GUI-sequence that leads to it. is somewhat more technical that the vast majority of eValid validations. We put it in there because it was appropriate for completeness -- to have a least one command that digs into the DOM and constructs something in your recorded script for you.

As you can imagine, there are many ways that the GUI sequences can lead you to construction of a script -- "from the GUI" and thereby somewhat automated -- and you might ask, "Why doesn't eValid have a richer set of such GUI sequences?"

The answer lies in this observation: No matter how complex you create the from-the-GUI operations, there will always be some case that you haven't thought of, and then you're stuck. Instead, with eValid our idea was to provide the mimimum amount of GUI-based help needed, consistent with keeping the GUI simple and intuitive. (We have seen some other offerings that boggle the mind with the complexity of the GUI sequences involved!)

So, instead -- to give the user the power needed for the "outside the GUI" case -- we created a simple set of "structural testing commands" that are easy to learn, few in number, and ultimaly can do anything and everything that you need. Here is a link to the manual page for that:

Structural Testing Commands

The set of primitive operations you have with these commands -- get, set, put, find, move, etc. -- are able to handle ANY situation you can imagine.

In effect, rather than try to guess what you might need (and know that there will always be a case in which it "won't work"), we have provided a simple programming tool -- actually a kind of "program the browser" tool -- that is FAR more efficient in terms of your time and effort than an elaborate and ultimately ineffective GUI sequence.

We hope you agree with our simple is better approach to eValid design and operation.

_____________
eValid Support

Selected User Forum Posts

2011-06-20T13:22:00.000-07:00

In your PageMap display, how often is the data updated? -- How PageMap works with dynamic pages.
When To Use Which Special Recording Modes -- eValid's alternative test recording modes.
Geometric Testing -- All about building super-durable tests.
64-bit Machine -- Operational status of 64-bit architectures.
"Safety factor" for AJAX script playbacks -- A critical factor for monitoring applications.

Selected User Forum Posts

2011-06-15T16:02:00.000-07:00

eValid Script Language -- eValid's command language is simple, clear and easy to use.
Why doesn't eValid have graphs of hit counts? -- Hit counts are for HTTP/S. evalid measure browser events.
How many simultaneous monitors do you recommend running? -- Performance and capacity estimates from real-world testing.
Give me the main reaons I should buy the evalid suite? -- The short answer: It just works!
The "Special Recording Modes" -- The simplest recording method is usually the best.

An Interesting Scalability Result

2011-06-02T15:57:00.000-07:00

As our BLOG readers may know, we have been working recently with developing better understanding of the scalability of the eValid solution. "Scalability," simply put, describes the degree to which the solution can be ramped up to provide for large numbers of Browser Users (BUs) to drive load into a server stack.

Using a single user on a modest Windows machine, we regularly achieve 100 BUs in parallel. Using multiple users and a fairly large cloud-based machine image we have hit 10,000 BUs.

But a different way of doing things, which involves having one parent eValid BU launch multiple sub-windows, seems to be capable of besting even these very good scalability numbers. The advantage of using sub-windows is that each one takes far less memory footprint than a complete browser instance. But a disadvantage is that all of the sub-windows can't be active at the same time; so this solution only works when the tests you're running have a low-enough duty cycle (percentage of time active) so that cumulatively they don't exceed 100% of the main browser's time-line.

Within those constraints, however, this approach does appear to work quite well. Here is a screen shot of a typical Windows desktop with a total of 50 eValid browsers, each running 10 sub-windows, for a total of 500 + 50 separate activities: 550 Browser Screen Desktop Image. (Look closely at the bottom of the images and you'll see the 550 browser + window count.)

Isn't one picture worth at least 550 words?

Selected User Forum Posts

2011-06-02T14:37:00.000-07:00

User-perspective Usability Testing -- Alternative uses for evalid technololgy.
Compare the work of making a recording and then scripting -- Recording from life and then hand modifying is much less work.
Does evalid have a way to measure inside the broswer user? -- The art of partial test timing data collection.
Local Database -- The SQL feature helps get test details for monitoring.
User Experience Measurement -- Applying the SaveRecord command.

Update on Results From A Small Experiment

2011-06-01T13:00:00.000-07:00

Previous BLOG items described a small experiment aimed at demonstrating how eValid can test the performance delivered to a particular smartphone. You can read about that experiment in detail in our Mobile Agent Loading Experiment #1 writeup.

To see if there had been any changes as the result of our experiment we reran the same test on the same application approximately 3 weeks after the first run. We're pleased report that improvements to that application's server stack has apparently resulted in a a 50% improvement in performance and capacity. You can see the relative improvement in the two charts shown in this Comparative Improvement of Performance.

How eValid Scales

2011-05-31T16:53:00.000-07:00

There has been a lot of discussion in various forums -- including in our own eValid User Forum -- about how to scale tests up from singleton runs to large load tests. The goal seems in all cases to be to preserve the precision and accuracy of an eValid playback but still run enough evalid's in parallel to yield a realistic load.

The natural questions are, when you increase the number of eValid instances, what resources are used up first and what limits are hit along the way. Here are some of the main points that contribute to this discussion, in order of where they form limits:

Initial RAM Usage
When eValid finishes loading -- even when the initial page is about:blank the footprint is about 12 MBytes. That means that an initial load of 100 BUs will consume 1.2 GBytes of RAM. Which is certainly feasible with a 3 GByte machine.

RAM Usage Growth
As each browser continues working its footprint grows because the browser stores in its workspace all of the JavaScript files and a lot of other material that it picks up along the way. The longer the test the greater the growth. We have noted footprints for very long tests as much as 100+ MBytes per BU. (This behavior argues for smaller tests and/or for more-often restart of the BUs in your load test run.)

Cache Space
As your tests run on and on, with more and more pages, there are more pages stored in the cache. In LoadTest runs, though, this doesn't matter as much because you typically don't use cache at all. (Periodic DeleteCache instructions are a good idea.)

Virtual Memory Size
There are special instructions that come with eValid to make adjustments to the Virtual Memory size, etc. This is not a fixed number, but when you do the "machine adjustments" on most Windows systems you ask Windows to use the best size of the paging files to the maximum available.

GDI Object Count
You can't control the GDI object count, but we know (from our conversations with Microsoft technical support) that you may run past the built-in Windows limit of between 256 and 65,536 GDI objects per user.

Input/Output [I/O] Bandwidth
This depends a lot on the web application you are testing. Some are very economical of I/O capacity, and others are very hungry. The only way to make sure this is not a limit is to look at the PerfMon data in the Task Manage to monitor the networking activity.

There are some other tricks to maximize the number of BUs per run -- tricks that we use to get over 10,000 BUs off one machine. That'll be a subject for a subsequent BLOG post.

Selected User Forum Posts

2011-05-25T20:16:00.000-07:00

The times in your "banded timings" for a script -- Accuracy is very important, and eValid timings deliver.
Comparing eValid and LoadRunner/TruClient -- Why eValid details are exposed rather than kept hidden.
eValid behave in LoadTest mode -- How good is the error recovery in LoadTest mode?
Is eValid agile? -- The best way to think of eValid in an agile context.
Bank Transfer Test Page -- How to move values around from one screen to another.

Selected User Forum Posts

2011-05-18T09:02:00.000-07:00

Machine Adjustments -- How to make special OS parameter adjustments to increase LoadTest driver machine capacity.
Do you have some simple way to look up a command? -- Quick access to the syntax of a command from the script file.
Is eValid patented? -- Discussion of eValid intellectual property protection.
Geometric Testing? -- How geometric testing works.
Problem during selcting the evm file in evmanage -- How to adjust parameters in an eV.Manager setup.

Results From A Small Experiment

2011-05-13T13:44:00.000-07:00

Previous test reports pointed out how eValid pretends to be a particular smartphone, looks at how several different websites behaved in delivering material to that smartphone client browser, and studies the differences in the responses by the Amazon website to different smartphones.

Next, we tried a simple experiment with a smartphone application that uses AJAX. We wanted to see how the use of AJAX in an application affects smartphone behavior. For our test, we chose a beta-status application of a regional transportation system that offered real-time schedule information delivery to smartphones.

The experiment involved a simple test: On the mobile application, eValid pretending to be an IPhone 4 browser, request schedule data, synchronize final delivery, and measure the amount of time that delivery took. We found from static tests of that script that the data download on a per-test basis was about 330 KBytes.

The LoadTest script was set to start Browser Users (BUs) off at a constant rate until a total of 100 was reached. The repetition count for each BU was set high enough so that at the end of the test all 100 BUs were repeating the test -- all independently. The timing tests were very interesting. The LoadTest Data Report suggests that the response time for this application degrades by a factor of about 5:1 when the number of simultaneous requests grows above 75.

(Please note that this experiment was done to illustrate eValid capability; we're not making any recommendations to the agency actually responsible for the application implementation.)

You can read about our short experiment in detail at this page: Mobile Agent Test Page -- Loading Experiment #1.

Selected User Forum Posts

2011-05-12T16:45:00.000-07:00

Synchronise the BUs -- Intra-script cross-BU synchronization options.
Screen comparison test check -- How to compare two FLASH pages.
Load test a shopping site -- Parametric access to multiple shopping cart selections.
The playback speed -- Methods to vary the playback speed of a browser instance.
A DOM index/element number -- How playback of a page link is done using DOM details.

More about Mobile Phone Response

2011-05-10T09:19:00.000-07:00

A previous item pointed out how eValid could pretend to be a particular smartphone, and looked at how several different websites behaved in delivering material to that smartphone client browser.

A companion experiment that we did tried out five different smartphones on the Amazon website. (We chose Amazon because it seemed to be one that had very good smartphone options.) The results are shown in Mobile Agent Test Page -- Multiple Phones.

What is interesting about these data points is how much the size of the download varies from phone to phone: a ratio of 6.15+ from largest to smallest download volume.

If you are concerned with optimization of an application to a range of different smartphones this comparison method will be of great interest.

Selected User Forum Posts

2011-05-09T13:33:00.000-07:00

More Than 50 Simultaneous Users -- LoadTest runs with multiple user account details.
I Don't Have Fixed ID Tags -- Overcoming pages with dynamic ID tags.
eValid Structural Commands -- Resource pages for eValid structural testing.
Recording the mouse position -- Explanation of eValid recording process for mouse clicks.
"Application mode" recording -- Recording from desktop (not browser) objects explained.

eValid Applied To Mobile Devices

2011-05-05T15:53:00.000-07:00

In response to some user's questions, we recently did some experimentation with evalid imitating different mobile devices. The question was, can eValid show the differences between server response on a variety of mobile devices?

We used the SetUserAgent command to set eValid to provide authentication as if from a different user agent. We chose to use the user agent string for an iPhone -- not for any particular reason and certainly not an endorsement -- just to see what would happen when imitating that popular device.

Next, we chose five popular sites to run a simple test on: Amazon, Meebo, Samsung, Sony, and Verizon. (Again, no endorsement implied).

The outcome of our experiment is shown in this simple summary: Mobile Agent Test Page

As you can see from this page, in most cases the servers responded differently -- as you would expect -- when told to feed data to a mobile device. The side-by-side comparisons of the single response page show how different things are. For three of the five applications the size of the delivered data was < 20% of the full-size file.

Note that the page rendering appeared to be quite normal (except for total screen area). We didn't note any essential differences.

But not in every case. The Sony and Verizon mobile downloads are around 90% of the full-size download. We checked the actual content and found for these it appears there were some Flash items and also some movie items in what is sent to the mobile device.

One obvious implication is that this technique may be very valuable in using eValid to drive substantial loads into a server that is serving mobile applications. eValid's inherent scalability is what does the trick.

Webinar: The eValid Real Client Loading Solution

2011-04-24T12:31:00.000-07:00

Functional Testing of AJAX/Web 2.0 Applications
Server Loading Scenario Development
Server Load Testing with eValid's Real "Browser Users"

Wednesday, 4 May 2011
2:00 PM Eastern Time / 11:00 AM Pacific Time

This webinar will focus on testing AJAX/Web 2.0 applications quickly and realistically. Whether you are focused on desktops, handhelds, or mobile applications, the number of different users makes predicting demand almost impossible. An incorrectly balanced server stack could create delivery contention and delays, and lead to customer click-away -- customers you can't afford to lose.

The eValid client-side testing approach is the original browser based test solution. eValid is a self-contained one-stop real client based solution for all of the most important quality and performance issues of AJAX/Web 2.0 applications. The eValid consolidated approach becomes a genuine force multiplier in your web application testing and loading efforts:

• Achieve effective functional testing of your AJAX/Web 2.0 applications.
• Create realistic loading scenarios with multiple functional behavio
• Impose very large real client application loads on your servers.

The eValid suite meets these requirements in a unified, easy-to-use, and easy-to-understand solution. eValid doesn't require programming skill, doesn't need special plugins or adapters. eValid runs quickly and efficiently. You'll achieve web quality solutions in days or weeks rather than weeks or months.

In this webinar you'll learn these important lessons:

AJAX/Web 2.0 Functional Testing: How to record perfectly reliable web application tests, out-of-the-box, including complex AJAX/Web 2.0 playback synchronization issues.

Handling Mobile Devices: How to use eValid features to model handheld and mobile devices.

Creating LoadTest Scenarios: How to organize multiple eValid functional tests into LoadTest scenarios that model your actual user's activity mix.

Running LoadTests: How to launch eValid load testing scenarios that can run 100's, 1,000's or even 10,000's of browser users (BUs).

Realistic Data: How to extract detailed performance data from each BU's activity.

Selected User Forum Posts

2011-04-20T14:28:00.000-07:00

Synchronise the BUs -- Intra-script cross-BU synchronization options.
Screen comparison test check -- How to compare two FLASH pages.
Load test a shopping site -- Parametric access to multiple shopping cart selections.
The playback speed -- Methods to vary the playback speed of a browser instance.
A DOM index/element number -- How playback of a page link is done using DOM details.

Selected User Forum Posts

2011-04-11T17:47:00.000-07:00

More Than 50 Simultaneous Users -- LoadTest runs with multiple user account details.
I Don't Have Fixed ID Tags -- Overcoming pages with dynamic ID tags.
eValid Structural Commands -- Resource pages for eValid structural testing.
Recording the mouse position -- Explanation of eValid recording process for mouse clicks.
"Application mode" recording -- Recording from desktop (not browser) objects explained.

Selected User Forum Posts

2011-04-06T18:07:00.000-07:00

Synchronise the BUs -- Intra-script cross-BU synchronization options.
Screen comparison test check -- How to compare two FLASH pages.
Load test a shopping site -- Parametric access to multiple shopping cart selections.
The playback speed -- Methods to vary the playback speed of a browser instance.
A DOM index/element number -- How playback of a page link is done using DOM details.

Selected User Forum Posts

2011-04-04T14:50:00.000-07:00

More Than 50 Simultaneous Users -- LoadTest runs with multiple user account details.
I Don't Have Fixed ID Tags -- Overcoming pages with dynamic ID tags.
eValid Structural Commands -- Resource pages for eValid structural testing.
Recording the mouse position -- Explanation of eValid recording process for mouse clicks.
"Application mode" recording -- Recording from desktop (not browser) objects explained.

Webinar: All About AJAX Testing: Functional + Monitor + Load

2011-03-25T16:20:00.000-07:00

Functional Testing of AJAX Applications
Provisioning AJAX Scripts for Monitoring
AJAX Server Loading and Capacity Checking

Wednesday, 30 March 2011
2:00 PM Eastern Time / 11:00 AM Pacific Time

This consolidated approach demonstrates how eValid becomes a genuine force multiplier in your AJAX web application testing, monitoring, and loading efforts. eValid is a self-contained one-stop solution for all three of the most important quality and performance issues of 2011 for AJAX web applications:

Achieving effective functional testing of an AJAX application.
Confirming continual correct AJAX application operation (monitoring).
Estimating required AJAX application server capacity needs.

The eValid suite meets all of these requirements in a single easy-to-use and easy-to-understand solution. eValid doesn't require programming skill, doesn't need special plugins or adaptors. eValid runs quickly and efficiently, and you'll arrive at a web quality solution days or weeks earlier.

In this webinar you'll learn these important lessons:

AJAX Functional Testing: How to record perfectly reliable AJAX tests, out-of-the-box, including complex AJAX playback synchronization issues.
Readying AJAX Tests for Monitoring: How to provision an eValid script for use in client-side monitoring, including interfacing the results into popular reporting engines.
Creating LoadTest Scenarios: How to insert functional tests into a server loading scenario that can run 100,s, 1,000's or even 10,000's of browser users (BUs).
Realistic Data: How to extract detailed performance data from each BU's activity.

Selected User Forum Posts

2011-03-11T17:19:00.000-08:00

Login Recording -- The most common problem with state preservation.
Full Synchronizing a Typical AJAX Page -- It is very simple to make an AJAX page synchronize.
The details of the property value for a web page -- Using the PageMap to read DOM values.
The New HP/Mercury "TruClient" Technology -- Observations that compare and contrast two technologies.
AJAX Synchronization -- Sure ways to synchronize AJAX playbacks.

Recent and Upcoming Conferences & Technical Community Events

2011-03-10T09:52:00.000-08:00

Here are links to some technical meetings that may be of interest to WQN readers:

4th IEEE International Conference on Software Testing, Verification and Validation (ICST 2011), 21-25 March 2011, Berlin, Germany, EU
4th International Workshop on Search-Based Software Testing [Co-located with ICST 2011], Berlin, Germany, EU
Mutation 2011: 6th International Workshop on Mutation Analysis [Co-located with ICST 2011], Berlin, Germany, EU
1st International Workshop on Testing and Debugging (TeBug2011) [Co-located with ICST 2011], Berlin, Germany, EU
12th IEEE Latin American Test Workshop -- LATW 2011) 27-30 March 2011, Porto de Galinhas, Brazil.

Selected User Forum Posts

2011-03-07T09:42:00.000-08:00

Multiple Machine/User Images -- Yes, multiple machine images can all run eValid playbacks independetly.
Two Nearly Identical, Untitled Desktop Windows -- More about how to handle overlapping, simultaneous arriving desktop windows.
Annual Subscription License Price -- General approach to monthly licensing fees.
Preserves the Client-user State -- What does it mean to "maintain client state"?
How can eValid tell me when a new value arrives? -- How to sense a DOM element change.

Dynamic ID Tags: Testing OracleCRM With Geometric Tests

2011-03-04T13:38:00.000-08:00

We have had several requests from our user community for details about applying eValid to the Oracle CRM On Demand application. It seems that one of the main issues with this application is that the implementation uses a special kind of dynamically assigned ID tags, which are updated to new values each time the application is launched. This "feature" defeats eValid's built-in Adaptive Playback feature: when a recording is made it works perfectly but later, when run in batch mode, it fails.

The way to overcome this is with a special kind of geometric testing, which works on explicitly identified pivot points on the web page that don't change (as the implicit ID tags do). Our worked-example writeup, Oracle CRM On Demand -- Example #1 , shows an original recorded "from life" script with quick structural and geometric script modifications that overcome the problem of dyamically assigned ID tag values.

PerformanceTest Special Winter 2011 Discount Offer

2011-03-02T20:20:00.000-08:00

eValid's PerformanceTest fixed-price projects offer superior value for high-capacity testing of AJAX applications with 100% realistic client-side operations. As the PerformanceTest Services synopsis shows, eValid can deliver up to 10,000 equivalent browser users at the best price/BU available. Check out the details of the special offer here: Winter 2011 Special PerformanceTest Offer.

Selected User Forum Posts

2011-03-01T15:39:00.000-08:00

Recording Protocol -- How to establish a simple recording-time checklist.
What is the story on scaling eValid instances? -- Limits and groundrules about high eValid BU counts.
The ValueSave File -- How to use eValid's DOM input/output commands.
Both Web and Desktop Testing at the Same Time -- How web and windows applications interact in eValid.
Automation of Sample Script -- Details of how to diagnose a script playback problem.

eValid Confirmed Operational With IE9 RC

2011-02-28T13:47:00.000-08:00

Last year we confirmed initial operation of eValid with the first Beta releases of Microsoft's IE9.

Microsoft has made the IE9 RC (Release Candidate) version available and we have retested eValid on that version.

We're pleased to report that all eValid features work 100% normally. There is, of course, some improvement in performand that that is due to the major upgrades Microsoft as made in the IE9 engine.

We'll keep our users posted in case any glitches with eValid operating on the IE9 base show up.

Selected User Forum Posts

2011-02-15T11:46:00.000-08:00

1,000 BU Load -- Details on launching 1,000 BUs in parallel.
Bug while playing back -- Explanation of apparently curious DWindowSet command behavior.
How do I look at the entire field in a "Script" element? -- Grabbing the complete contents of an attriute.
How does eValid do on YUI-built systems? -- How well does eValid handle the YUI libraries?
eValid IndexFind vs. Xpath -- Performance comparison and operational details explained.

Geometric Testing Example

2011-02-12T13:37:00.000-08:00

Here is a fully worked example that illustrates this basic idea: Oracle CRM On Demand -- Example #1. In this example a "from life" recording of with over 100 commands is converted into a 100% geometric test by adding just 7 new commands to replace some which rely on objects for which the ID tags change continually.

Selected User Forum Posts

2011-02-10T08:14:00.000-08:00

How long does a section in the middle of my script take? -- Methods for using the built-in timer and related special commands.
Capturing Runtime Obj Property -- How to read and write specific page element properties.
What can you guys do about cross-browser testing? -- Using SetUserAgent to "spoof" non-IE browsers.
Using multiple Object Properties -- IndexFindElementEX can match on multiple object/value pairs.
DWindowSet Command -- Explanation of special properties of the DWindowSet command.

Geometric Testing Explained

2011-02-07T15:41:00.000-08:00

We had a question asked on the forum today about our use of the term "geometric testing"

There's been talk about eValid offering another kind of testing based on using the eValid automated browser called "geometric testing". Could you kindly let everyone know what that is all about?

Yes, you are correct, and we're pleased to explain.

In testing some AJAX based web applications the "recorded from life" script may only work part of the time, due to the dynamic nature of AJAX. So the procedure used to prevent playback de-synchronization is to script extra commands into the script that synchronize the playbak, even in spite of AJAX asynchronous operation. At the same time, we often alter the "from life" commands so that they use DOM-dependent searching and checking, drawing from the rich set of structural testing commands that eValid supports.

We generally call this second level of script a "structural" test because it operates based on structural properties of the page.

When the application is built in a way that the usual structural cues are no longer constant -- most commonly seen when the ID tags in a page all change values between runs -- then something more is needed.

From the structural test we then generate a third variation of it that only uses references to fixed objects on the page -- objects that we know by experimentation don't change value from run to run or when the application is updated.

This extension of a structural test to one that does not rely on specific DOM values is then called a "geometric test" because it relies only on the properties of the geometry of the page...and not on any specific contents.

The sequence of eValid scripts then is natural --> structural --> geometric, in increasing order of independence of page specific facts and properties.

Here is a fully worked example that illustrates this basic idea: Oracle CRM On Demand -- Example #1. In this example a "from life" recording of with over 100 commands is converted into a 100% geometric test by adding just 7 new commands to replace some which rely on objects for which the ID tags change continually.

Look on our website for additional examples of pure-geometric web application tests.

Selected User Forum Posts

2011-02-03T15:09:00.000-08:00

How do you think eValid fits into applications development? -- Where the fit is best.
eValid Lowers The Total Cost Of Testing -- ROI and TCO.
Do my ID tags have to be constant for eValid to work? -- Overcoming dynamic ID-tag value assignments.
eValid vs. TruClient -- External validation of eValid Technology and Approach.
Customized Error Report -- How to generate conditional error reports.

Webinar Questions and Answers: AJAX Performance/Load Testing

2011-01-31T17:15:00.000-08:00

As you know, we present webinars about eValid on a regular basis, each time trying to focus on a different aspect about eValid technology and applications.

Our webinar last week was aimed at the convergence of the need for testing AJAX applications, and the requirement to impose significant load on a server stack from multiple users. This was a very popular webinar -- one of the best attended ones that we've done -- and we plan to schedule a repeat very soon.

What made the webinar most interesting was the variety and depth of the questions attendees asked. Those questions ranged from basic details about using cloud resources to drive the servers, to details about how eValid can synchronize on arrival (or change) of DOM values.

Due to time constraints, we weren't able to answer every attendee question during the event, but we did prepare answers for every question asked, given here:

http://www.soft.com/eValid/Promotion/Webinars/AJAX.Performance.Load.Testing/questions.asked.html

We hope everyone will find our responses interesting and valuable.

Selected User Forum Posts

2011-01-21T15:31:00.000-08:00

Different Scripts -- Practical comments about using different or similar scripts for loading.
Can eValid validate the results of JavaScript executions? -- How to use special eValid structural commands to extract DOM data for validation.
Is eValid Compatible with VS2010's CUIT System? -- Technology comparison between eValid and CUIT.
Run 10 Users on One Machine for Loading -- How the EPRISEnn license works to support multiple users on one machine.
Different Scripts -- Discusses allocation of scripts into loading scenarios.
Generating Random values -- How to generate random inputs: use eV.Generate.

Webinar: AJAX Performance/Load Testing

2011-01-17T12:29:00.000-08:00

Test Complex AJAX Applications Quickly, Easily
Run 1,000's of Browser Users (BU's)
Realistic Loads Can Pinpoint Your Server-Stack Issues

Thursday, 27 January 2011
2:00 PM Eastern Time / 11:00 AM Pacific Time

QA/Testing/Tuning/Performance projects need to qualify performance of complex AJAX web applications -- within strict budget and time constraints -- to make sure their server-stack setups can meet the load.

The traditional methods of ramping up load, based on using HTTP/S simulations or "VUs", don't always work when asynchronous AJAX applications are involved. VU's don't do AJAX. You need a browser.

eValid server loading methods offer both quick-to-create, realistic, and fully synchronized AJAX functional tests. Plus you can lift those tests into performance/loading scenarios that can involve 100's or 1,000's or 10,000's of Browser Users ("BUs") per machine.

In this webinar you'll learn: how special eValid commands overcome problems with variable-time playback dependency; how to create full-reality AJAX tests quickly; how to adjust tests to be totally self-synchronizing under stressed AJAX conditions; how to incorporate tests in an eValid LoadTest scenario; how to launch 100's or 1,000's or 10,000's of Browser User (BU) instances; and, how to analyze consolidated performance summary data to identify server-stack bottlenecks.

This unique approach demonstrates how eValid becomes a genuine force multiplier in your web application performance testing efforts.

Topic Summary:

* eValid Architecture and Structure: How eValid functional and performance testing works.
* Functional Testing: How to make reliable recordings of AJAX applications.
* Making AJAX Tests "LoadTest Safe": How to augment tests for complete AJAX synchronization.
* Creating LoadTest Scenarios: How to use the LoadTest scenario editor to organize realistic LoadTest control scripts.
* Running LoadTests: How to launch single and multiple-instance runs using "cloud computing" resources.
* Finding Bottlenecks: How to read the LoadTest and other raw data to help spot server-stack issues.

You are cordially invited to attend this free Webinar.
REGISTER NOW!

Selected User Forum Posts

2011-01-11T17:39:00.000-08:00

What Do You eValid Guys Do About AJAX Synchronization? -- Options to assure reliable AJAX playback. [12Nov10]
Where Does eValid Do Its Site Analysis? -- Explantion of Site Analysis operational details. [14Nov10]
How Often Does An AJAX Application Fail To Sync? -- Practical AJAX application failure rates. [14Nov10]
Problem with IndexElement -- Diagnosing an IndexFindElement Command. [29Nov10]
HTTP Interface -- How to drive an HTTP interface directly. [30Nov10]
Bug while playing back -- Discussion of properties of DWindowSet command [01Dec10]

eValid Positioning

2011-01-06T11:51:00.000-08:00

Question Recently Posted On The eValid User Forum:
Been going around looking at all of the different kinds of test tools in the past few months and I'm just wondering if someone there could clarify where eValid is positioned, technology-wise and market-wise?

Our Response On The Forum:
That's a very broad question, but here are some of the key points that may give you the answer you're looking for.

eValid tests web applications primarly, but also tests web/windows interactions. eValid is thus intended to be a complete web application test engine.

eValid is self-contained, implemented inside an IE compatible browser -- it is in fact a "clone" of the installed IE browser (IE 5.50 and later only).

Because of the implementation as a browser eValid does NOT interfere with the JavaScript operations that support AJAX at all. This is important because eValid then can do non-JavaScript synchronization commands that solve tricky AJAX playback stability issues.

eValid playbacks scale nicely, which leads to the LoadTest solution that eValid provides: multiple, parallel, simultaneous eValid instances produce real Browser User (BU) load on your application.

eValid is sold as a complete solution, and not as a toolkit. A QA/Test group's needs are usually completely met by the eValid solution.

eValid is aimed mainly for use in small and medium enterprises, where the QA/Test and PerformanceTest teams may not necessarily have deep programming skill. eValid gets real results, quickly and efficiently, and withoutrequiring programming skill -- although having such skill may make learning eValid easier.

Finally, eValid licensing and pricing is aimed to scale in with the budget contraints of small and medium enterprises.

eValid Support Team

Selected User Forum Posts

2011-01-05T15:32:00.000-08:00

A Comparison of eValid with the New VS2010 CUIT -- Techology assessment of eValid relative to Microsoft's newest offering.
Does eValid Support XPATH? -- How eValid's IndexFindElementEx works as a faster version of "xpath".
How Does eValid Support Small and Medium Enterprises (SMEs)? -- General characteristics of eValid that work well with SMEs.
The Largest Number of eValid BUs -- An explanation of the sizing rule for eValid.
Can eValid do Monitoring from the Cloud? -- Discussion of criteria for monitoring from cloud images.
Can You Explain How the IndexFindElementEX Command Works? -- Details about the RegEx Find command.

User Forum Posts

2011-01-03T13:10:00.000-08:00

Can you put ANY script into monitoring mode? -- Some simple groundrules for incorporating a script in "monitoring mode" operation.
What is that situation about eValid and IE9? -- Confirmatino of IE9 Operation with eValid V9.
Checking Links -- Discussion of mixing functional testing and site analysis processes.
Record the Set of Times My Multi-step Playback Takes -- How to set up a test script to record internal "step times" into an external file.
Saturate My T1 Connection -- About the capacity of eValid to saturate input/output channels.
How Does eValid Know Which DOM To Use? -- Discussion of eValid access to a current page's DOM structure.

User Forum Posts

2010-12-20T16:48:00.000-08:00

Can I run an eValid test from the command line? -- Discusses how to run eValid tests from the command line on your Windows machine.
Automatic Page-Waiting Algorithm -- Discusses the operation -- and the importance -- of eValid's built-in page download synchronization logic.
Does eValid have short-cut function keys? -- Explains how to get to the pre-defined short-cut function keys for eValid.
To Measure The Time -- How to use eValid's built-in stopwatch to time events in your page.
Do I have to login/logout every time on my tests? -- Discusses the ground rules under which you will or won't have to log in afresh in an application that requires a login.
Can you sense a change in a DOM value? -- How eValid commands can sense changes in a selected DOM property value.

Holiday Greetings! Here's Our Year-End Special Gift To Everyone

2010-12-15T15:39:00.000-08:00

Please accept our very best wishes for a Happy Holiday Season and a Prosperous New Year.

As a special promotional offer, valid only until 15 January 2011, we will provide a license -- including a 1-year maintenance subscription -- to any eValid product bundle (Regression Testing, Server Loading, Site Analysis) for just $995. The only requirement is that the order be handled via credit card and be completed before close of business on 15 January 2011. Click here for the Complete Offer Details.

HAPPY HOLIDAYS!

eValid vs. TruClient

2010-12-13T13:16:00.000-08:00

Question Recently Posted On The eValid User Forum:
Hey eValid! What do you think about HP's announcement of their TruClient technology? Isn't this what you guys do? --WebTestGadfly

Our Response On The Forum:
Yes, true client based testing and performance analysis -- including server loading -- is what we have been offering with our solution for many years. We believe we were the first to put this idea out there, and we believe we have the very best everything-in-the-browser test engine possible.

The HP solution appears (we have not yet had much direct experience with it) to be based on the Selenium solution, in which JavaScript executions drive the browser. If that's the case, then the objections we have always expressed about the JavaScript approach would apply equally to HP's offering: High overhead, skewing of timing data, lack of scaling, and the requirement to be a programmer to get useful results.

On a positive note many here at eValid think that HP's announcement represents a real "sea change" in the community. Previously the position by the "big three" was that simulating the behavior of an AJAX user on an application could be adequately handled with the "http/s traffic generation" methods that have been highly developed over time.

The "sea change" is that at least one of the "big three" has explicitly admitted that to accurately simulate an AJAX user you really DO have to use a complete browser. So this is a victory -- perhaps even a genuine vindication, if you will -- for the eValid approach, which has ALWAYS been based on the use of the fully automated eValid browser.

Recent and Upcoming Conferences & Technical Community Events

2010-12-12T09:36:00.000-08:00

Here are links to some technical meetings that may be of interest to WQN readers:

Special Issue on Advances in Software Testing, International Journal of Systems Assurance Engineering and Management
ISEC 2011: 4th India Software Engineering Conference, 23-27 February 2011, Thiruvananthapuram, India.
Special Issue on Automated Specification and Verification of Web Systems, Submission Due 07 March 2011.

Community events like these deserve very careful consideration!

Selected User Forum Posts

2010-12-10T09:35:00.000-08:00

Can I run an eValid test from the command line? -- Discusses how to run eValid tests from the command line on your Windows machine.
Automatic Page-Waiting Algorithm -- Discusses the operation -- and the importance -- of eValid's built-in page download synchronization logic.
Does eValid have short-cut function keys? -- Explains how to get to the pre-defined short-cut function keys for eValid.
To Measure The Time -- How to use eValid's built-in stopwatch to time events in your page.
Do I have to login/logout every time on my tests? -- Discusses the ground rules under which you will or won't have to log in afresh in an application that requires a login.

BizDev Opportunities with eValid

2010-12-01T13:21:00.000-08:00

People often ask us, is there some way to partner with eValid? How can we support our customers with eValid better? What are the available partnering or BizDev opportunities.

We welcome inquires about Business Development Opportunities from qualified firms worldwide. The possibilities can include such areas as: PerformanceTest service partnerships, testing of AJAX applications, licensing of eValid technology, development of custom branded versions of the eValid test enabled browser, support to contractors with Commercial Licenses, etc.

If this catches your attention, we want to hear from you! Please Contact Us with your idea and we will respond quickly!

Selected User Forum Posts

2010-11-29T16:18:00.000-08:00

Encountered Script Error... -- More about script errors.
WPF Application -- Discusses how eValid operates relative to Windows Presentation Foundation (WPF) type objects.
The Timer and The DOM Synchronization Commands -- How the internal peformance time interacts with DOM synchronizations.
The Agile Method Idea -- How does eVaild fit into the "Agile Manifesto" as a testing suite.
Unique Test Number -- How to use the eVaild $RunCount reserved variable in your tests.

Webinar: AJAX Performance/Load Testing

2010-11-27T11:35:00.000-08:00

Test Complex AJAX Applications Quickly
Run 1,000's of Browser Users (BU's) Easily
Use Realistic Loads To Pinpoint Server-Stack Problems

Wednesday, 8 December 2010
2:00 PM Eastern Time / 11:00 AM Pacific Time

QA/Testing/Tuning/Performance groups need to qualify performance of complex AJAX web applications -- within strict budget and time constraints -- to make sure their server-stack setups can meet the need.

The traditional methods of ramping up load, based on using HTTP/S simulations or "VUs", don't always work when asynchronous AJAX applications are involved. VU's don't do AJAX very well.

eValid server loading methods offer both quick-to-create realistic and fully synchronized AJAX functional tests, and can lift those tests into performance/loading scenarios that can involve 100's or 1,000's or 10,000's of Browser Users ("BUs") per machine.

In this webinar you'll learn: how special eValid commands overcome problems with variable-time playback dependency; how to create a full-reality AJAX test quickly; how to adjust it to be totally self-synchronizing under stressed conditions; how to incorporate it in an eValid load test scenario; how to launch 100's or 1,000's or 10,000's of Browser User (BU) instances; and, how to analyze consolidated performance summary files to identify server-stack bottlenecks.

This unique approach demonstrates how eValid becomes a genuine force multiplier in your web application performance testing efforts.

Webinar Topic Summary:

eValid Architecture and Structure: How eValid functional and performance tests work.
Functional Testing: How to make reliable recordings of AJAX applications.
Making AJAX Tests Safe: How to augment test scripts for complete AJAX synchronization.
Creating Realistic LoadTest Scenarios: How to use the LoadTest scenario editor to create realistic LoadTest control scripts.
Running Heavy LoadTests: How to launch single and multiple-instance runs using "compute cloud" resources.
Finding Bottlenecks: How to read the LoadTest and other raw data to help spot server-stack problems.

You are cordially invited to attend this free Webinar.

REGISTER NOW!

Illustration of eValid Structural Commands

2010-11-14T11:39:00.000-08:00

From time to time users ask us to provide more examples of how to use the special eValid structural testing commands. These commands provide a way to extract, save, read, and insert values into the DOM of a page.

You would typically use these kinds of eValid commands to convert a script that you have recorded from life (using the eValid recording engine) into one that is immune to changes in the content, or even in the structure, of the page you are testing. This technique is the antidote to the "brittle test" issue -- the complaint made about the record/play method that the recorded scripts break too easily.

Almost every eValid "from-life" recorded script command has a structural equivalent that can be derived easily using the command lexicon and the details of the page that you find from the eValid PageMap command.

It's even possible to use the structural commands to manipulate key variables from within a page. The Value Setting Loop example explains one way to use these special commands to extract a session id from a page and then insert that same session id into the query-string section of a page. This kind of scripting method applies to any number of values that you want to save locally and then communicate back into your playback script. That's a powerful feature worth taking advantage of.

User Forum Posts

2010-11-13T08:38:00.000-08:00

Why does a large scan -- about 2400 pages -- take so long? -- Some gruond rules about site analysis.
Does eValid Use HTTP Traffic In Simulating User Load? -- Browser driven traffic isn't the same as basic HTTP traffic.
What does "Script Error" mean in the EventLog? -- The meaning of the Script Error message in the eValid Event Log.
How does eValid approach the issue of testing web services? -- Drive web services from an HTML testbed and drive the testbed from eValid.
Running Monitors On An AJAX Application -- Recommendations on how to provision a script for monitoring an AJAX application.
The New Microsoft Coded UI Test System -- Some comments about Microsoft's latest software development environment.
Automatically Snap A Screenshot -- A simple command sequence solves a common problem.
Problem in pop-up box recording -- A detailed explanation of how to handle recording and processing a pop-up box.

Please Contact Us if you have questions about any of these topics.

eValid Used at Sacramento State

2010-11-12T01:51:00.001-08:00

We're pleased to pass on the news that the eValid web testing system is being used as an example testing suite at California State University at Sacramento in a course taught by Prof. Ahmen Salem on Software Verification and Validation.

Prof. Salem's students are downloading and using eValid in their laboratory work for the CSc 131 and CSc 234 courses, which involve both graduate and undergraduate students, throughout the Fall 2010 semester. Their work with eValid involves both functional testing of selected AJAX applications and generating server loads with multiple eValid instances.

Synchronizing Multiple eValid Instances

2010-11-10T11:11:00.000-08:00

In many of our fixed-price PerformanceTest Service Projects a common requirement is to provide a sharp increment in the total number of BUs -- a kind of "step function" for the server stack that acts to "ring out" problems the same way an step function signal does for an electronic circuit.

eValid provides multiple techniques to do this, all of based on the use of the internal synchronization/logic commands such as SyncOnText. When (for example) 100's or 1,000's of eValid BUs are all running in parallel and they are all waiting for a particular DOM element to take on a particular value, the total "width" of the step function measures out to about 1-2 seconds, even when the BUs are located on separate machines.

The real world results we get doing this kind of step function are very interesting because a near-simultaneous burst of new activity can often overload servers, load-balancers, database back-end machine, etc. A lot of performance and diagnostic data in a very short time.

Recent and Upcoming Conferences & Technical Community Events

2010-11-02T17:17:00.000-07:00

Here are links to some technical meetings that may be of interest to WQN readers:

Special Issue on Advances in Software Testing, International Journal of Systems Assurance Engineering and Management
ISCE 2011: 4th India Software Engineering Conference, 23-27 February 2011, Thiruvananthapuram, India.
7th Workshop on Model-Based Testing 2-3 April 2011, Saarbruken, Germany, EU
8th International Conference on Information Technology New Generations: ITNG 2011 - Software Testing, 11-13 April 2011, Las Vegas, Nevada,USA

Community events like these deserve very careful consideration!

Small and Medium Enterprise (SME) Support

2010-10-22T11:43:00.000-07:00

Test Complex AJAX Applications Quickly and Efficiently
Run 1,000's of Browser Users (BU's) Easily and Efficiently
Monitor End-User Performance Accurately
Analyze Websites From The Client Perspective
Very Attractive SME License Offers

Wednesday, 27 October 2010
2:00 PM Eastern Time / 11:00 AM Pacific Time

In small and medium enterprises the world over, QA/Testing/Performance groups often face the job of qualifying the quality and performance of complex web applications -- including AJAX applications -- and are expected to do this within severe budget and time constraints.

eValid offers a comprehensive and self-contained solution to 99% of web application testing issues that is optimized for use in Small and Medium Enterprises (SMEs).

In this webinar you'll learn how eValid's features overcome common problems with variable-time playback dependency; how to create a full-reality AJAX regression test quickly; how to lift a test into a sever loading scenario; that launchs 100's or 1,000's of Browser User (BU) instances; and, how to set up a site analysis scan to validate a complete website.

Overall, you'll see how easily eValid can become a genuine force multiplier in your company's web application quality assurance, testing and performance tuning projects.

This Webinar Will Discuss:

* eValid Architecture: How eValid functional testing, monitoring, performance testing and site scanning components.
* Functional/Regression Testing: How to make regression recordings -- including complex AJAX applications.
* Imposing Server Load: How to create a LoadTest scenario and run it against your servers, with 100's or 1,000's of BUs.
* Creating a Realistic Monitor Script: How to provision a script for use as a monitoring agent for complex applications -- including AJAX.
* How to Scan a WebSite Quickly and Accurately Search criteria and limits setup to yield a very effective site scan.

You are cordially invited to attend this free Webinar.
REGISTER NOW!

eValid for Use in Coursework at Universidade Federal de Campina Grande (UFCG)

2010-10-20T12:02:00.000-07:00

We recently provided copies of eValid for use in coursework at Universidade Federal de Campina Grande (UFCG) and here are some comments by one of the students.

What course was this for at UFCG?
I'm currently taking undergraduate classes in the course of Computer Science at UFCG.
The professor name is Patricia Machado.
What was your goal for the demonstration.
The goal of the demonstration was to show to the crowd a demo of a tool. The teacher choose a bunch of tools test related, and I was asked to perform a demo of eValid.

What about eValid functionality did you show off and what were the impressions you got?
I've focused on the test functionality. Basically I've created a test on a website, then I've modified the website and ran the test again, showing that the test was resulting in a error.
Besides, I took a look on the performance analysis functionality, and I've really appreciated it.
I really enjoyed the [page] metrics.
What was the reaction of the others?
Everybody enjoyed the presentation, and the teacher was really impressed about all functionalities provided by eValid.

May we use your name and email address?
Sure you can, just let me know as soon as it happens, please.
I have nothing else to say, other than congratulates the team for the good job. I'm also very satisfied about this kind of contact you guys have provided me and I would be very glad if this kind of behavior keep coming.
Cheers!
Felipe Vieira,
UFCG Student
(felipe29vieira@gmail.com)

User Forum Posts

2010-09-30T11:02:00.000-07:00

The Comparative Level of Effort of eValid Recording... -- Discusses the relative level of effort of using eValid versus other kinds of technical approaches.
Comparing An Actual Value At Some DOM Element -- Explains how to compare an actual value in a page by reading and manipulating the current contents of the DOM.
eValid Faithfully Emulates An IE Browser -- Discusses how we made sure that eValid was as close to IE behavior as possible.
Can eValid Test An Application That Involves Two Browsers? -- Describes conditions under which two [or more] eValid instances can operate at the same time, in combinations of record AND playback.
DOM Values -- How to use eValid commands to extract, save, and insert DOM values from one test context to another.
Wht is your structural testing really good at? -- Describes when the structural testing methods are needed, and how effective they are.
About "SyncOnText" -- Describes how eValid AJAX synchronization logic works, and what the overhead is likely to be when you are waiting for a DOM feature to synchronize AJAX activity.
Query Strings -- Explains how to deal with URLs that have complex query string (client side parameters).
A Different IP Address -- Details on when eValid can "spoof" being on a different IP address.

Please Contact Us for additional details about any of these topics.

Recent and Upcoming Conferences & Technical Community Events

2010-09-15T09:21:00.000-07:00

Here are links to some technical meetings that may be of interest to WQN readers:

2nd International Workshop on the Quality of Service-Oriented Software Systems (QUASOSS'10), 4 October, 2010, Oslo, Norway
5th International Workshop on Systems Software Verification (SSV'10), 6-7 October 2010, Vancouver, BC, Canada.
8th CREST Open Workshop: Advanced Mootation* Testing 27 October 2010, University College, London, England (* "Mootation Testing" is a second order mutant of "Mutation Testing")
2nd International Workshop on Software Aging and Rejuvenation (Part of ISSRE 2010), 1 November 2010, San Jose, California, USA.
22nd IFIP International Conference on Testing Software and Systems (ICTSS), 8-10 November 2010, Natal, Brazil.
2010 Workshop on Verification, Validation and Uncertainty Analysis in High-Performance Computing (VVUHPC 2010), 14 November 2010, New Orleans, LA, USA.
The Future of Software Engineering Symposium 22-23 November 2010, ETH, Zurich, Switzerland.

Community events like these deserve very careful consideration!

High BU-Count Server Performance Testing Projects

2010-09-10T08:29:00.000-07:00

eValid users are already aware of the use of multiple eValid instances -- called Browser Users (BUs) -- to generate 100% realistic load on servers. Each BU runs as a separate Windows process, and each can handle very complex applications including AJAX and database-related applications.

To get high BU counts requires that we use two methods:

The first method involves making OS parameter changes to increase the size of the virtual memory, increase the desktop heap size, and change the performance option to optimize for background operations.
Even with these adjustments, each desktop user still runs up against a limit on the number of GDI Objects per user session. To overcome this constraint, we open additional user accounts on the driver machine.

Using EC2 cloud machines to drive very heavy BU-count loads, we usually employ 8-core High-Memory and/or High-CPU machine images. One such machine, properly configured, yields 1,000+ BUs per machine depending on the test duty-cycle and machine resource use. All of these methods come together in our PerformanceTest projects, which offer load generation up to 100,000 BUs in parallel,

User Forum Posts

2010-08-25T23:30:00.000-07:00

Here are some eValid User Form posts of interest to the greater eValid community:

Sync On The Same Document Object Property -- When two scripts synchronize on the same object property there is no interference because there are separate processes involved.
The Value Of "sourceIndex" And "elementValue" -- How to use the values of these two critical DOM-based variables inside your current script.
How Do You Do Step Functions In Load?" -- What's the best way to ramp up load when running multiple eValid instances, and how to arrange for a "step function" in that ramp schedule.
Can Two eValid's Be Running At One Time? -- eValid technology allows multiple instances to run in parallel and not interfere with each other at all.
Does eValid use HTTP Traffic in Simulating User Load? -- The way eValid generates load exploits the browser's ability to download pages, but there is an alternative utility that can drive simple non-brower HTTP traffic if you wish.

Please Contact Us for additional information about any of these topics.

The Fuss About ID Tags

2010-08-18T12:49:00.000-07:00

One of the biggest advantages eValid has is its Adaptive Playback feature. This feature compensates for non-consequential web page changes by pivoting on page element's HTML ID Tags structure. The basic idea of an ID Tag is to insert something "constant" into the page HTML, and most web page editors and production systems assure fixed IDtags and assure that they are kept unique (although uniqueness is not a requirement). The ID tags were recorded into the original page, and so long as they don't change, Adaptive Playback makes sure the test runs OK. To assure full disclosure, however, eValid always reports successful use of Adaptive Playback as a non-fatal Warning message in the test's event log.

A simple example is an HTML anchor tag -- just a regular dynamic link -- where the URL and/or visible text can change but eValid can still work correctly even when the visible text changes or when the underlying URL changes -- so long as the ID tags are left alone.

When ID Tags Change: But, if the ID tags on a page DO change there's nothing to pivot on and the Adaptive Playback feature will either fail or at best give a misleading result. The test that relies on constant ID tags, in turn, will also fail. Not the best possible outcome at all.

In fact there are many applications where ID tags change and for good reason. There are some environments where the ID tags are automatically generated with new values every time the page is edited and re-published. In THAT case, pivoting tests on ID tags clearly won't be effective. Tests that ought to be OK now show false negative results.

Structural Testing Enters The Picture: In this case, a slightly stronger mode can be used to provide protection for tests even when ID tags are non-constant. What such tests do is rely on Structural/Algorithmic Testing -- where you use as a pivot any feature of the web page that know WILL be constant over time. While it takes a bit longer to create such a "structural test" the benefit is a playback that continues to reliably indicate a valid page even when the internals change.

Recent and Upcoming Conferences & Technical Community Events

2010-08-10T12:51:00.000-07:00

Here are links to some technical meetings that may be of interest to WQN readers:

Testing: Academic and Industrial Conference -- Practice and Research Techniques (TAIC PART 2010), 3-5 September 2010, Windsor, UK.
26th IEEE International Conference on Software Maintenance (ICSM 2010) , 12-18 September 2010, Timisoara, Romania.
1st International Workshop on Evolution Support for Model-Based Development & Testing , 16 September 2010, Ilmenau University of Technology, Erfurt, Germany, EU.
7th International Workshop on Web Services and Formal Methods , 16-17 September, Stevens Institute of Technology, Hoboken, New Jersey, USA
25th IEEE/ACM International Conference on Automated Software Engineering (ASE 2010), 20-24 September 2010, Antwerp, Belgium, EU.
8th International Symposium on Automated Technology for Verification and Analysis (ATVA 2010), 21-24 September 2010, Singapore.
4th International Workshop on Testing, ANalysis, and Verificatino of Web Software (TAV-WEB 2010), 21 September 2010, Antwerp, Belgium, EU.
Model-Based Testing and Test Automation, 27 September 2010 - 2 October 2010, Leipzig, Germany.
Workshop on Trustworthy Self-Organizing Systems, TSOS 2010, 28 September 2010, Budapest, Hungary, EU.
Events like these deserve very careful consideration!

Current User Forum Posts

2010-07-20T09:35:00.000-07:00

Sync On The Same Document Object Property -- When two scripts synchronize on the same object property there is no interference because there are separate processes involved.
The Value Of "sourceIndex" And "elementValue" -- How to use the values of these two critical DOM-based variables inside your current script.
How Do You Do Step Functions In Load?" -- What's the best way to ramp up load when running multiple eValid instances, and how to arrange for a "step function" in that ramp schedule.
Can Two eValid's Be Running At One Time? -- eValid technology allows multiple instances to run in parallel and not interfere with each other at all.
Does eValid use HTTP Traffic in Simulating User Load? -- The way eValid generates load exploits the browser's ability to download pages, but there is an alternative utility that can drive simple non-brower HTTP traffic if you wish.

Structural Testing: Making Web Tests More Durable

2010-07-19T17:36:00.000-07:00

We often hear users complaining that their web pages changed and the tests that used to PASS yesterday now FAIL. Is there anything that can be done to have tests endure page changes that don't affect functionality, they ask?

The answer lies in the use of the Structural/Algorithmic Testing testing approach. Instead of basing test on properties of the page, when a particular feature is likely to move around, the test is based on (pivots on) that feature. The result (as the table outlines) is a little bit more work in exchange for a much higher level of durability -- without any loss in test quality (no increase in false-negative results).

A simple technique that we recommend, a Manual Script Creation Process, takes the mystery out of converting a "from life" recording into a highly durable validation test. Once the key page elements are identified, you modify the script using the Structural Testing Commands.

P.S. If this structural testing approach still isn't strong enough for your needs, with some additional programming effort you can exploit the eValid Programmatic Interface [EPI] to create a C++ program that runs your test at any level of durability you want.

User Forum Posts

2010-07-10T13:12:00.000-07:00

Encountered Script Error... -- What does a "script error" mean when you see this message in the eValid EventLog? What are the implications?
Automatically Snap A Screenshot -- How do you set up eValid to automatically capture a screenshot when there is a FAIL status?
How long does a section in the middle of my script take? -- Explains how to extract timing and other performance details from the middle of an eValid playback sequence using the SaveRecord command.
The Internal State Of The Browser -- A discussion about the meaning -- and the value -- of a fully state-preserving test system. Plus, some information about how eValid is architected.
The New Microsoft Coded UI Test System -- How does eValid compare with Microsoft's latest functional test offering? Key areas to study and important factors to consider.