Performance vs. load vs. stress testing

Performance testing

The goal of performance testing is not to find bugs, but to eliminate bottlenecks and establish a baseline for future regression testing. To conduct performance testing is to engage in a carefully controlled process of measurement and analysis. Ideally, the software under test is already stable enough so that this process can proceed smoothly.

A clearly defined set of expectations is essential for meaningful performance testing. If you don't know where you want to go in terms of the performance of the system, then it matters little which direction you take (remember Alice and the Cheshire Cat?). For example, for a Web application, you need to know at least two things:

• expected load in terms of concurrent users or HTTP connections
• acceptable response time

Once you know where you want to be, you can start on your way there by constantly increasing the load on the system while looking for bottlenecks. To take again the example of a Web application, these bottlenecks can exist at multiple levels, and to pinpoint them you can use a variety of tools:

• at the application level, developers can use profilers to spot inefficiencies in their code (for example poor search algorithms)
• at the database level, developers and DBAs can use database-specific profilers and query optimizers
• at the operating system level, system engineers can use utilities such as top, vmstat, iostat (on Unix-type systems) and PerfMon (on Windows) to monitor hardware resources such as CPU, memory, swap, disk I/O; specialized kernel monitoring software can also be used
• at the network level, network engineers can use packet sniffers such as tcpdump, network protocol analyzers such as ethereal, and various utilities such as netstat, MRTG, ntop, mii-tool

From a testing point of view, the activities described above all take a white-box approach, where the system is inspected and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed, and as a result, tuning is done.

However, testers also take a black-box approach in running the load tests against the system under test. For a Web application, testers will use tools that simulate concurrent users/HTTP connections and measure response times. Some lightweight open source tools I've used in the past for this purpose are ab, siege, httperf. A more heavyweight tool I haven't used yet is OpenSTA. I also haven't used The Grinder yet, but it is high on my TODO list.

When the results of the load test indicate that performance of the system does not meet its expected goals, it is time for tuning, starting with the application and the database. You want to make sure your code runs as efficiently as possible and your database is optimized on a given OS/hardware configurations. TDD practitioners will find very useful in this context a framework such as Mike Clark's jUnitPerf, which enhances existing unit test code with load test and timed test functionality. Once a particular function or method has been profiled and tuned, developers can then wrap its unit tests in jUnitPerf and ensure that it meets performance requirements of load and timing. Mike Clark calls this "continuous performance testing". I should also mention that I've done an initial port of jUnitPerf to Python -- I called it pyUnitPerf.

If, after tuning the application and the database, the system still doesn't meet its expected goals in terms of performance, a wide array of tuning procedures is available at the all the levels discussed before. Here are some examples of things you can do to enhance the performance of a Web application outside of the application code per se:

• Use Web cache mechanisms, such as the one provided by Squid
• Publish highly-requested Web pages statically, so that they don't hit the database
• Scale the Web server farm horizontally via load balancing
• Scale the database servers horizontally and split them into read/write servers and read-only servers, then load balance the read-only servers
• Scale the Web and database servers vertically, by adding more hardware resources (CPU, RAM, disks)
• Increase the available network bandwidth

Performance tuning can sometimes be more art than science, due to the sheer complexity of the systems involved in a modern Web application. Care must be taken to modify one variable at a time and redo the measurements, otherwise multiple changes can have subtle interactions that are hard to qualify and repeat.

In a standard test environment such as a test lab, it will not always be possible to replicate the production server configuration. In such cases, a staging environment is used which is a subset of the production environment. The expected performance of the system needs to be scaled down accordingly.

The cycle "run load test->measure performance->tune system" is repeated until the system under test achieves the expected levels of performance. At this point, testers have a baseline for how the system behaves under normal conditions. This baseline can then be used in regression tests to gauge how well a new version of the software performs.

Another common goal of performance testing is to establish benchmark numbers for the system under test. There are many industry-standard benchmarks such as the ones published by TPC, and many hardware/software vendors will fine-tune their systems in such ways as to obtain a high ranking in the TCP top-tens. It is common knowledge that one needs to be wary of any performance claims that do not include a detailed specification of all the hardware and software configurations that were used in that particular test.

Load testing
We have already seen load testing as part of the process of performance testing and tuning. In that context, it meant constantly increasing the load on the system via automated tools. For a Web application, the load is defined in terms of concurrent users or HTTP connections.

In the testing literature, the term "load testing" is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with. Load testing is sometimes called volume testing, or longevity/endurance testing.

Examples of volume testing:

• testing a word processor by editing a very large document
• testing a printer by sending it a very large job
• testing a mail server with thousands of users mailboxes
• a specific case of volume testing is zero-volume testing, where the system is fed empty tasks

Examples of longevity/endurance testing:

• testing a client-server application by running the client in a loop against the server over an extended period of time

Goals of load testing:

• expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
• ensure that the application meets the performance baseline established during performance testing. This is done by running regression tests against the application at a specified maximum load.

Although performance testing and load testing can seem similar, their goals are different. On one hand, performance testing uses load testing techniques and tools for measurement and benchmarking purposes and uses various load levels. On the other hand, load testing operates at a predefined load level, usually the highest load that the system can accept while still functioning properly. Note that load testing does not aim to break the system by overwhelming it, but instead tries to keep the system constantly humming like a well-oiled machine.

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. In my experience, many important bugs simply do not surface unless you deal with very large entities such thousands of users in repositories such as LDAP/NIS/Active Directory, thousands of mail server mailboxes, multi-gigabyte tables in databases, deep file/directory hierarchies on file systems, etc. Testers obviously need automated tools to generate these large data sets, but fortunately any good scripting language worth its salt will do the job.

Stress testing

Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose behind this madness is to make sure that the system fails and recovers gracefully -- this quality is known as recoverability.

Where performance testing demands a controlled environment and repeatable measurements, stress testing joyfully induces chaos and unpredictability. To take again the example of a Web application, here are some ways in which stress can be applied to the system:

• double the baseline number for concurrent users/HTTP connections
• randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example)
• take the database offline, then restart it
• rebuild a RAID array while the system is running
• run processes that consume resources (CPU, memory, disk, network) on the Web and database servers

I'm sure devious testers can enhance this list with their favorite ways of breaking systems. However, stress testing does not break the system purely for the pleasure of breaking it, but instead it allows testers to observe how the system reacts to failure. Does it save its state or does it crash suddenly? Does it just hang and freeze or does it fail gracefully? On restart, is it able to recover from the last good state? Does it print out meaningful error messages to the user, or does it merely display incomprehensible hex codes? Is the security of the system compromised because of unexpected failures? And the list goes on.

Read more »

WATERFALL vs. AGILE METHODOLOGY

There is no IT meeting that does not talk and debate endlessly about Waterfall vs. Agile development methodologies.  Feelings run strong on the subject with many considering Agile ‘so of the moment’, just so right, while Waterfall is thought to be passé!  But, before deciding which is more appropriate, it is essentially important to provide a little background on both.

Waterfall
A classically linear and sequential approach to software design and systems development, each waterfall stage is assigned to a separate team to ensure greater project and deadline control, important for on-time project delivery.  A linear approach means a stage by stage approach for product building, e.g.

1.      The project team first analyses, then determining and prioritisingbusiness requirements / needs.
2.      Next, in the design phase business requirements are translated into IT solutions, and a decision taken about which underlying technology i.e. COBOL, Java or Visual Basic, etc. etc. is to be used.
3.      Once processes are defined and online layouts built, codeimplementation takes place.
4.      The next stage of data conversion evolves into a fully tested solution for implementation and testing for evaluation by the end-user.
5.      The last and final stage involves evaluation and maintenance, with the latter       ensuring everything runs smoothly.
However, in case a glitch should result, changing the software is not only a practical impossibility, but means one has to go right back to the beginning and start developing new code, all over again.  That’s Waterfall for you! Now, as for minimal risk Agile, it is a low over-head method that emphasizes values and principles rather than processes.  Working in cycles i.e. a week, a month, etc., project priorities are re-evaluated and at the end of each cycle.  Four principles that constitute Agile methods are:

1.      The reigning supreme of individuals and interactions over processes and tools.
2.      As does, working software over comprehensive documentation.
3.      Likewise, customer collaboration over contract negotiation.
4.      And again, responding to change over plan follow-throughs.

To synopsise the difference between the two, one can say the classic waterfall method stands for predictability, while Agile methodology spells adaptability.  Agile methods are good at reducing overheads, such as, rationale, justification, documentation and meetings, keeping them as low as is possible.  And, that is why Agile methods benefit small teams with constantly changing requirements, rather more than larger projects.

Agile, based on empirical rather than defined methods (Waterfall) is all about light maneuverability and sufficiency for facilitating future development.  By defined methods what one means is that one plans first and then enforces these plans.  However, Agile methods involve planning what one wants and then adapting these plans to the results.  Extreme Programming (XP) is an excellent example of Agile methodology i.e.:

1.      Communication between customers and other team members;
2.      Simple, clean designs;
3.      Feedback given on Day 1 of software testing;
4.      Early delivery and implementation of suggested changes. 

Agile methodology means cutting down the big picture into puzzle size bits, fitting them together when the time is right e.g. design, coding and testing bits.  So, while there are reasons to support both the waterfall and agile methods, however, a closer look clarifies why many software and web design firms make the more appropriate choice of employing Agile methodology.  The following table enumerates the raison d’être for choosing Agile methodology over the Waterfall method.

1. Once a stage is completed in the Waterfall method, there is no going back, since most software designed and implemented under the waterfall method is hard to change according to time and user needs.  The problem can only be fixed by going back and designing an entirely new system, a very costly and inefficient method.Whereas, Agile methods adapt to change, as at the end of each stage, the logical programme, designed to cope and adapt to new ideas from the outset, allows changes to be made easily.  With Agile, changes can be made if necessary without getting the entire programme rewritten.  This approach not only reduces overheads, it also helps in the upgrading of programmes.
2.  Another Agile method advantage is one has a launchable product at the end of each tested stage.  This ensures bugs are caught and eliminated in the development cycle, and the product is double tested again after the first bug elimination.  This is not possible for the Waterfall method, since the product is tested only at the very end, which means any bugs found results in the entire programme having to be re-written.
3. Agile’s modular nature means employing better suited object-oriented designs and programmes, which means one always has a working model for timely release even when it does not always entirely match customer specifications.  Whereas, there is only one main release in the waterfall method and any problems or delays mean highly dissatisfied customers.
4. Agile methods allow for specification changes as per end-user’s requirements, spelling customer satisfaction.  As already mentioned, this is not possible when the waterfall method is employed, since any changes to be made means the project has to be started all over again.
5. However, both methods do allow for a sort of departmentalization e.g. in waterfall departmentalization is done at each stage.  As for Agile, each coding module can be delegated to separate groups. This allows for several parts of the project to be done at the same time, though departmentalization is more effectively used in Agile methodologies.

 In conclusion, though on the plus side, waterfall’s defined stages allow for thorough planning, especially for logical design, implementation and deployment, Agile methodology is a sound choice for software development and web design projects.  More and more firms are becoming Agile!

Read more »

QTP interview questions and answers

1. What are the Features & Benefits of Quick Test Pro (QTP 8.0)? - Operates stand-alone, or integrated into Mercury Business Process Testing and Mercury Quality Center. Introduces next-generation zero-configuration Keyword Driven testing technology in Quick Test Professional 8.0 allowing for fast test creation, easier maintenance, and more powerful data-driving capability. Identifies objects with Unique Smart Object Recognition, even if they change from build to build, enabling reliable unattended script execution. Collapses test documentation and test creation to a single step with Auto-documentation technology. Enables thorough validation of applications through a full complement of checkpoints. 
2. How to handle the exceptions using recovery scenario manager in QTP? - There are 4 trigger events during which a recovery scenario should be activated. A pop up window appears in an opened application during the test run: A property of an object changes its state or value, A step in the test does not run successfully, An open application fails during the test run, These triggers are considered as exceptions.You can instruct QTP to recover unexpected events or errors that occurred in your testing environment during test run. Recovery scenario manager provides a wizard that guides you through the defining recovery scenario. Recovery scenario has three steps: 1. Triggered Events 2. Recovery steps 3. Post Recovery Test-Run
3. What is the use of Text output value in QTP? - Output values enable to view the values that the application talks during run time. When parameterized, the values change for each iteration. Thus by creating output values, we can capture the values that the application takes for each run and output them to the data table.
4. How to use the Object spy in QTP 8.0 version? - There are two ways to Spy the objects in QTP: 1) Thru file toolbar, In the File Toolbar click on the last toolbar button (an icon showing a person with hat). 2) True Object repository Dialog, In Object repository dialog click on the button object spy. In the Object spy Dialog click on the button showing hand symbol. The pointer now changes in to a hand symbol and we have to point out the object to spy the state of the object if at all the object is not visible. or window is minimized then, hold the Ctrl button and activate the required window to and release the Ctrl button.
5. How Does Run time data (Parameterization) is handled in QTP? - You can then enter test data into the Data Table, an integrated spreadsheet with the full functionality of Excel, to manipulate data sets and create multiple test iterations, without programming, to expand test case coverage. Data can be typed in or imported from databases, spreadsheets, or text files.
6. What is keyword view and Expert view in QTP? - Quick Test’s Keyword Driven approach, test automation experts have full access to the underlying test and object properties, via an integrated scripting and debugging environment that is round-trip synchronized with the Keyword View. Advanced testers can view and edit their tests in the Expert View, which reveals the underlying industry-standard VBScript that Quick Test Professional automatically generates. Any changes made in the Expert View are automatically synchronized with the Keyword View.
7. Explain about the Test Fusion Report of QTP? - Once a tester has run a test, a Test Fusion report displays all aspects of the test run: a high-level results overview, an expandable Tree View of the test specifying exactly where application failures occurred, the test data used, application screen shots for every step that highlight any discrepancies, and detailed explanations of each checkpoint pass and failure. By combining Test Fusion reports with Quick Test Professional, you can share reports across an entire QA and development team.
8. Which environments does QTP support? - Quick Test Professional supports functional testing of all enterprise environments, including Windows, Web,..NET, Java/J2EE, SAP, Siebel, Oracle, PeopleSoft, Visual Basic, ActiveX, mainframe terminal emulators, and Web services.
9. What is QTP? - Quick Test is a graphical interface record-playback automation tool. It is able to work with any web, java or windows client application. Quick Test enables you to test standard web objects and ActiveX controls. In addition to these environments, Quick Test Professional also enables you to test Java applets and applications and multimedia objects on Applications as well as standard Windows applications, Visual Basic 6 applications and.NET framework applications
10. Explain QTP Testing process? - Quick Test testing process consists of 6 main phases:
11. Create your test plan - Prior to automating there should be a detailed description of the test including the exact steps to follow, data to be input, and all items to be verified by the test. The verification information should include both data validations and existence or state verifications of objects in the application.
12. Recording a session on your application - As you navigate through your application, Quick Test graphically displays each step you perform in the form of a collapsible icon-based test tree. A step is any user action that causes or makes a change in your site, such as clicking a link or image, or entering data in a form.
13. Enhancing your test - Inserting checkpoints into your test lets you search for a specific value of a page, object or text string, which helps you identify whether or not your application is functioning correctly. NOTE: Checkpoints can be added to a test as you record it or after the fact via the Active Screen. It is much easier and faster to add the checkpoints during the recording process. Broadening the scope of your test by replacing fixed values with parameters lets you check how your application performs the same operations with multiple sets of data. Adding logic and conditional statements to your test enables you to add sophisticated checks to your test.
14. Debugging your test - If changes were made to the script, you need to debug it to check that it operates smoothly and without interruption.
15. Running your test on a new version of your application - You run a test to check the behavior of your application. While running, Quick Test connects to your application and performs each step in your test.
16. Analyzing the test results - You examine the test results to pinpoint defects in your application.
17. Reporting defects - As you encounter failures in the application when analyzing test results, you will create defect reports in Defect Reporting Tool.
18. Explain the QTP Tool interface. - It contains the following key elements: Title bar, displaying the name of the currently open test, Menu bar, displaying menus of Quick Test commands, File toolbar, containing buttons to assist you in managing tests, Test toolbar, containing buttons used while creating and maintaining tests, Debug toolbar, containing buttons used while debugging tests. Note: The Debug toolbar is not displayed when you open Quick Test for the first time. You can display the Debug toolbar by choosing View — Toolbars — Debug. Action toolbar, containing buttons and a list of actions, enabling you to view the details of an individual action or the entire test flow. Note: The Action toolbar is not displayed when you open Quick Test for the first time. You can display the Action toolbar by choosing View — Toolbars — Action. If you insert a reusable or external action in a test, the Action toolbar is displayed automatically. Test pane, containing two tabs to view your test-the Tree View and the Expert View ,Test Details pane, containing the Active Screen. Data Table, containing two tabs, Global and Action, to assist you in parameterizing your test. Debug Viewer pane, containing three tabs to assist you in debugging your test-Watch Expressions, Variables, and Command. (The Debug Viewer pane can be opened only when a test run pauses at a breakpoint.) Status bar, displaying the status of the test
19. How does QTP recognize Objects in AUT? - Quick Test stores the definitions for application objects in a file called the Object Repository. As you record your test, Quick Test will add an entry for each item you interact with. Each Object Repository entry will be identified by a logical name (determined automatically by Quick Test), and will contain a set of properties (type, name, etc) that uniquely identify each object. Each line in the Quick Test script will contain a reference to the object that you interacted with, a call to the appropriate method (set, click, check) and any parameters for that method (such as the value for a call to the set method). The references to objects in the script will all be identified by the logical name, rather than any physical, descriptive properties.
20. What are the types of Object Repositories in QTP? - Quick Test has two types of object repositories for storing object information: shared object repositories and action object repositories. You can choose which type of object repository you want to use as the default type for new tests, and you can change the default as necessary for each new test. The object repository per-action mode is the default setting. In this mode, Quick Test automatically creates an object repository file for each action in your test so that you can create and run tests without creating, choosing, or modifying object repository files. However, if you do modify values in an action object repository, your changes do not have any effect on other actions. Therefore, if the same test object exists in more than one action and you modify an object’s property values in one action, you may need to make the same change in every action (and any test) containing the object.
21. Explain the check points in QTP? - A checkpoint verifies that expected information is displayed in an Application while the test is running. You can add eight types of checkpoints to your test for standard web objects using QTP. A page checkpoint checks the characteristics of an Application. A text checkpoint checks that a text string is displayed in the appropriate place on an Application. An object checkpoint (Standard) checks the values of an object on an Application. An image checkpoint checks the values of an image on an Application. A table checkpoint checks information within a table on a Application. An Accessibilityy checkpoint checks the web page for Section 508 compliance. An XML checkpoint checks the contents of individual XML data files or XML documents that are part of your Web application. A database checkpoint checks the contents of databases accessed by your web site
22. In how many ways we can add check points to an application using QTP? - We can add checkpoints while recording the application or we can add after recording is completed using Active screen (Note : To perform the second one The Active screen must be enabled while recording).
23. How does QTP identify objects in the application? - QTP identifies the object in the application by Logical Name and Class.
24. What is Parameterizing Tests? - When you test your application, you may want to check how it performs the same operations with multiple sets of data. For example, suppose you want to check how your application responds to ten separate sets of data. You could record ten separate tests, each with its own set of data. Alternatively, you can create a parameterized test that runs ten times: each time the test runs, it uses a different set of data.
25. What is test object model in QTP? - The test object model is a large set of object types or classes that Quick Test uses to represent the objects in your application. Each test object class has a list of properties that can uniquely identify objects of that class and a set of relevant methods that Quick Test can record for it. A test object is an object that Quick Test creates in the test or component to represent the actual object in your application. Quick Test stores information about the object that will help it identify and check the object during the run session.
26. What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
27. What is the Diff between Image check-point and Bit map Check point? - Image checkpoints enable you to check the properties of a Web image. You can check an area of a Web page or application as a bitmap. While creating a test or component, you specify the area you want to check by selecting an object. You can check an entire object or any area within an object. Quick Test captures the specified object as a bitmap, and inserts a checkpoint in the test or component. You can also choose to save only the selected area of the object with your test or component in order to save disk Space. For example, suppose you have a Web site that can display a map of a city the user specifies. The map has control keys for zooming. You can record the new map that is displayed after one click on the control key that zooms in the map. Using the bitmap checkpoint, you can check that the map zooms in correctly. You can create bitmap checkpoints for all supported testing environments (as long as the appropriate add-ins are loaded). Note: The results of bitmap checkpoints may be affected by factors such as operating system, screen resolution, and color settings.
28. How many ways we can parameterize data in QTP? - There are four types of parameters: Test, action or component parameters enable you to use values passed from your test or component, or values from other actions in your test. Data Table parameters enable you to create a data-driven test (or action) that runs several times using the data you supply. In each repetition, or iteration, Quick Test uses a different value from the Data Table. Environment variable parameters enable you to use variable values from other sources during the run session. These may be values you supply, or values that Quick Test generates for you based on conditions and options you choose. Random number parameters enable you to insert random numbers as values in your test or component. For example, to check how your application handles small and large ticket orders, you can have Quick Test generate a random number and insert it in a number of tickets edit field.
29. How do u do batch testing in WR & is it possible to do in QTP, if so explain? - Batch Testing in WR is nothing but running the whole test set by selecting Run Test set from the Execution Grid. The same is possible with QTP also. If our test cases are automated then by selecting Run Test set all the test scripts can be executed. In this process the Scripts get executed one by one by keeping all the remaining scripts in Waiting mode.
30. If I give some thousand tests to execute in 2 days what do u do? - Adhoc testing is done. It Covers the least basic functionalities to verify that the system is working fine.
31. What does it mean when a check point is in red color? what do u do? - A red color indicates failure. Here we analyze the cause for failure whether it is a Script Issue or Environment Issue or a Application issue.
32. What is Object Spy in QTP? - Using the Object Spy, you can view the properties of any object in an open application. You use the Object Spy pointer to point to an object. The Object Spy displays the selected object’s hierarchy tree and its properties and values in the Properties tab of the Object Spy dialog box.
33. What is the file extension of the code file & object repository file in QTP? - Code file extension is.vbs and object repository is.tsr
34. Explain the concept of object repository & how QTP recognizes objects? - Object Repository: displays a tree of all objects in the current component or in the current action or entire test (depending on the object repository mode you selected). We can view or modify the test object description of any test object in the repository or to add new objects to the repository. Quicktest learns the default property values and determines in which test object class it fits. If it is not enough it adds assistive properties, one by one to the description until it has compiled the unique description. If no assistive properties are available, then it adds a special Ordinal identifier such as objects location on the page or in the source code.
35. What are the properties you would use for identifying a browser & page when using descriptive programming? - Name would be another property apart from title that we can use.
36. Give me an example where you have used a COM interface in your QTP project? - com interface appears in the scenario of front end and back end. for eg:if you r using oracle as back end and front end as VB or any language then for better compatibility we will go for an interface. of which COM will be one among those interfaces. Create object creates handle to the instance of the specified object so that we program can use the methods on the specified object. It is used for implementing Automation(as defined by Microsoft).
37. Explain in brief about the QTP Automation Object Model. - Essentially all configuration and run functionality provided via the Quick Test interface is in some way represented in the Quick Test automation object model via objects, methods, and properties. Although a one-on-one comparison cannot always be made, most dialog boxes in Quick Test have a corresponding automation object, most options in dialog boxes can be set and/or retrieved using the corresponding object property, and most menu commands and other operations have corresponding automation methods. You can use the objects, methods, and properties exposed by the Quick Test automation object model, along with standard programming elements such as loops and conditional statements to design your program.

Read more »

Website Cookie Testing, Test cases for testing web application cookies?

We will first focus on what exactly cookies are and how they work. It would be easy for you to understand the test cases for testing cookies when you have clear understanding of how cookies work? How cookies stored on hard drive? And how can we edit cookie settings?

What is Cookie?
Cookie is small information stored in text file on user’s hard drive by web server. This information is later used by web browser to retrieve information from that machine. Generally cookie contains personalized user data or information that is used to communicate between different web pages.

Why Cookies are used?
Cookies are nothing but the user’s identity and used to track where the user navigated throughout the web site pages. The communication between web browser and web server is stateless.
For example if you are accessing domain http://www.example.com/1.html then web browser will simply query to example.com web server for the page 1.html. Next time if you type page as http://www.example.com/2.html then new request is send to example.com web server for sending 2.html page and web server don’t know anything about to whom the previous page 1.html served.
What if you want the previous history of this user communication with the web server? You need to maintain the user state and interaction between web browser and web server somewhere. This is where cookie comes into picture. Cookies serve the purpose of maintaining the user interactions with web server.

How cookies work?
The HTTP protocol used to exchange information files on the web is used to maintain the cookies. There are two types of HTTP protocol. Stateless HTTP and Stateful HTTP protocol. Stateless HTTP protocol does not keep any record of previously accessed web page history. While Stateful HTTP protocol do keep some history of previous web browser and web server interactions and this protocol is used by cookies to maintain the user interactions.
Whenever user visits the site or page that is using cookie, small code inside that HTML page (Generally a call to some language script to write the cookie like cookies in JAVAScript, PHP, Perl) writes a text file on users machine called cookie.
Here is one example of the code that is used to write cookie and can be placed inside any HTML page:
Set-Cookie: NAME=VALUE; expires=DATE; path=PATH; domain=DOMAIN_NAME;
When user visits the same page or domain later time this cookie is read from disk and used to identify the second visit of the same user on that domain. Expiration time is set while writing the cookie. This time is decided by the application that is going to use the cookie.
Generally two types of cookies are written on user machine.

1) Session cookies: This cookie is active till the browser that invoked the cookie is open. When we
close the browser this session cookie gets deleted. Some time session of say 20 minutes can be set to expire the cookie.
 
2) Persistent cookies: The cookies that are written permanently on user machine and lasts for months or years.

Where cookies are stored?
When any web page application writes cookie it get saved in a text file on user hard disk drive. The path where the cookies get stored depends on the browser. Different browsers store cookie in different paths. E.g. Internet explorer store cookies on path “C:\Documents and Settings\Default User\Cookies”
Here the “Default User” can be replaced by the current user you logged in as. Like “Administrator”, or user name like “Vijay” etc.
The cookie path can be easily found by navigating through the browser options. In Mozilla Firefox browser you can even see the cookies in browser options itself. Open the Mozila browser, click on Tools->Options->Privacy and then “Show cookies” button.

How cookies are stored?
Lets take example of cookie written by rediff.com on Mozilla Firefox browser:
On Mozilla Firefox browser when you open the page rediff.com or login to your rediffmail account, a cookie will get written on your Hard disk. To view this cookie simply click on “Show cookies” button mentioned on above path. Click on Rediff.com site under this cookie list. You can see different cookies written by rediff domain with different names.

Site: Rediff.com Cookie name: RMID
Name: RMID (Name of the cookie)
Content: 1d11c8ec44bf49e0… (Encrypted content)
Domain: .rediff.com
Path: / (Any path after the domain name)
Send For: Any type of connection
Expires: Thursday, December 31, 2020 11:59:59 PM

Applications where cookies can be used:

1) To implement shopping cart:
Cookies are used for maintaining online ordering system. Cookies remember what user wants to buy. What if user adds some products in their shopping cart and if due to some reason user don’t want to buy those products this time and closes the browser window? When next time same user visits the purchase page he can see all the products he added in shopping cart in his last visit.

2) Personalized sites:
When user visits certain pages they are asked which pages they don’t want to visit or display. User options are get stored in cookie and till the user is online, those pages are not shown to him.

3) User tracking:
To track number of unique visitors online at particular time.

4) Marketing:
Some companies use cookies to display advertisements on user machines. Cookies control these advertisements. When and which advertisement should be shown? What is the interest of the user? Which keywords he searches on the site? All these things can be maintained using cookies.

5) User sessions:
Cookies can track user sessions to particular domain using user ID and password.

Drawbacks of cookies:

1) Even writing Cookie is a great way to maintain user interaction, if user has set browser options to warn before writing any cookie or disabled the cookies completely then site containing cookie will be completely disabled and can not perform any operation resulting in loss of site traffic.

2) Too many Cookies:
If you are writing too many cookies on every page navigation and if user has turned on option to warn before writing cookie, this could turn away user from your site.

3) Security issues:
Some times users personal information is stored in cookies and if someone hack the cookie then hacker can get access to your personal information. Even corrupted cookies can be read by different domains and lead to security issues.

4) Sensitive information:
Some sites may write and store your sensitive information in cookies, which should not be allowed due to privacy concerns.
This should be enough to know what cookies are. If you want more cookie info see Cookie Central page.

Some Major Test cases for web application cookie testing:
The first obvious test case is to test if your application is writing cookies properly on disk. You can use the Cookie Tester application also if you don’t have any web application to test but you want to understand the cookie concept for testing.

Test cases: 

1) As a Cookie privacy policy make sure from your design documents that no personal or sensitive data is stored in the cookie.

2) If you have no option than saving sensitive data in cookie make sure data stored in cookie is stored in encrypted format.

3) Make sure that there is no overuse of cookies on your site under test. Overuse of cookies will annoy users if browser is prompting for cookies more often and this could result in loss of site traffic and eventually loss of business.

4) Disable the cookies from your browser settings: If you are using cookies on your site, your sites major functionality will not work by disabling the cookies. Then try to access the web site under test. Navigate through the site. See if appropriate messages are displayed to user like “For smooth functioning of this site make sure that cookies are enabled on your browser”. There should not be any page crash due to disabling the cookies. (Please make sure that you close all browsers, delete all previously written cookies before performing this test)

5) Accepts/Reject some cookies: The best way to check web site functionality is, not to accept all cookies. If you are writing 10 cookies in your web application then randomly accept some cookies say accept 5 and reject 5 cookies. For executing this test case you can set browser options to prompt whenever cookie is being written to disk. On this prompt window you can either accept or reject cookie. Try to access major functionality of web site. See if pages are getting crashed or data is getting corrupted.

6) Delete cookie: Allow site to write the cookies and then close all browsers and manually delete all cookies for web site under test. Access the web pages and check the behavior of the pages.

7) Corrupt the cookies: Corrupting cookie is easy. You know where cookies are stored. Manually edit the cookie in notepad and change the parameters to some vague values. Like alter the cookie content, Name of the cookie or expiry date of the cookie and see the site functionality. In some cases corrupted cookies allow to read the data inside it for any other domain. This should not happen in case of your web site cookies. Note that the cookies written by one domain say rediff.com can’t be accessed by other domain say yahoo.com unless and until the cookies are corrupted and someone trying to hack the cookie data.

8 ) Checking the deletion of cookies from your web application page: Some times cookie written by domain say rediff.com may be deleted by same domain but by different page under that domain. This is the general case if you are testing some ‘action tracking’ web portal. Action tracking or purchase tracking pixel is placed on the action web page and when any action or purchase occurs by user the cookie written on disk get deleted to avoid multiple action logging from same cookie. Check if reaching to your action or purchase page deletes the cookie properly and no more invalid actions or purchase get logged from same user.

9) Cookie Testing on Multiple browsers: This is the important case to check if your web application page is writing the cookies properly on different browsers as intended and site works properly using these cookies. You can test your web application on Major used browsers like Internet explorer (Various versions), Mozilla Firefox, Netscape, Opera etc.

10) If your web application is using cookies to maintain the logging state of any user then log in to your web application using some username and password. In many cases you can see the logged in user ID parameter directly in browser address bar. Change this parameter to different value say if previous user ID is 100 then make it 101 and press enter. The proper access message should be displayed to user and user should not be able to see other users account.

Read more »

Selenium IDE and RC Tutorial

 Selenium IDE and RC Tutorial and Workshop 

View more presentations from testinggeek

Read more »

Selenium IDE Tutorial – Part 2

In part 1 of this tutorial you have seen how to write a simple web test using Selenium. In this part we will see how to use the test generated using the IDE with Selenium RC and PHPUnit.

In this part we will use Delicious as our test target. I’ve already created a small Selenium test that you can use.

Starting up.
Restart the Selenium IDE. In the source tab replace the default HTML with the one from the above file. Set the Base URL to http://delicious.com/. After doing this your IDE should like below.



If you are on a slow internet connection than it may help to slow the test speed.



So what exactly does this test do? Simple:
1. First it opens the delicious home page
2. Then it searches by the keyword ‘selenium’
3. Reads the total number of bookmarks from the results page as shown below



Now this may not look like a test for you , but it is just an exercise in using Selenium, you will surely apply it to a good test case. Onwards…
Before running this test make sure that you are logged out of Delicious.
Now run the selenium IDE test. After the test is successfully completed this is what you should see in the log pane.



Is that all?
If this was all there was I wouldn’t be really writing this post. The interesting part comes now – running the same test from PHP. From the file menu select ‘Export Test Case As…’ and as we are using PHP, select ‘PHP – Selenium RC’. Save the file by the name ‘Example.php’.



This is what we will get in the ‘Example.php’ file.

<?php   require_once 'PHPUnit/Extensions/SeleniumTestCase.php';   class Example extends PHPUnit_Extensions_SeleniumTestCase { function setUp() { $this->setBrowser("*chrome"); $this->setBrowserUrl("http://delicious.com/"); }   function testMyTestCase() { $this->open("/"); $this->type("homepage-searchinput", "selenium"); $this->click("homepage-searchsubmit"); $this->waitForPageToLoad("30000"); $this->getText("xpath=/html/body[@id='index']/div [@id='doc3']/div[@id='bd']/div [@id='yui-main']/div[@id='content']/h3/div/em"); } } ?>

A short notice.
Before we proceed a few points to consider. Make sure that the file name and the class name are the same, here ‘Example’. Say if you change the class name to ‘Delicious’ make sure that you also change the file name to ‘Delicious.php’.
What the code basically does is it initializes the browser to ‘chrome’ (you can set the browser to your preference by changing to ‘*iexplore’ or ‘*firefox’); sets the baseurl to ‘http://delicious.com/’ and runs the test.

Whats the complex looking line!?
Everything may look fine to you except maybe the last line – ‘$this->getText…’. The function ‘getText’ returns the text found at the element specified in the expression. The complex looking line is an xpath expression to the results element:



You don’t have to rake your brains to find an xpath to an element. An easy way is to use the XPather Firefox addon to get the xpath of any element on a page. After you download and install the addon, restart Firefox, right-click on any element and from the context menu select ‘Show in XPather’. XPather retrieves the xpath to the specified element, which you can than use in your test code.
Note: This site uses ‘delicious’ as an example, so even some minor changes made by them to their site will render the xpath invalid, throwing an ‘element not found’ error in selenium. If this happens please notify me or you can find the new xpath using the Xpather plugin.

Downloading and installing Selenium RC
Selenium RC is a Java based command line server that starts browsers and runs commands you pass from your tests.
1. First make sure you have a Java runtime installed on your machine.
2. Download Selenium RC from here.
3. After extracting the files from the archive copy the ‘selenium-server.jar’ file to any directory you feel appropriate. I copied it to my PHP installations bin directory.
4. Start the Selenium RC server from the command-line by issuing the following command:
java -jar selenium-server.jar
This will start the server on port 4444.
5. Now the server is ready to accept test commands from your PHP script. Make sure you keep this server running till you finish testing.

Installing PHPUnit
1. An easy way to install PHPUnit is to use the PEAR installer. The PEAR channel (pear.phpunit.de) is used to distribute PHPUnit so make sure that it is registered with your local PEAR environment:
pear channel-discover pear.phpunit.de
After the channel is registered install PHPUnit:
pear install phpunit/PHPUnit

Actual testing
Now that PHPUnit is installed and the Selenium RC server is up and running, its time to run our test we saved before in our ‘Example.php’ file. Type the following on your command-line:
phpunit Example
This will start the test. The PHPUnit Selenium driver will execute each test command from your file and send it to the Selenium server, which does the job of launching the appropriate browser, opening web pages, and performing various specified actions; and closing the browser after the test completes.
Now the above test does basically nothing important. So we will add some simple conditional statements to the code.

. . $bookmarks = $this->getText("xpath=/html/body[@id='index']/div [@id='doc3']/div[@id='bd']/div [@id='yui-main']/div[@id='content']/h3/div/em");   echo ($bookmarks > 3000)? "On the top" : "Still not there"; } }   ?>

This is what PHPUnit has to say about the above test.


Or we can use an assertion as below, deliberately rigged to fail the test:

. . $bookmarks = $this->getText("xpath=/html/body[@id='index']/div [@id='doc3']/div[@id='bd']/div [@id='yui-main']/div[@id='content']/h3/div/em");   $this->assertGreaterThan(33000, $bookmarks);           } }   ?>

Now PHPUnit will shout failure:




This concludes the second part of the tutorial.

Read more »

Selenium IDE Tutorial – Part 1

Selenium IDE is an integrated development environment for performing Selenium tests.  Selenium tests can be written as HTML tables or coded in various languages like C#, PHP, Perl, Python and can be run directly in most modern browsers.The IDE can help you to record, edit and debug tests. Currently the IDE is only available for Firefox (as if we developers will use any other) as a addon.

Here is a possible scenario for using Selenium. Imagine you have created a HTML form with about twenty fields and you have to repeatedly test the form. Filling the form every time can quickly become tedious. With Selenium you can automate the whole process and run the test as required. In this part we will see how to create a simple test in Selenium. So let’s get started.

STEP 1 – installation :

You can download Selenium IDE from this locations.
http://selenium-ide.openqa.org/download.jsp
https://addons.mozilla.org/en-US/firefox/addon/2079
Once the addon is installed make sure you do not forget to restart Firefox.

STEP 2 – running a simple test :

a. Start Selenium IDE  in Firefox: Tools->Selenium IDE. You will see the following popup.

b. Click on the red record button on the right.
c. Browse to Google.com and enter ‘selenium’ in the search box and click enter.
d. Click on the first result, which is that of selenium.openqa.org.
e. Stop the recording by clicking on the record button.

You should see something like below. If you click on the ‘ Source’ tab you can see the test html generated by selenium.

The ‘table’ tab shows the commands recorded by Selenium.

f. Open a new tab in Firefox and click on the Selenium IDE’s play button to run the recorded test.

The IDE should play your recorded test. After the test is complete you should have landed on the selenium page (http://selenium.openqa.org/). The IDE after the test run is shown below. In the ‘Log section’ you can see the various events run by the test. In the table tab you can see that all the rows are green, which means that the test ran successfully.

Now lets add a small assertion to the above test.

a. Click on the blank line below after the last ‘clickAndWait’ command and insert ‘assertTextPresent’ command from the drop down box as shown below. You should see something like this.

This test checks to see if the text ‘Selenium News’ is present in the last rendered page. Run the test again and you should see that the test has run successfully. No try replacing the text with something that is not present on the page, take for example ‘elvis’ and run the test again. Now the test fails and you should get the screen below.

You have just completed your first test run. And it was so simple.

‘assertTextPresent’ is one of the hundreds of commands available for your testing needs. Just browse throught the command drop down to get a feel of what you can you get.

In the next part we will see more advanced features of Selenium.

Read more »

Selenium: Overview for beginners

1. Selenium Introduction 2. Selenium Installation
3. Getting Started/Selenium RC steps
4. Selenium Tips and Quirks/FAQ
5. Extending Selenium
6. Selenium 2(pros/cons)
7. Load testing with selenium
8. Good Resources
9. Alternatives to Selenium

Selenium Introduction/Overview
What does selenium do ?
You can use opensource(ie free!) - Selenium tool ( selenium IDE is a plugin to firefox) to record and playback tests (like WinRunner, QTP). You can then export the recorded test in most language e.g. html, Java , .net , perl , ruby etc. The exported test can be run in any browser and any platform using "selenium remote control".


Selenium Flash Demo(Highly recommended for beginners) :
http://wiki.openqa.org/download/attachments/400/Selenium+IDE.swf?version=1
What are the components of selenium ?

Selenium IDE - Plugin to Firefox to record and play test in firefox
and also export tests in different languages. The most appealing
format is the html test case which seems to based on fit html .
Selenium RC- Allows playing of exported test in different platform/OS
Selenium Grid - Allows to control lots of selenium machines(you typically dont need this only for load test -and hence may ignore it)..


Other Project Management/wrapper tools around selenium:

Bromine(wrapper for selenium tests):
http://bromine.openqa.org/
NOTE: The above tool seemed hard to install - besides looking primitive when I last checked - and did not support "HTML test case"
How does Selenium remote control internally works ?
Selenium RC launches the browser with itself as the proxy server (hence you may get certificate warning in some modes) and then it injects javascript - to play the test. This also means it can easily work in ALL browsers/platform - and it can be easily used to test AJAX(unlike professional tools).

What are the different modes that Selenium RC uses ?
With latest release of Selenium RC(1.0 beta 2) there
are two modes for IE *iexplore(same as *iehta) and *iexploreproxy.
Similarly for Firefox use *firefox(same as *chrome) and *firefoxproxy.
You should prefere *iexplore(ie *iehta) and *firefox (ie *chrome)
respectively as they work with cross domain.

Who should use it ?

Developers can use it - for “browser” regression testing ( and replace htmlunit/httpunit in some cases) .
Per the one of the forces behind selenium(Neal ford) - it should really be used by Business Analyst first .
QA should enhance/use it do regression test/cross browsers testing on all platforms .

Selenium IDE
In below screen shot - you can see that you need to do recording in main firefox window.
You can use selenium IDE to type any additional command that you may have missed(e.g. timeout)
In selenium IDE - you can open Selenium IDE Options - to change default timeout or
to add you own user extension to extend selenium.


Selenium Installation

Pre Requisite :
Install Firefox
Install Java jdk: Download jdk (not JRE) without Netbeans IDE or JEE. I am using jdk1.5(but it should work with jdk1.6)
Install (optional) Junit for java rc client(ONLY if you are using java client, you also need to add selenium java client driver in classpath)
Main TOOLS Installation:
Install Selenium IDE
After installation - you should see selenium ide option in firefox.
Install Selenium RC
After installation - go to directory containing file 'selenium-server.jar'-
you will need to add appropriate bat/sh script as mentioned in
below steps.

Getting Started with Selenium/ Selenium RC steps


Step a) Record your test case using Selenium IDE as shown in earlier flash demo.
For intial recording- avoid using "cross domain" testing (eg www.abc.com, www.def.com) and
preferably use http site only.(till you figure out the workarounds)
eg You can use my test case that opens this tutorial(it searches for selenium tutorial
on google and opens this tutorial):

<html>

<!--NOTE -  Sometimes you may have wrong or blank base url - as seen on top of IDE-->

<link rel="selenium.base" href="http://www.google.com/" />

<title>New Test</title>

</head>

<body>

<table cellpadding="1" cellspacing="1" border="1">

<thead>

<tr><td rowspan="1" colspan="3">New Test</td></tr>

</thead><tbody>

<!-- Open the base URL like google.com-->

<tr>

    <td>open</td>

    <td>/</td>

    <td></td>

</tr>

<!-- type selenium tutorial in text field which has id/name of q -->

<tr>

    <td>type</td>

    <td>q</td>

    <td>selenium tutorial</td>

</tr>

<!-- click on submit button which has html id/name of btnG and wait for results -->

<tr>

    <td>clickAndWait</td>

    <td>btnG</td>

    <td></td>

</tr>




<tr>
<!-- click on google results with text/link/href of below patern -->
<!-- Note selenim ide noted this as click. you have to change it manually to clickAndWait-->



<td>clickAndWait</td>
<td>link=Selenium Tutorial for Beginner/Tips for Experts</td>
<td></td>
</tr>
</tbody></table>
</body>
</html>


Step b) Play your test case using selenium IDE itself to make sure
that your test case works.. Modify the recorded code if necessary(eg replace click by clickAndWait
or change timeout) as explained in Quirks section.

Step c) Export your recording test case - as html file
and/or optionally in programming languages like java, perl etc

Do I Export in HTML or do I export in programming language like java client ?
If you need dynamic capability (e.g. reading data from csv file or database) then
you have to export it in programming language like java.
You would also need to export in java - if your page has lots of flash
components. e.g.
http://www.adobe.com/devnet/flash/articles/flash_selenium_04.html
There is also attempt to make silverlight work with java and .net out here.
The client code using java also is more stable (less permission denied errors) then
the html case.
But the html is easy to modify/maintain and it generates nice results - whereas in
java client code it tends to be cryptic(but again html test case - does not seem to be flexible).
I personally prefer the html option for the long term(if i don't need dynamic capability)

Step d) Create a test suite
After you saved your html test case (eg save as a.html) - you need to create
a "Test Suite" (eg suite.html) - as you can NOT play test case directly.
Test suite is just a collection of test case which are defined/ordered using html table tags .
If you are adding another test case in the same test suite, then - initially - try to keep it to the
same - base URL(ie starting point)
- eg below

Suite.html
<table >
<tr><td>Overall Tests </td></tr>
<tr><td><a href="./a.html">TEST CASE A</a></td></tr>
</table>

Note:
1) The first row (e.g Overall Tests) in your suite is always considered as Header
2) Selnium RC does not support a TestSuite where you go across domains
(ie the only domain that you would be allowed is for e.g. www.abc.com)
You may be able to work around it with *chrome for firefox and *iehta for IE.

Continue Next step?
If you are happy in running you test suite manually in Firefox only with Selenium
IDE - then you can skip below steps.(Its also the most stable - with no cross domain/security
warnings)
But If you wish to run you test suite in IE or if you want to schedule your
test suite bat file automatically via inbuilt Microsoft Scheduler on XP
or cron on Unix - then follow below steps.

Step e) Run your exported html test case in IE(or firefox) using selenium rc
run it in command line/bat file with below command :
java -jar selenium-server.jar -multiwindow -htmlSuite "*iexplore" "https://www.abc.com" "C:\suite.html" "C:\results.html"
REM Use *chrome for firefox and try *iexploreproxy
You can then see the nice results it generated and you can write a simple script to
email the results to yourself in case of failure(The result html file will contained "failed"
keyword).
You may also schedule your test suite with default/inbuilt Microsoft Scheduler on XP.

NOTE:
1) You may use -port option to change ports in above.
2) You may have problems in using *iehta (necessary for removing security warning
and going across domains) with above options..
3)Warning:Its better to use *chrome mode - which is more stable. Else- You are likely
to get weird permission denied error or some script error with this option - or
maybe it will ask you to download your -.do/.faces page.
(But note - I don't seem to get similar problem with java client code ie
using the code that was exported to java)
In short firefox is more likely to be stable as compared with IE for selenium.
In rare cases - You may get some lock errors with firefox though.(in
which case you may try to create seperate profile eg http://seleniumhq.org/docs/05_selenium_rc.html#specifying-the-firefox-profile
or in worst case - reboot machine/reinstall it )

Continue Next step ?
If you want dynamic capability (eg reading from CSV file or reading from database) - then
you may want to export your test case in programming language like java etc.
else skip next section.

Step f) OPTIONAL - Run your exported test case in programming like java .
Once selenium rc is extracted/installed then go to the directory containing selenium-server.jar.
If you want to use selenium-rc using programming language like java/.net/php etc(most cases)
You may want to add a bat/sh script to keep selenium rc always running..
REM set PATH variable to contain location of firefox/IE binary
REM set PATH=%PATH%;C:\firefox/bin
java -jar selenium-server.jar -multiwindow
REM -interactive is more user friendly - but does not work with websites using non frameable pages
REM - optionally use -port , if you want to run more then 1 instance
Now compile your selenium java client code with junit jar files , use selenium
java client driver, point to above selenium RC server and run it.
Assuming you exported the test case with name "NewTest.java" - you
can try to use below bat file(after getting the junit*jar and selelenium-java-client.jar - and creating clases directory in same folder)


--------- eg. put below in compileJava.bat ----------------------------

REM Get  junit*.jar (got from junit.org)

REM  Get selenium-java-client-driver.jar 

REM (got from selenum rc installation/selenium-java-client-driver* folder)

REM  NOTE: classes directory should already exist in current directory


set CLIENT_JAR=.

set CLASSPATH=%CLASSPATH%;.;.\classes;;%CLIENT_JAR%\junit-4.4.jar;%CLIENT_JAR%\selenium-java-client-driver.jar
REM COMPILE(note default package at first line is com.example.tests)

javac -d classes NewTest.java
REM ensure that selenim rc server is already running on current machine

REM Now running the actual test.
java junit.textui.TestRunner com.example.tests.NewTest
REM - remove pause if you are going to schedule it.

pause;

-------------------------------------------------------------------
This implies that java(or programming) code is dependent on
selenium rc server to be running on same(or other) machine. But the "html test case" - launches the "selenium rc server" every time.(hence you need different port - if you run html test cases in parallel)

TIP: By default selenium java client does not give nice reports(seen with html option) You may use loggingselenium" project to get the nice reports(along with timing). Also if you use this, it will also embed the screenshot that you called- in the html file. Best way is to run/edit- after right click/saving this attached code as the usage on their website is incomplete/confusing. It depends on loggingselenium and apache commons jar.

Step g)(OPTIONAL) - Migrating your test case to Selenium 2
Since Selenium IDE does not support Selenium 2(also note that Selenium 2 does not support HTML test case).
You may use WebDriverBackedSelenium steps to migrate to Selenium 2 .

package com.example.tests;

import com.thoughtworks.selenium.*;

import org.junit.After;

import org.junit.Before;

import org.junit.Test;

import java.util.regex.Pattern;

//*NEW _ ADDED THIS IMPORT

import org.openqa.selenium.*;

import org.openqa.selenium.ie.*;

import org.openqa.selenium.firefox.*;

import org.openqa.selenium.chrome.*;

public class Selenium2Test extends SeleneseTestCase {

 @Before

 public void setUp() throws Exception {

//*NEW - Commented below line and added the remaining 2 lines.  

//selenium = new DefaultSelenium

//("localhost", 4444, "*chrome", "http://www.bing.com/");

//  WebDriver driver = new InternetExplorerDriver();


  WebDriver driver = new FirefoxDriver();

  selenium = new WebDriverBackedSelenium

                         (driver, "http://www.bing.com");

//comenting below implies taht the default selenium rc server at port 4444 is running

//else if you uncommment it - then it will automatically launch selenium rc server.

//  selenium.start();

 }
@Test

 public void testUntitled2() throws Exception {

  selenium.open("/");

  selenium.type("sb_form_q", "selenium tutorial");

  selenium.click("sb_form_go");

  selenium.waitForPageToLoad("80000");

  selenium.click("link=www.jroller.com");

  selenium.waitForPageToLoad("80000");

 }

 @After

 public void tearDown() throws Exception {

  selenium.stop();

 }

}

Warning: all cases may not be supported and even though selenium 2 is faster due to native support in browser - certain methods in WebDriverBackedSelenium
will take more time.
Hence it may be better to write clean Selenium 2 tests as shown at : http://seleniumhq.org/docs/03_webdriver.html .
PS: Without IDE (and html test case)- its best to avoid Selenium 2. You may for now prefer stable libraries like HtmlUnit(java) for regression testing(assuming you dont need browser testing) You may download all the above code/scripts here.
 
Selenium Tips and Tricks/FAQ

• Do check the reference manual here to get concepts of selenium locators
  and the various commands that are available. You could get condensed version of the manual here.
  
  Without reading the manual - You may easily miss useful commands like pause(to test session timeout), storeHtmlSource (you can potentially apply selenium regular expression to verify the values of hidden variables insides that html) , typeKeys (to do things like page up etc)
• Assert vs verify: If you use seleniums verify command - then the test suite/case - will still run - and fail at the end. Hence in most cases you should use assert command which will cause the test to stop.
• Sometimes selenium IDE may use "click" instead of "clickAndWait" for some
  "submit" button cases resulting in your next page's assert/verify to fail. Hence
  you should change it manually or add waitForPageToLoad with some timeout.
• If the recording is not working - then go to firefox and click on "Tools->Add on -> Selenium IDE " .First disable it and then enable it and restart the browser. Sometimes clicking on Selenium IDEs
  Options->HTML Format helps.
• If web site does not allow itself to be frameable then use -multiwidow option(instead of -interactive) with selenium rc.
• To eliminate certificate warning(and work cross domains ) use chrome/iehta modes -
  which is now default(as compared with rc beta1 version)
  http://selenium-rc.openqa.org/experimental.html
• 
  If you check selenium rc options eg "java -jar selenium-server.jar -h"
  then you will find that it has a "timeout" option. But that timeout is for the overall
  run(and not for individual "clickAndWait" ) . Hence you may be advised to use "setTimeOut" via your
  java program or in case of HTML use -Selenium IDE open test case-> Add Insert new command
  -> setTimeout with value(not Target) of say 240000 (ms)
• Selneium has this cool - captureEntirePageScreenShot command which works
  well with IDE/HTML . Consider using it at critical points(to debug
  issues, if you schedule your test cases).
  But with programming language like java - it may not work(try
  using the experimental - captureScreenshot .
• Consider writing a wrapper for your TestCase/TestSuite in .net/jsp
  so that you can maintain your testcase/test suite in HTML.
• Sometimes the browsers or the java process may not get killed after the test is run.. So you may need to manually
  kill it(you can try to use some automated tools like Pskill on Windows ).
• When your client java test (using RC) finishes with error - then it does not give clear error -
  you just the line number of the failed assert in stack trace - Hence you may need to trap all exceptions and - print the last body test
• If you are testing with page where the id dynamically changes - then You may need to use lots of xpath in your test . In that case install firebug and use "Inspect element" by right clicking on the element -> Copy XPath by right click on bottom-> Add "/" in front of value you got in your clipboard-> paste this as a "target" in Selenium IDE for your "clickAndWait" command. You can check the validility by - click on find button in Selenium IDE to ensure that your XPath is correct(assuming that page is open) . You may find more tip on xpath using firebug/xpather here
• If you are using Selenium HTML test case - then one way to remove hard coded values
  in your selenium test is to have simple JSP(or asp/php) that converts CSV file to HTML file.
  Then you just need to invoke that JSP page (for particular CSV) - and call various "store" values
  and replace hard coded values in your test with "get" values .
  That way you can just change your test/data by just replacing the CSV file.
• Do use (minor) javascript in HTML test case. You would need javascript to access stored values anyway.
  eg store 1 abc (store on one page)
  , type id javascript{storedVars['abc'].toUpperCase()} (extract or evaluate it on another page) However if ther are too much javascripts - consider use programming languate.
  
• The latest selenium rc(1.0 beta 2) does support Firefox 3..
  With firefox 3 you can easily add your local https server with invalid certificate in safe list(to remove warning message).
  Its difficult to do that with Firefox 2.
• If you are running selenium rc test with IE a site which already has invalid certificate -
  then IE will give a warning for "invalid certificate". You can have selenium
  automatically click on "Continue" link in it(which has ID of overridelink) by adding
  below command in your IE specific test case.
  <tr>
  <td>clickAndWait</td> <td>//a[@id='overridelink'] </td>
  <td></td>
  </tr>
  With firefox 3 - you can permanently store the exception. However selenium -rc starts
  new firefox profile every time(probably to get around lock issues) - hence you may
  need to start firefox with same/particular profile.
  
  
  You may also consider changing IE options to be strict (eg Tool->Internet Options->Advanced->Display
  Notification on every script error) to have selenium catch any javascript issues/bugs in your website.
  
• Consider use of UI-Element - which is now supported in RC here
  
  
• There are other good tips here. 9-important-tips-for-selenium-remote-control-java-client-test-tool
  
  Selenium core faq(eg permission denied) - Must read
  Selenium rc Faq - Must read
  
• if you read above FAQ - you will realize selenium has limitation in IE with certain dialog box(eg upload file, basic auth). You can get around some of the limitation with autoit script
  from this site.
  Its advisable not to directly download the exe from above site - but rather install autoit, compile autoit script - from one of it program menus options - to Exe and than use the compiled exe.
  For whatever reasons (firefox does not expose its model)- the above scripts(or autoit) does not help in Firefox dialog boxes(but you dont need it anyway as in chrome mode - selenium automatically gets rids of dialog boxes)
  However if you are trying to get rid of basic authentication dialog - you can directly pass user/pass in URL.(however for IE you would need to add some registery settings) PS: Autoit and autohotkey seem to have same codebase - and both have primitive recorder.
  (else we probably wont have needed selenium)
  But in autoit vs autohotkey - I will give slight edge to autoit.
  NOTE: On windows - you should use 'start' in front of exe/.au3 file to make it run in background -
  and then run your selenium rc test in the same batch file(eg start autotItscript.au3)
  

Extending Selenium

There is nice article which links to - google download code - to extend selenium to
support loop(while) and goto..
http://51elliot.blogspot.com/2008/02/selenium-ide-goto.html
You should though export your test case in programming language - if you need
this kind of facility.(or see if you can wrap the html in JSP/asp .net)
You can get a list of all GOOD extension(eg UI-Element) at:
http://wiki.openqa.org/display/SIDE/Contributed+Extensions+and+Formats
 
Load testing with Selenium

Selenium is NOT designed for load test. However - some companies
can try to use Selenium Grid for low load testing or even give
it to some external companies - who will test your selenium test - with lots
of browsers.
There is also interesting attempt by "PushtoTest" to convert selenium
test to low level - "htmlunit" test - and hence make it suitable
for load testing. I don't think its stable yet - but please let me know if i am wrong.
eg
http://www.pushtotest.com/products/selenium-how-to/
 
Selenium 2

You may have noted that selenium requires lots of workarounds - as it uses javascript (which are restricted by browsers) to drive the tests.
However in future selenium(2.0) - will merge with webdriver - allowing selenium tests to
easily run even without browser(and hence suitable for load test). Selenium 2 uses
native plugin to browser to run its tests - and hence does not faces the javascript limitations.

Disadvantage of Selenium 2:
It may not be easy for selenium 2 to support newer browser (eg androids) or newer IE versions- as they are not going
to provide native browser plugin for all browser(or even make it consistent).
As of now (may 2010)- I dont see any support for IDE(i expect this to happen eventually) nor is
there any talk of supporting HTML Test case(not sure if they would ever support this). (in fact I was hoping that
would support JSP natively - which should be easy for selenium rc)
Selenium 2 is documented at :
http://groups.google.com/group/webdriver/browse_thread/thread/b089f9a193e4eec9?pli=1
 
Good Resources/External Links

Selenium Main site
Using selenium with command line for IE or using Ant
Effective Selenium presentation zip
Advanced Selenium presentation pdf
Selenium with Maven
If you are using "exported" java client code with selenium RC,
then the API for two main class are:
a)
http://release.openqa.org/selenium-remote-control/0.9.2/doc/java/com/thoughtworks/selenium/SeleneseTestCase.html
(There is facility for screen shot - but it appears to be protected )

b)

http://release.openqa.org/selenium-remote-control/0.9.0/doc/java/com/thoughtworks/selenium/DefaultSelenium.html

Alternatives to selenium


Selenium vs Low/Network level tools like Webtest/HttpUnit/HtmlUnit(java)

If you are using a "low/network" level tool like htmlunit then it
has the advantage of being fast - but its javascript will
not work work in most complex websites(besides not being a browser test)
Selenium(java, .net) code also has the disadvantage of being dependent on RC server(not big deal). You should continue low level tools like httpunit etc to use
it for monitoring - but don't use those tools for browser/regression testing.

Selenium vs Watir
If you like your test cases in Ruby (or maybe .net) and are willing to
dig through "hard to understand" documentation - then Watir is a tool
for you.(if you are perl/php/java shop - don't even bother considering Watir
for now)
Watir does have advantage that its recorder uses IE to record.
With Selenium - if your site does not work with Firefox - you will
have to manually write the code(not big deal)..
But the biggest advantage of Selenium is its support for FIT HTML
and the vast languages/browsers/platform that it supports.
(with good documentation)
But maybe things may change in future.

Selenium vs Imacros
Imacros appeared better in recording. But overall its not as powerful for automation(or scheduling).

Selenium vs Dejaclick
Dejaclick is probably the best (free) macro tool for websites record and play. However Selenium beats DejaClick in overall features(multiple language suport vs only XML support, free automation vs paid automation).
Dejaclick Advantage:
It nicely plays back the recorded stuff - by highlighting appropriate items in green.
It can easily record flash/silverlight - by recording position.(however it may mean that it may not
nicely work in all computers)
It has facility to encrypt sensitive data in your file.
Dejaclick easily allow you to bookmark your script. So that you can play it back in one click.
You can easily toggle Dejaclick to True Screen mode(ie recording by mouse positions) by pressing Alt + \
(some stuff are hard to record by name/ids)
Disadvantage:
Dejaclick recording tool is free - But for automation you have to rely on paid site alertsite (AlertSite however provides great facility for running your dejaclick script and even soap xml from most of cities/region in US/world. This will allow you to monitor your CDN performance/reliability or what content your website is serving across different regions)

Read more »