Automating Functional Tests Using Selenium Antawan Holmes and Marc Kellogg Digital Focus [email protected], [email protected] Abstract Ever in search of a silver bullet for automated functional testing for Web Applications, many folks have turned to Selenium. Selenium is an open-source project for in-browser testing, originally developed by ThoughtWorks and now boasting an active community of developers and users. One of Selenium’s stated goals is to become the de facto open-source replacement for proprietary tools such as WinRunner. Of particular interest to the agile community is that it offers the possibility of test-first design of web applications, red-green signals for customer acceptance tests, and an automated regression test bed for the web tier. This experience report describes the standard environment for testing with Selenium, as well as modifications we performed to incorporate our script pages into a wiki. It includes lessons we learned about continuous integration, script writing, and using the Selenium Recorder (renamed IDE). We also discuss how long it took to write and maintain the scripts in the iterative development environment, how close we came to covering all of the functional requirements with tests, how often the tests should be (and were) run, and whether additional automated functional testing below the GUI layer was still necessary and/or appropriate. While no silver bullet, Selenium has become a valuable addition to our agile testing toolkit, and is used on the majority of our web application projects. It promises to become even more valuable as it gains widespread adoption and continues to be actively developed. 1. Introduction Automating functional tests is one of the traditional problems facing software development projects. Whereas automated unit testing has achieved deep traction, in agile and non-agile projects alike, functional testing frequently is still done manually. In recent years, tools such as Fit and FitNesse [1] have helped make it easier to automate functional testing of software applications. However, web applications have long remained difficult to test because of multitiered architecture, multiple browsers, and web technologies such as JavaScript. Some teams have chosen expensive solutions such as Mercury Interactive’s WinRunner or Rational Robot (with mixed results). Recently the open-source tool Selenium, originally developed by ThoughtWorks, has gained attention as a possible silver bullet for the problems of automated testing for Web Applications. We have been using Selenium on five different web application projects for about a year. Our teams consisted of 4 to 12 developers and 1 to 2 business analysts, with the latter role responsible for writing our Selenium tests (along with other responsibilities). One of the projects used a .NET architecture, while the four others were J2EE applications. 1.1. Background Since our teams practice an agile methodology, we are committed to writing acceptance tests before beginning development on stories. Passing these tests signals completion of a story, so we write functional tests that capture these acceptance criteria. For a web application, a functional test could be simply that a user manually navigates through the application to verify the application behaves as expected. But since automating a test is the best way to make sure it is run often, we try to automate our functional acceptance tests whenever we can. And our holy grail is to be able to write the tests before development, so that the development team can have a runnable verification that the story is complete. Our team came to Selenium after using several other tools to automate web testing. We tried the opensource tools Canoo WebTest and HttpUnit, but found them to be insufficient, as they could not handle most instances of in-page JavaScript. The JavaScript problem is solved by the proprietary tool QuickTest Professional (from Mercury Interactive), which offers Proceedings of AGILE 2006 Conference (AGILE'06) 0-7695-2562-8/06 $20.00 © 2006record-and-play of test scripts and also runs tests directly in a browser. However, we found that most of our recorded scripts in QTPro would break after small page changes, so a significant amount of script maintenance was necessary. In addition, it was not obvious that we would be able to write QTPro scripts before development. As a result, we turned to Selenium – it showed the promise of easy-to-write scripts that were relatively easy to maintain and which could be written before the code. 1.2. Selenium Selenium is a web testing tool which uses simple scripts to run tests directly within a browser. It uses JavaScript and iframes to embed the test automation engine into the browser. [2] This allows the same test scripts to be used to test multiple browsers on multiple platforms. Figure 1: Selenium inside Internet Explorer Selenium gives the user a standard set of commands such as open (a URL), click (on an element), or type (into an input box); it also provides a set of verification commands to allow the user to specify expected values or behavior. The tests are written as HTML tables and run directly in the browser, with passing tests turning green and failing tests turning red as the user watches the tests run. Because Selenium is JavaScript-based and runs directly in the browser (the user can see the test running), it overcomes some of the problems encountered by users of HttpUnit or Canoo WebTest, particularly problems related to testing JavaScript functionality. Two additional useful tools are also available for use with Selenium: 1. the Selenium IDE (originally called the Recorder), which allows users to navigate their applications in Firefox and record their actions, forming tests; and, 2. a “Remote Control” server which allows users to write tests directly within the programming language of their choice -- thus enabling conditional logic within tests, try/catch blocks, and other powerful functionality available only in programming languages. 1.3. Getting started Setting up Selenium is easy, although there is a catch. The basic installation of Selenium must be hosted by the same web server as the Application Under Test (AUT). This restriction is due to the fact that JavaScript has built-in security against cross-site scripting. [3] After installation, the user simply needs to begin writing and running tests. 2. Writing good tests Selenium tests are not difficult to write. Because Selenium allows the identification of elements using the browser’s DOM object, the test can be written using specific identifiers of the necessary element, such as name, id, or xpath: Table 1: Input data for a Selenium Test type name=theField Text to submit clickAndWait id=SubmitButton assertText xpath=//h1/span Success! This test might be written for an HTML page as simple as:

Success!