TestingWithSelenium

Testing Foswiki with Selenium

Selenium "... is a suite of tools to automate web app testing across many platforms."

UnitTestContrib integrates with Selenium via the FoswikiSeleniumTestCase class - tests that derive from this class may exercise Foswiki via a browser.

As of April 2010, this is still highly experimental.

Issues

Extra variables affect test results

The browser-in-the-loop tests have so many more variables compared to regular tests, so they may be rather brittle compared to the regular tests. When the selenium-based tests fail, these are some of the additional things to check:

• Is the selenium RC server running?
• Can the machine running the unit tests access the machine where selenium RC is running, taking into account firewalls etc, and is the selenium RC server port open?
• Does the selenium RC server support the installed version of the browser?
• Is the webserver configured to allow the access to the machine running the selenium RC server?
• Is the browser profile used by selenium RC compatible with the webserver e.g. for authentication methods, handling of cookies etc?
• Does LocalSite.cfg contain a valid username and password?
• Does FoswikiSeleniumTestCase support your site's login manager?
• Does the webserver configuration (e.g. apache rewrite rules) affect the test?
• ...
• Add more things to check here

The tests might also fail because of browser variations, or even because of a bug.

Use the SeleniumConfigTests to verify the basics of your selenium RC test setup.

Selenium is finicky

It can be tricky to test with Selenium. Some of this may be Selenium-specific, some of it may be a function of browser variations, and some may simply be a product of the HTML/JavaScript specifications:

• You can retrieve the text of a <textarea> element. You can also retrieve the value of a <textarea>, and they can differ. You can "type" into a <textarea> and that sets the value, not the text.
• Asynchronous javascript is common, and you have to allow time for the events to fire.
• Selenium has commonly-encountered problems
• Selenium has bugs. For example, it is reported that onsubmit events are not always triggered.
• There are some things that Selenium can't do. For example, Selenium cannot deal with popups that come from a page's onload event.
• Selenium loads the browser with a specific profile. That affects the available plugins, which could affect tests for content that relies on plugins (e.g. flash or java). It also means that, by default, firebug is not available, which makes debugging more difficult.

Browser variations

• Selenium does not behave the same in all browsers. Selenium reports success for some browsers when, on others, it reports failure.
• Selenium does not support all versions of all common browsers. Selenium is updated to support new browsers from time to time.
• Selecting elements based on their style works differently in IE compared to Firefox, Chrome etc.

ASSERTs and rest handlers

If you are testing with FOSWIKI_ASSERTS set to 1 in LocalLib.cfg, and your selenium test invokes a REST handler that triggers an ASSERT, Foswiki responds with a 500 but the error message might not end up in the apache error log at all. Your test may simply time out. What went wrong? You can find out be setting FOSWIKI_ASSERTS to 'soft' which enables the asserts but failed asserts go to the debug log without interrupting the normal execution flow - then repeat the test, and this time the ASSERT output will be captured in the apache error log.

This issue does not affect other kinds of unit tests because this problem only manifests when invoking REST handlers via a browser.

Known-working configurations

Operating system	JRE	Selenium RC	Browser	Additional config	Comments	Reported by
Ubuntu 8.10	Sun's 1.5.0 (1.5.0_19-b02)	1.0.3	FF 3.0.5			MichaelTempest, 25 April 2010
XP SP3	Sun's 1.6.0 (1.6.0-b105)	1.0.3	FF 3.6.3			MichaelTempest, 25 April 2010

Note that Selenium RC server 1.0.3 reports its version as "v2.0 [a2]"

Installing on Debian / Ubuntu

1. you need to download selenium from their website. Put this in a selenium directory. You need at least the server part. Uncompress it, so you have a selenium/selenium-server-1.0.3/selenium-server.jar
2. you need a decent java, so install Sun's JVM: apt-get install sun-java6-jre
3. you need a X server. If you don't have one, for example on a headless machine, you can use Xvfb: apt-get install xvfb

Then, all you need to do is to start both before starting your tests, that is:

Xvfb :1 &
DISPLAY=:1 java -jar selenium/selenium-server-1.0.3/selenium-server.jar&

This should create a selenium server listening on port 4444, which you can then configure in your LocalSite.cfg:

• Note that you need to specify an existing user that has 'sufficient' permissions ***fillmein***

$Foswiki::cfg{UnitTestContrib}{SeleniumRc}{Browsers} = { firefox => { host => 'localhost', port => 4444, browser => '*firefox' } };
$Foswiki::cfg{UnitTestContrib}{SeleniumRc}{Username} = 'SomeFoswikiUserName';
$Foswiki::cfg{UnitTestContrib}{SeleniumRc}{Password} = 'some Foswiki user password';

And you should be all set! Don't forget to kill -15 %2 %1 when you're done, to avoid having your selenium server and your Xvfb laying around for nothing.

Problematic configurations

Avoid selenium RC server version 1.0-beta-2 as it has problems on XP with IE8. (Version 1.0.3 is much better.)

Operating system	JRE	Selenium RC	Browser	Additional config	Status	Reported by
XP SP3	Sun's 1.6.0 (1.6.0-b105)	1.0.3	IE 8.0.6001.18702		The simpleParas ROUNDTRIP test in BrowserTranslatorTests fails because the TML ends up with windows-style newlines ( \x{d}\x{a} instead of \x{a} )	MichaelTempest, 25 April 2010

Todo

• Support apache login
• Make the login manager pluggable
• Lots more tests to be written
• Define a strategy for choosing which tests must be done via the browser
• Find out how to enable firebug in the selenium browser profile

See also

• UnitTests
• Tasks.Item1326

Discussion

it uses Test::WWW::Selenium ?

and If i'm lucky Alien::Selenium will install it for me - though I've not decided what computer to lumber with a java install :/

-- SvenDowideit - 12 Nov 2010

Yes - there is WWW::Selenium and also Test::WWW::Selenium - and I installed both from CPAN, IIRC.

I am not sure how Alien::Selenium will help. I downloaded SeleniumRC from the selenium website and extracted the .jar file. That was all there was to installing it. It was somewhat trickier finding out how to use it

The only working example in SVN (that I know about) is WysiwygPlugin's BrowserTranslatorTests

-- MichaelTempest - 12 Nov 2010

yup, the Alien package only installes the core so i had to do more.

updated your cfg setting above, as you spelt it wrong :), and I needed to set the username&password, in this case i'm using the admin user...

Another thing that might trip up someone trying it out, is that as the unit test creates topics that are then to be manipulated by the webserver from browser requests, you must run the unit tests from the same user as the cgi's are run (which is not necessary for normal perl only unit tests)

-- SvenDowideit - 21 Nov 2010

Excellent - I am delighted that someone else is playing with this. There is another assumption here - the web server is not using a persistent cache in RAM i.e. it is fine to change files on disk.

I have been running the unit tests as the web-server user because I could get it to work that way, not because I think it is the right way to do it. I am still not happy with the way I am managing the permissions. There is obviously more than one way to do it, and different ways are better for different circumstances, but there must be some common patterns. I am an embedded-systems developer from the world of windoze and not a linux admin, so I for one would appreciate some pointers to "good practice" for managing file permissions and ownership.

BTW - I installed the Selenium perl libraries from the ubuntu .deb repositories this weekend, so it may well be in Debian's too. libtest-www-selenium-perl

-- MichaelTempest - 22 Nov 2010

I downloaded selenium-server-standalone-2.3.0.jar but this gave an error

Failed to start new browser, Firefox 3 could not be found in the path!

This can be solved by added the absolute path to your Firefox browser. In case of Mac OS:

$Foswiki::cfg{UnitTestContrib}{SeleniumRc}{Browsers} = { firefox => { host => 'localhost', port => 4444, browser => '*firefox /path/to/my/Firefox.app/Contents/MacOS/firefox-bin' } };

-- ArthurClemens - 11 Aug 2011