JavaScript, test automation, tools, UI Testing, Web

Unit testing JavaScript with VisualStudio and Resharper – running on buildserver + code coverage using Chutzpah

Introduction

Unit tests are priceless but only when run automatically. Otherwise they can become forgotten and stay red (or ignored) for the rest of their lifetime. In the last two posts about unit testing in Javascript [1, 2] we have prepared our solution. Now it’s a very good time to tell our buildserver to run them during the build process. I chose Chutzpah as the test runner.

Chutzpah

Chutzpah, pronounced [hutz-pah] from Yiddish/Hebrew, means “shameless audacity, utter nerve”. It’s a JavaScript tests runner that uses internally PhantomJS, Jasmine/QUnit to run the tests and BlanketJS to generate code coverage. The package contains both VisualStudio extension to run the tests (but I prefer Resharper) and console runner used by buildservers (especially TeamCity). Beside its core functionality chutzpah can also generate report with code coverage of JavaScript. Among all known approaches to run javascript test, in my opinion using Chutzpah is the simplest.

Getting started with Chutzpah

  1. Download Chutzpah and extract it to some well-known place (in my case C:\BuildTools\chutzpah)
  2. Add chutzpah.json configuration file to the test project folder. This step is optional for chutzpah, but in our case we use root path while referencing files in unit tests and this needs to be configured specially in the config file (there is also some additional information about code coverage pattern):3
  3. Run chutzpah console runner with parameters:
    /path – place where all the testes are
    /codecoverage – tell chutzpah to generate html report with code coverage

2

There is also an extra output file with code coverage information:

4

Preparing package for build server

Now we are able to run tests on build server – just run chutzpah.console.exe with specific parameters. the build should fail if any test fails. The other thing is the reporting to the build server about the test result. Fortunately Chutzpah has built-in support for TeamCity (/teamcity command line swith) but also serves XML output (/junit switch).

To have it on Teamcity I needed only to prepare MSBUILD file with the build procedure (I prefer to have everything in source control, especially build procedure, that’s why I don’t believe in TeamCity build steps):1

After adding this file to TeamCity we should see:

5

And after configuring report tab for “JS Code Coverage”:

6

Have fun!

Demo

Download and play with the demo solution:
FrontEndTools_buildserver.zip

Advertisements
test automation, UI Testing

Why to do automated UI tests?

imageWe live in the time of TDD and unit testing. We already know that it’s one of the best practices so we do it in hope that it solves most of our problems. But is unit testing really enough? If you feel it’s not, and have heard of other types of test automation, but you’re not sure if they are for you, let us think it over together.

 

 

What is test automation

Test automation is all about reducing repeatable manual testing effort by automated scenarios. Manual testing (in web application) is checking if page looks OK in different browsers, checking if JavaScript code runs correctly, data is persisted to the database, there are no performance hits, email sending mechanism does its job etc.. These scenarios can be both recorded or coded. It is not only about UI testing (with Selenium or WatiN). In testing environment exists the term of “test automation pyramid” (by Mike Cohen) that places different kind of automated tests on specific level according to it’s complexity (the higher the more complex) and number of tests (the least number on the top).

Common arguments

The most common argument for test automation is that it reduces the cost of testing. It’s how people perceive this kind of practice. And indeed, hours of our manual clicking through the system (especially if you do not have dedicated tester) costs a lot, and in the same time you could do something considerably more valuable. Test automation gives you faster feedback after finishing some feature – automated tests should run in a few minutes and tell if it’s done or by doing one improvement you broke three other parts of the system.

As long this is pretty common understanding, it’s sometimes not enough for people who are not struggling with such problems because they are not doing regression at all (very common problem which is even worse than long and expensive manual regression).

Among this we forget about few more important arguments for test automation, especially from developer’s perspective.

Most important benefits

1. Test automation enables high level refactoring – the most basic rule behind TDD is: red, green, refactor. And you might have felt that because of unit testing you were able to refactor the code. But unit testing is about a good design of project, not about the best refactoring. Low level testing allows low level refactoring and high level testing allows high level and exciting refactoring.
My teammate and I have a list of expensive refactorings that we want to do if we have time – we call it “wishlist”. Several items from the top: upgrade NHibernate, upgrade jquery, upgrade jquery ui, css cleanup, clean up db objects, remove legacy ORM etc.
Does unit testing allow us to realize all of these wishes? Of course not. But thank to UI testing we can refactor the UI part – jQuery, CSS etc., due to acceptance tests we can fix business logic, due to integration tests – upgrade external libraries and so on. This is the most exciting ability that test automation gives to developers.

2. Test automation enables continuous delivery – continuous delivery is amazing because we can get feedback from the user in a very short time. It means that the team needs to deliver frequently and it’s obvious that every deploy to production needs to be preceded by a lot of testing. Without test automation it’s even impossible to proceed with such a practice.

3. Test automation stabilize regression – the first question is – do you have a proper regression scenario? I’d bet in most cases it’s like that: for the next “X” amount of time everyone is testing the system. Do we know what to test? Are we sure that we do the same regression as for the last time? No, because manual work -  deployment or testing – is error-prone. We do not like going step-by-step through the manual. In contrary – computer likes it. What is possible to automate should be automated.

Summary

As you can see there are many pros to test automation. Why isn’t it as common as unit testing? The answer is simple – it’s hard to work out proper test automation solution, and in opposite it’s considerably easy to set up very bad and impossible to maintain test automation that discourages managers from further investment in such practice. But of course it’s worth trying.

In future I’m planning to write more about rules for writing good and stable automated tests.

tools, UI Testing

Resharper template for unit test method (replacing spaces in the test name with underscore sign)

Resharper live templates are great to automate repetitive work. An example of such work is unit test method. In the first part I’ll show how to create a basic template that will build the structure of a unit test. In the second one I’ll show how to make R# give a correct name to such test based on human-readable description.

Test template

Below we have an example of a test method in AAA style:

[Test]
public void when_doing_something_then_expect_something()
{
   //ARRANGE
   var x = 100;

   //ACT
   var result = this.Calculator.Sqrt(100);

   //ASSERT
   Assert.AreEqual(10, result);
}

As we can see to create a nice structure of AAA unit test we need to remember about three comment labels, attribute, correct test name. Such structure can be easily extracted as resharper live template. To do it:

1. Go to Resharper –> Templates explorer

2. In “Templates explorer” select “C#” section and click “New template”:

image

 

3. Fill in live template form – edit shortcut and content:

image

After that the new template is ready to be used. Just type the shortcut entered before, click ENTER and this is it!

image

Converting a sentence into a valid test name

Another problematic thing in writing unit tests is naming the test. We want to give self-describing names to tests. Sometime it’s a rule, to name tests like “When something then something”. This way we use a human-readable sentence in a place, where spaces between words are not allowed. Thus we are forced to replace them with underscore (‘_’) sign, or capitalize each word – both ways are highly uncomfortable and drive developers mad.

However it is possible to use resharper macros to do additional logic for template fields. We can choose a macro for each field (look at the picture of the template form). Unfortunately here is a first disappointment – there is no bult-in macro to achieve this. It’s even worse, because there is a macro called “Value of another variable, where spaces will be replaced with ‘_’”, but this one cannot be applied to the same variable (test name in our case).

 

To overcome this disadvantage it’s needed to create custom macro applied as a plugin. Below there is a code of the macro:

    [Macro("chsreplacespacewithunderscore",
            LongDescription = "Replace spaces with '_' (i.e. \"should return nothing\" into \"should_return_nothing\")",
            ShortDescription = "Replace spaces with '_'")]
    public class SpacesToUnderstrokesMacro : IMacro
    {
        public static string Evaluate(string text)
        {
            return text.Replace(" ", "_");
        }

        public ParameterInfo[] Parameters
        {
            get { return EmptyArray<ParameterInfo>.Instance; }
        }

        public HotspotItems GetLookupItems(IHotspotContext context, IList<string> arguments)
        {
            return null;
        }

        public string GetPlaceholder(IDocument document)
        {
            return "sentence with spaces";
        }

        public string EvaluateQuickResult(IHotspotContext context, IList<string> arguments)
        {
            if (context.HotspotSession == null || context.HotspotSession.CurrentHotspot == null)
            {
                return null;
            }
            var currentValue = context.HotspotSession.CurrentHotspot.CurrentValue;
            return Evaluate(currentValue);
        }

        public bool HandleExpansion(IHotspotContext context, IList<string> arguments)
        {
            context.HotspotSession.HotspotUpdated += this.CurrentHotspotUpdated;

            return false;
        }

        public void CurrentHotspotUpdated(object sender, System.EventArgs e)
        {
            var hotspotSession = sender as IHotspotSession;
            if (hotspotSession != null)
            {
                hotspotSession.CurrentHotspot.QuickEvaluate();
            }
        }

To compile, it is needed to reference the following R# libraries (from the installed version of R#, in my case R# 7.0):

JetBrains.Platform.ReSharper.DocumentModel.dll
JetBrains.Platform.ReSharper.Shell.dll
JetBrains.Platform.ReSharper.Util.dll
JetBrains.ReSharper.Feature.Services.dll
JetBrains.Platform.ReSharper.ComponentModel.dll

Most of the code is about implementing the standard IMacro interface. Most important here is the method EvaluateQuickResult that evaluates the value of the current field (hotspot) in a template. The same function is invoked each time the HotspotUpdated event is triggered.

Newly build plugin can be now added to resharper. Just copy it to “C:\Users\some.user\AppData\Roaming\JetBrains\ReSharper\v7.0\Plugins” folder and restart VisualStudio. After that the new macro can be added to the template and the test name will be transformed to the valid identifier on the fly.

Download

Instead of building your own plugin, you can also download it from here

Update

I also highly recommend creating another macro just to replace spaces with ‘_’. I’ve got ‘testn’ = ‘test name’ and it’s used extensively for renaming unit tests

UI Testing

Get value of Selenium IWebElement

Introduction

Selenium.WebDriver (SeleniumHQ) is a great tool that lets you automate your Web UI Tests. With the tool you can navigate through web pages, find elements, put text into input controls or click links or buttons placed in that pages. Unfortunately as life shows, you can always find some lacks. The lack of element’s value getter is one of them.
 

Example scenario

You build a system where the user can enter data in some popup dialogs. The obvious test you may write will be the test for saving the given data, but another one may be the test checking that after saving the data and reopenning the form all the fields are empty again. Unfortunatelly you cannot use the Text property getter, because you are dealing with the INPUT control (which moreover is changed by JavaScript thus the DOM’s innerHtml property is not set).
 

Solution

You could say that unless the element were IWebElement in Selenium.WebDriver but mere DOM object you could get it’s value using javaScript, i.e. jQuery val() function. Happily Selenium.WebDriver provides a way to execute javaScript in the browser. The idea is to make the element identifiable and execute a script that finds the element and gets the value. Here’s my implementation of that idea (I have put everything to C# extension method):
public static string GetValue(this IWebElement element)
{
//Ensure that there is are identyfiable elements
Contract.Requires(element != null);
Contract.Assert(String.IsNullOrWhiteSpace(element.GetAttribute("id")) == false,
"Empty element 'id' attribute");

//Build selector to get the element by the browser
var cssSelector = String.Format("{0}#{1}",
element.TagName, element.GetAttribute("id"));

//Get WebDriver to look for elements and execute javaScript
var webDriver = ((RemoteWebElement)element).WrappedDriver;

//Ensure that the javaScirpt script finds the exact element
var foundElements = webDriver.FindElements(By.CssSelector(cssSelector));

Contract.Assert(foundElements .Count == 0, 
"No element found using the given selector");
Contract.Assert(foundElements .Count > 1, 
"Unique ID constraint violated for the element");

//Build and execute the javaScript
var script = String.Format(@"return $(""{0}"").val()", cssSelector);
return ((IJavaScriptExecutor)webDriver).ExecuteScript(script) as string;
}
 

The design-by-contract programming takes considerable amount of code here but the idea remains simple. The javaScript line is written as jQuery code but if you want you can write here anything that obtains the element’s value.
 

Disadvantages?

The most fragile part of the solution is grabbing the element. There must be a way to identify it with javaScript – I used element’s ‘id’ attribute because it’s simple, but you can construct XPath expression or anything you want to do that. If you test the software you create, you can always influece the way the element was created and predefine the identifier. For autogenerated elements you may provide a way of getting the element using XPath or your own way of getting the descendant of another identifiable element.
What about performance – are we looking for the same element twice!? Yes, but this could takes miliseconds while the whole session of UI Tests could take tens of minutes. The waste of time is repaid by the new functionality coverage.
 
Happy UI Testing!