JavaScript, test automation, tools, UI Testing, Web

Unit testing JavaScript with VisualStudio and Resharper – running on buildserver + code coverage using Chutzpah


Unit tests are priceless but only when run automatically. Otherwise they can become forgotten and stay red (or ignored) for the rest of their lifetime. In the last two posts about unit testing in Javascript [1, 2] we have prepared our solution. Now it’s a very good time to tell our buildserver to run them during the build process. I chose Chutzpah as the test runner.


Chutzpah, pronounced [hutz-pah] from Yiddish/Hebrew, means “shameless audacity, utter nerve”. It’s a JavaScript tests runner that uses internally PhantomJS, Jasmine/QUnit to run the tests and BlanketJS to generate code coverage. The package contains both VisualStudio extension to run the tests (but I prefer Resharper) and console runner used by buildservers (especially TeamCity). Beside its core functionality chutzpah can also generate report with code coverage of JavaScript. Among all known approaches to run javascript test, in my opinion using Chutzpah is the simplest.

Getting started with Chutzpah

  1. Download Chutzpah and extract it to some well-known place (in my case C:\BuildTools\chutzpah)
  2. Add chutzpah.json configuration file to the test project folder. This step is optional for chutzpah, but in our case we use root path while referencing files in unit tests and this needs to be configured specially in the config file (there is also some additional information about code coverage pattern):3
  3. Run chutzpah console runner with parameters:
    /path – place where all the testes are
    /codecoverage – tell chutzpah to generate html report with code coverage


There is also an extra output file with code coverage information:


Preparing package for build server

Now we are able to run tests on build server – just run chutzpah.console.exe with specific parameters. the build should fail if any test fails. The other thing is the reporting to the build server about the test result. Fortunately Chutzpah has built-in support for TeamCity (/teamcity command line swith) but also serves XML output (/junit switch).

To have it on Teamcity I needed only to prepare MSBUILD file with the build procedure (I prefer to have everything in source control, especially build procedure, that’s why I don’t believe in TeamCity build steps):1

After adding this file to TeamCity we should see:


And after configuring report tab for “JS Code Coverage”:


Have fun!


Download and play with the demo solution:

test automation, UI Testing

Why to do automated UI tests?

imageWe live in the time of TDD and unit testing. We already know that it’s one of the best practices so we do it in hope that it solves most of our problems. But is unit testing really enough? If you feel it’s not, and have heard of other types of test automation, but you’re not sure if they are for you, let us think it over together.



What is test automation

Test automation is all about reducing repeatable manual testing effort by automated scenarios. Manual testing (in web application) is checking if page looks OK in different browsers, checking if JavaScript code runs correctly, data is persisted to the database, there are no performance hits, email sending mechanism does its job etc.. These scenarios can be both recorded or coded. It is not only about UI testing (with Selenium or WatiN). In testing environment exists the term of “test automation pyramid” (by Mike Cohen) that places different kind of automated tests on specific level according to it’s complexity (the higher the more complex) and number of tests (the least number on the top).

Common arguments

The most common argument for test automation is that it reduces the cost of testing. It’s how people perceive this kind of practice. And indeed, hours of our manual clicking through the system (especially if you do not have dedicated tester) costs a lot, and in the same time you could do something considerably more valuable. Test automation gives you faster feedback after finishing some feature – automated tests should run in a few minutes and tell if it’s done or by doing one improvement you broke three other parts of the system.

As long this is pretty common understanding, it’s sometimes not enough for people who are not struggling with such problems because they are not doing regression at all (very common problem which is even worse than long and expensive manual regression).

Among this we forget about few more important arguments for test automation, especially from developer’s perspective.

Most important benefits

1. Test automation enables high level refactoring – the most basic rule behind TDD is: red, green, refactor. And you might have felt that because of unit testing you were able to refactor the code. But unit testing is about a good design of project, not about the best refactoring. Low level testing allows low level refactoring and high level testing allows high level and exciting refactoring.
My teammate and I have a list of expensive refactorings that we want to do if we have time – we call it “wishlist”. Several items from the top: upgrade NHibernate, upgrade jquery, upgrade jquery ui, css cleanup, clean up db objects, remove legacy ORM etc.
Does unit testing allow us to realize all of these wishes? Of course not. But thank to UI testing we can refactor the UI part – jQuery, CSS etc., due to acceptance tests we can fix business logic, due to integration tests – upgrade external libraries and so on. This is the most exciting ability that test automation gives to developers.

2. Test automation enables continuous delivery – continuous delivery is amazing because we can get feedback from the user in a very short time. It means that the team needs to deliver frequently and it’s obvious that every deploy to production needs to be preceded by a lot of testing. Without test automation it’s even impossible to proceed with such a practice.

3. Test automation stabilize regression – the first question is – do you have a proper regression scenario? I’d bet in most cases it’s like that: for the next “X” amount of time everyone is testing the system. Do we know what to test? Are we sure that we do the same regression as for the last time? No, because manual work -  deployment or testing – is error-prone. We do not like going step-by-step through the manual. In contrary – computer likes it. What is possible to automate should be automated.


As you can see there are many pros to test automation. Why isn’t it as common as unit testing? The answer is simple – it’s hard to work out proper test automation solution, and in opposite it’s considerably easy to set up very bad and impossible to maintain test automation that discourages managers from further investment in such practice. But of course it’s worth trying.

In future I’m planning to write more about rules for writing good and stable automated tests.