test automation, tools

Debugging NSpec with VisualStudio/Resharper – simple trick

NSpec is purely awesome, although coming with little painful disadvantage – debugging. The default approach (DebuggerShim.cs) is weak because forces me to always navigate to a separate file and change some code. Authors of this great library claim in the documentation that they already work on “fully integrated support”, but it’s not there yet. Fortunately you can apply one simple trick to make it work far better.

1. Get Nspec: install-package nspec

2. Get NUnit (in version referenced by NSpec): install-package nunit -Version

3. Go to DebuggerShim.cs file and change it to look like that:

nspec - DebuggerShim.cs

And that’s it! You can go back to your specs and be delighted by the “unit test” icon beside your spec class.

NSpec - test result

…and test runner:

NSpec - Run spec

What has happened?

DebuggerShim was a tool (nunit TestFixture) that run some given specification. By renaming it to “nspec” class it’s become a wrapper for all specifications. For NSpec it doesn’t matter because it doesn’t care about NUnit attributes nor abstract classes. If it’s in the same namespace as other specifications, then we do not have to change anything and Resharper starts to detect it as NUnit test fixture (as it derives from our custom “nspec” test fixture).


  • Simple – no need to install any plugin/adapter
  • Unobtrusive – no need to change existing specifications
  • Easy to update – if NSpec gets fully integrated debugging support, then just delete custom abstract nspec wrapper
  • Additionaly as a test output we see specification generated by NSpec runner


  • It runs the whole specification, it’s not possible to debug single context. This is particularly painful in case of big (and slow) UI testing scenarios. In that case I try to split them into several more specific scenarios.

You can download the test sample here: https://github.com/chrisseroka/Blog.NSpec.Debugging

test automation, tools, Uncategorized

Improve your testing experience with NSpec, best BDD test framework

Amongst long list of benefits that automated testing gives our projects we have ‘documentation’ feature. I think it takes even one of the first places. Many times I was looking at unit tests trying to decipher what they did and why they were there. Ask yourself a question if your unit test project is a mixture of test methods or it shapes a specification that you can share with your client. Saying ‘specification’ I meant something like that:


This is a test result generated by NSpec for a sample project. NSpec is a beautiful test framework that I’ve focused on for the last several months and probably will stay for a longer time.

If you really want to change your testing experience it’s more often about the approach not a framework. I hate when people see a goal in moving from NUnit to xUnit because it’s almost only changing attributes used for testing. NSpec also allows you to omit class/method attributes but the key concept is to create scenarios that are cool to maintain. For me the testing experience is close to Jasmine (JavaScript) but here we can use lambdas instead of long “function(){}” syntax. The image is worth a thousand words so:


It generates the following result:


Why better than: MSTest/NUnit/xUnit

These are just TDD frameworks with flat structure. No way to define scenarios. NSpec enables you to build real scenarios. It also simplifies the way we organise tests (far easier to add new test, find the proper place). Contexts decrease the size of Arrange/Given part.

Why better than MSpec:

Both are BDD frameworks, but NSpec has more user-friendly syntaxt. It’s possible to avoid hard string concatenation (like ShouldDoSomething, or should_do_something_else). Test result for complex scenarios looks better, you can have more levels. MSpec also takes advantage of lambdas. If you don’t like it, with NSpec you can write tests without them (using conventions).

Why better than SpecFlow:

I don’t believe that the client can develop SpecFlow*.feature files. Because of that I don’t like the workload involved with maintaining *.feature files and their mappings. I’m a developer and what I really want to do is to write code. Although what’s good in SpecFlow is creating specification-by-example.

NSpec disadvantages:

* debugging – see this official tutorial or install additional test adapter
* no resharper support – still in progress
* lot of lambdas (non-mandatory) – for some people it’s an advantage but maybe not for all

Regarding debugging with resharper I have my own solution.Check this out here

Keep in touch.

JavaScript, test automation, tools, Web

JavaScript/TypeScript unit testing –11 sample projects for Resharper, Chutzpah and Karma


An example is worth a thousand words. But seriously it’s a lot easier to follow some given scenario than searching for tutorials for some tools and then trying to apply them to a project that is not supported in the end. Moreover unit testing the client side of the project is more complicated than the back-end part. Maybe it’s because we’ve got used to creating test projects based on nunit/xunit/mstest/mspec etc. In Javascript world it’s a little bit more tangled: we have Javascript but also move forward to TypeScript. We have plain code without modules or use requirejs to benefit from AMD. For each configuration the setup of unit test project is different. I think having all these samples in one place could help us to start TDD in client-side.
All the sample are based on the sample application from my previous posts about unit testing JavaScript. Actually these are not 11 different sample projects, but 11 versions of the same project so that you can easily catch the difference between the configurations.

Samples (links)


JavaScript JavaScript TypeScript TypeScript
Resharper link link* link** (none)**
Chutzpah link link link link
Karma link link link link

** Resharper 8.2 should support support TypeScript testing [RSRP-394249]

* AMD support in this sample is artificial, code must be based on named modules that are loaded staticaly.

Tools review




Resharper lets you run unit test for JavaScript. Actually it could be enough to just start using it as a test runner. It’s awesome to right-click on solution, choose “run unit tests” and watch the solution passing all the tests: back-end and front-end in one go. Moreover it’s also super-easy to set up. Unfortunately the practice is more cruel – it’s ok for simple Javascript project without AMD.  TypeScript support is not yet in live and AMD support is not yet even planned. And there is no command line runner to let the tests run in buildserver. Otherwise it would be the perfect choice.



As an alternative we have Chutzpah. To integrate it with Visual Studio  you can install “Chutzpah Test Runner Context menu Extension” or “Chutzpah Test Explorer Adapter” (default one for Visual Studio). You can also download command line tool that will run the tests in console or in buildserver (it even generates code coverage report). It has builtin support for requirejs and TypeScript. Unfortunately tests discovery is a little bit too slow to run it spontaneously after each change.



This runner is a must-have in AngularJS world and is a JavaScript alternative to NCruch (runs tests continuously after saving changes). It’s a little bit tricky to configure but after that it is super cool to play with. It doesn’t have TypeScript support out of the box, but because TypeScript is compiled to javascript it deals with it successfuly. It’s possible to configure karma to use Chrome (or any other favourite browser) to be the test runner and it makes it very fast and natural to debug. It’s also possible to set the “run once” mode to run it on buildserver using phantomJS. The dark side of the tool is that it’s not integrated with VisualStudio and you have to stuck with console window (actually there is one plugin that runs karma in VS output window but it’s still a hidden console and not Test explorer adapter).

Happy testing folks!

BTW: if you like the samples, remember to leave a github star 😉

JavaScript, test automation, tools, UI Testing, Web

Unit testing JavaScript with VisualStudio and Resharper – running on buildserver + code coverage using Chutzpah


Unit tests are priceless but only when run automatically. Otherwise they can become forgotten and stay red (or ignored) for the rest of their lifetime. In the last two posts about unit testing in Javascript [1, 2] we have prepared our solution. Now it’s a very good time to tell our buildserver to run them during the build process. I chose Chutzpah as the test runner.


Chutzpah, pronounced [hutz-pah] from Yiddish/Hebrew, means “shameless audacity, utter nerve”. It’s a JavaScript tests runner that uses internally PhantomJS, Jasmine/QUnit to run the tests and BlanketJS to generate code coverage. The package contains both VisualStudio extension to run the tests (but I prefer Resharper) and console runner used by buildservers (especially TeamCity). Beside its core functionality chutzpah can also generate report with code coverage of JavaScript. Among all known approaches to run javascript test, in my opinion using Chutzpah is the simplest.

Getting started with Chutzpah

  1. Download Chutzpah and extract it to some well-known place (in my case C:\BuildTools\chutzpah)
  2. Add chutzpah.json configuration file to the test project folder. This step is optional for chutzpah, but in our case we use root path while referencing files in unit tests and this needs to be configured specially in the config file (there is also some additional information about code coverage pattern):3
  3. Run chutzpah console runner with parameters:
    /path – place where all the testes are
    /codecoverage – tell chutzpah to generate html report with code coverage


There is also an extra output file with code coverage information:


Preparing package for build server

Now we are able to run tests on build server – just run chutzpah.console.exe with specific parameters. the build should fail if any test fails. The other thing is the reporting to the build server about the test result. Fortunately Chutzpah has built-in support for TeamCity (/teamcity command line swith) but also serves XML output (/junit switch).

To have it on Teamcity I needed only to prepare MSBUILD file with the build procedure (I prefer to have everything in source control, especially build procedure, that’s why I don’t believe in TeamCity build steps):1

After adding this file to TeamCity we should see:


And after configuring report tab for “JS Code Coverage”:


Have fun!


Download and play with the demo solution:

JavaScript, test automation, Web

Unit testing JavaScript in VisualStudio with ReSharper – improvements

This is another approach to the previous subject: Unit testing JavaScript in VisualStudio with ReSharper. I kindly encourage you to get back to it before further reading.

Referencing production code in Test project

Last time I suggested “link-referencing” files with production code. Then during the build process files were copied to the test project. This approach was a bit cumbersome:

  • we had code duplication in our solution structure
  • sometimes it wasn’t obvious which file was edited – original file or the file copied to the test project
  • R# was complaining about missing references before the build process
  • You need to prepare test project file (.csproj) to make it work
  • You need to reproduce reference structure and keep it up-to-date while working with tests

Fortunately there is a cleaner approach to referencing production code. The solution is to reference files using root reference path:image

In this case we get the root of the test project, go to parent folder (..) and take the files directly from the project with production code (FrontEndTools). Now the only disadvantage is that each test file should know what’s the name of the folder of the tested project and change it when the folder name is changed – but this is very rare case, isn’t it?

If your Resharper marks the referenced file red, add reference to the tested project (FrontEndTools in my case).

From now on the solution structure looks much cleaner:


Switching to Jasmine

Previously I mentioned that I prefer QUnit with its TDD style, but after some time using both I changed my mind. Here are some of my thoughts:

  • Jasmine is more popular – I don’t have any statistics, this is just my feeling after browsing some open source projects and mixing with angularJS communities – I’ve just found Jasmine as a standard
  • Jasmine with its beforeEach/afterEach is more similar to NUnit SetUp/TearDown
  • Jasmine has own (extendable) matchers that simplifies doing even complex assertions
  • Built-in mocking framework is awesome – and this is actually killer feature – in QUnit you should build mocks manually or add external libraries (like SinonJS)
  • Both are supported by default (and only these two) by Resharper
  • Both have very good support and community background (with plugins and extensions)

Other demo solution improvements

  • jasmine.js referenced to test scenarios – Jasmine is by default supported by Resharper, but to get IntelliSense and get rid of Resharper annotations it’s good to reference proper Jasmine.js file explicitly
  • added tests for ColorBoxViewModel with a sample of Jasmine mocking


You can download the demo here: FrontEndTools_Jasmine.zip

JavaScript, test automation, Web

Unit testing JavaScript in VisualStudio with ReSharper

imageCurrently a huge number of projects is driven by TDD. There are tests of Services, Repositories, Domain objects etc. The attitude to these kind of practices changed and now it’s nothing exciting but our bread and butter. Moreover the architecture of current Web Applications changed as well. In average we spend more than 50% of our time developing front-end side of the app (JavaScript etc) but having in mind, that TDD is only applicable to back-end code. As a ReSharper user I’ll show how easy it is to set up a proper solution with another level of unit testing.


Test application

Let’s imagine that we are doing a web application supporting front-end developers. Among all the functionalities we have a simple reusable module to recalculate color values: RGB/HEX.


Separation of Concerns is one of the most important things while doing TDD, that’s why I chose “KnockoutJS” as the framework connecting our view with the logic. Architecture of such application is very simple. At the moment we don’t even have any “back-end” code, but still can do TDD.


The first test

Let’s pick out the ColorCalculator as the target. This is a “class” that is responsible for calculating color values between RGB and HEX.

Thank to Resharper 6+ we have JavaScript test runner provided out of the box. We can write JavaScript tests with two frameworks: QUnit (TDD style) and Jasmine (BDD style). Here I chose QUnit but you can see version with Jasmine in my next post: Unit testing JavaScript in VisualStudio with ReSharper – improvements. Because these two framework are supported by default, we don’t even have to download/reference any additional libraries. This is how our basic test will look like:


Immediately after closing the test() function resharper will add “Run test” button on the left. In case of QUnit Resharper will run the test in the default browser using QUnit default template, showing the result as it is:


Test result is also shown normally in ReSharper “Unit Test Sessions” window.

Extracting test project

Of course it’s not the best choice to have tests shipped with production code. For most of modules for back-end code we have dedicated “.Test” project. Now let’s extract such a separate project for JavaScript test code.

First add a new project to the current solution. The project type should be “Empty Web Application”  (in contrary to normal unit test project when we add “Class library”). Then you can add your test files to the project. You also need to reference the tested code. To do it, add existing script files as “links”:


This is the drawback of our solution. Whole reference tree needs to be reproduced . But it’s not that bad, we don’t change the reference tree that often but instead we have all the code in one place and it’s checked during the build time. Complete test solution should look like that:


Next you need to do a little trick to copy referenced files to the solution. To do it “Edit project file” and add the following code to the end of the test project:


I hope in the next releases of Visual Studio such behaviour will be applied automatically.

Finally modify a little the test code adding a reference to the code under testing (///<reference …/> expression):


And that’s it! – after adding some tests and implementing the functionality correctly you can right click the solution, click “Run all tests” and be sure that everything is correct!

UPDATE: There is also better and simpler approach to reference production code. I write about it here: Unit testing JavaScript in VisualStudio with ReSharper – improvements

Debugging the tests

Wiseman said: “If you need to debug unit test then something is wrong with it’s unitness.”

PS: If it’s really needed you can debug it in browser as you normally debug JavaScript code.

Switching to PhantomJS

If you don’t want to run the tests in browser you can run it with headless PhantomJS runner (also by default supported by ReSharper). Go to “Resharper –> Options –> Unit testing –> JavaScript Tests” and change the “Run test with” option:


You need to specify the path to PhantomJS that can be downloaded from here.

Further reading

I write also about some improvements in this area (Test framework, Mocking, Referencing production code) and integrating JavaScript unit testing with our build server:
Unit testing JavaScript in VisualStudio with ReSharper – improvements
Unit testing JavaScript with VisualStudio and Resharper – running on buildserver + code coverage using Chutzpah

Download and play

You can download the whole solution here: FrontEndTools.zip

There’s still a lot to do – you can add tests to the ColorBoxViewModel and test “toRgb” function that is far more interesting.
Happy testing!

test automation, UI Testing

Why to do automated UI tests?

imageWe live in the time of TDD and unit testing. We already know that it’s one of the best practices so we do it in hope that it solves most of our problems. But is unit testing really enough? If you feel it’s not, and have heard of other types of test automation, but you’re not sure if they are for you, let us think it over together.



What is test automation

Test automation is all about reducing repeatable manual testing effort by automated scenarios. Manual testing (in web application) is checking if page looks OK in different browsers, checking if JavaScript code runs correctly, data is persisted to the database, there are no performance hits, email sending mechanism does its job etc.. These scenarios can be both recorded or coded. It is not only about UI testing (with Selenium or WatiN). In testing environment exists the term of “test automation pyramid” (by Mike Cohen) that places different kind of automated tests on specific level according to it’s complexity (the higher the more complex) and number of tests (the least number on the top).

Common arguments

The most common argument for test automation is that it reduces the cost of testing. It’s how people perceive this kind of practice. And indeed, hours of our manual clicking through the system (especially if you do not have dedicated tester) costs a lot, and in the same time you could do something considerably more valuable. Test automation gives you faster feedback after finishing some feature – automated tests should run in a few minutes and tell if it’s done or by doing one improvement you broke three other parts of the system.

As long this is pretty common understanding, it’s sometimes not enough for people who are not struggling with such problems because they are not doing regression at all (very common problem which is even worse than long and expensive manual regression).

Among this we forget about few more important arguments for test automation, especially from developer’s perspective.

Most important benefits

1. Test automation enables high level refactoring – the most basic rule behind TDD is: red, green, refactor. And you might have felt that because of unit testing you were able to refactor the code. But unit testing is about a good design of project, not about the best refactoring. Low level testing allows low level refactoring and high level testing allows high level and exciting refactoring.
My teammate and I have a list of expensive refactorings that we want to do if we have time – we call it “wishlist”. Several items from the top: upgrade NHibernate, upgrade jquery, upgrade jquery ui, css cleanup, clean up db objects, remove legacy ORM etc.
Does unit testing allow us to realize all of these wishes? Of course not. But thank to UI testing we can refactor the UI part – jQuery, CSS etc., due to acceptance tests we can fix business logic, due to integration tests – upgrade external libraries and so on. This is the most exciting ability that test automation gives to developers.

2. Test automation enables continuous delivery – continuous delivery is amazing because we can get feedback from the user in a very short time. It means that the team needs to deliver frequently and it’s obvious that every deploy to production needs to be preceded by a lot of testing. Without test automation it’s even impossible to proceed with such a practice.

3. Test automation stabilize regression – the first question is – do you have a proper regression scenario? I’d bet in most cases it’s like that: for the next “X” amount of time everyone is testing the system. Do we know what to test? Are we sure that we do the same regression as for the last time? No, because manual work -  deployment or testing – is error-prone. We do not like going step-by-step through the manual. In contrary – computer likes it. What is possible to automate should be automated.


As you can see there are many pros to test automation. Why isn’t it as common as unit testing? The answer is simple – it’s hard to work out proper test automation solution, and in opposite it’s considerably easy to set up very bad and impossible to maintain test automation that discourages managers from further investment in such practice. But of course it’s worth trying.

In future I’m planning to write more about rules for writing good and stable automated tests.