Purpose & Scope
Provide basic information about Visual Studio Unit Test framework for Software Engineer to get started with it.Introduction
Since at least Visual Studio 2012 the well known IDE from Microsoft does support its very own test framework.Visual Studio Online also known as Visual Studio Team Services is capable of running your unit tests through that framework and provides reporting.
I have limited experience with Visual Studio test or other test frameworks. I usually find them over complicated.
In fact I find anything beyond checking an executable exit code, zero for passed, failed otherwise, over complicated.
It seems to me like all those test frameworks insist on having you pack your tests in some kind of DLL plug-in for the framework to load and run.
I guess such systems scale possibly better than plain old executable but that won't even start to be an issue until you have hundreds of unit tests to run.
Even then nothing is preventing you to pack as many tests you want in a single executable and you can still get really good reporting by using good old console output.
So in keeping with the KISS principle I set on a quest to find a way to run test executables through Visual Studio Team Services VsTest task.
Tools
At the core of Visual Studio Testing Tools is the command line executable vstest.console.exe.Through so called Test Adapters VsTest can theoritically run unit tests from any framework.
By default I could not get it to load my executables, be it locally or on the Visual Studio Online hosted build agent.
I'm assuming it was expecting some kind of DLL implementing Microsoft own test framework. As mentioned above I just wanted to use plain executables rather than adding dependency on some unit test framework. So I set on implementing my own Test Adapter capable of running my executable unit tests.
Test Adapter
A Test Adapter allows VsTest to load or discover your tests and run them. You typically get Test Adapters from various main stream test framework so that they can hook in VsTest.Here is a blog post documenting how to author a new Visual Studio unit test. Since a Test Adapter is fundamentally just a Visual Studio extension you will find that post useful too.
You can give your Test Adapter DLL any name you want and it will be loaded by VsTest once copied to the extension folder which should look like the following for Visual Studio 2015:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
However if you want your Test Adapter to be loaded automatically by Visual Studio Team Services hosted agent, the DLL name must end with ".TestAdapter.dll".
If you want to test locally your test adapter in the same fashion that it will be used on Visual Studio Team Services, make sure you do not copy it to the Visual Studio extension folder and then run a command similar to that one:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" TestHeapCheckError.exe /TestAdapterPath:C:\Dev\VisualStudio\ExeTestAdapter\bin\Debug
The Test Adapter for executables we produced is now freely available as a NuGet package. We will probably publish the source code at some point.
It works as follow:
- If your executable exit code is different than zero your test will fail.
- If your executable output anything other than blank lines on the error output your test will fail.
- Your tests outputs are visible on the Visual Studio Team Services console.
- On Visual Studio Team Services test reporting page for your build each unit test as an attachment containing its output.
Visual Studio IDE integration
The above Test Adapter will enable your continuous testing on Visual Studio Online. However we omitted the parts allowing integration with Visual Studio IDE test explorer.I understand test explorer conveniently lets you run your unit tests from your Visual Studio IDE.
Here are a few links that could help you get started implementing your own:
Last edited: