Using MSTEST With Go

This article describes how to integrate tests automated with MSTEST into Go pipelines. There are three successively more complete layers to this integration. You can implement the first, the first and second or all three.

Download this file and expand it into a directory. I recommend putting these files in a “tools” folder in your source control repo and checking them in so they are conveniently available on Go agents.

There are four files:

• test.bat: A batch file that runs all three “steps”. The idea is to use this bat file or one like it from a Custom Command in Go.
• msxsl.exe: A free utility from Microsoft for applying an XSL transformation to XML.
• MSBuild-to-NUnit.xslt: Transformation specification to convert trx emitted by MSTEST into XML compliant with NUnit. This transformation is necessary since Go understands NUnit test results.
• InspectTrx.exe: A utility that inspects the test results encapsulated in a trx file and emits either a 0 or 1 result code to signal Go that the tests ran successfully or not. This is needed because the BAT file masks the result code returned by MSTEST.

My pipeline is three stages: build, test and publish

Drilling into the RunTests stage we find,

Here’s the Edit dialog for the Custom Command:

My test.bat command looks like this:

@echo off
SET PATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest;%PATH%

:TEST
echo Running tests...
del /S %2
mstest /testcontainer:%1 /resultsfile:%2
echo Test run completed.
IF ERRORLEVEL 0 goto TRANSFORM
echo The MSTEST pass produced one or more errors in test results.
SET ERRORLEVEL=1
exit

:TRANSFORM
echo Applying XSL transformation to results...
tools\msxsl %2 %3 -o %4
echo Finished transformation.
IF ERRORLEVEL 0 goto INSPECT
echo The XSL transformation of test results caused an error.
SET ERRORLEVEL=1
exit

:INSPECT
echo Inspecting results for errors...
tools\inspecttrx.exe %2
echo Finished inspecting results.
:END
exit

This command file takes four parameters:

• Relative path to the test container
• Relative path to the test results
• Relative path to the XSLT transformation definition
• Relative path to the NUnit

It is divided into three steps. First the tests are run. Next the test results are transformed to NUnit format. Finally, the test results are inspected for errors so that we can signal Go to display green or red for the test phase.

Using Go With Team Foundation Server

With the release of Go 12.1 our platform for automated build and continuous integration/delivery directly supports Team Foundation Server (TFS) source control as a source of materials. In this post I want to give you a deeper understanding of using this feature. For reference you can find the official documentation about using TFS materials with Go here.

Let’s start with Go’s pipeline model. Here’s the pipeline I use to build the Mingle Extension for Visual Studio. In a nutshell you are looking at the essence of Go here. It would be terribly difficult to model this using Team Foundation build alone. This pipeline is composed of five “stages”: compile, two phases of automated testing, a stage to acknowledge completion of manual testing and finally a stage that packages the installer for the tested product to be published. Note the seamless integration of automated and manual stages of the pipeline. It is quite difficult to do this with Team Foundation Build alone.

Here are two look at the pipeline, one in-flight. The pointers in between the stages indicate whether subsequent stages fire automatically or await manual intervention. In my case I stop the pipeline after the completion of automated testing for a bit of manual testing. When all the manual tests are completed we trigger the resumption of the pipeline through to the end.

Imagine a world where that “publish” stage is polishing off a deployment of your e-commerce app built from Team Foundation source control. You can see how powerful this is.

This pipeline interacts with TFS in several ways as it proceeds. Let’s dig deeper.

Here is the configuration for pulling build material from TFS:

The compile-build stage runs a job that executes MSBuild from a batch file.

The job is configured as a Custom Command that runs a batch file containing:

set path=%PATH%;C:\Windows\Microsoft.NET\Framework\v4.0.30319;

msbuild /t:Rebuild /p:configuration=%1 /fileLogger /flp:logfile=buildlog.txt /flp1:logfile=builderrors.txt;errorsonly /flp2:logfile=buildwarnings.txt;warningsonly vsc.sln

This build produces a number of “artifacts”: logs, a library of automated tests and the testable executable product that will ultimately be packaged for release.

Stage two, automated API testing, runs MSTest over the test library artifact produced by the compile-build stage. This stage is also configured with a job running a Custom Command that calls a batch file, which contains:

del /S testresults.trx

mstest /testcontainer:website.tests\build\tests\website.tests.dll /resultsfile:website.tests\build\tests\testresults.trx

tools\msxsl website.tests\build\tests\testresults.trx tools\MSBuild-to-NUnit.xslt -o website.tests\build/tests/testresults.trx.xml

The final stage, publish, uses TFS’s command line tool, tf.exe, to label the source tree using the value of GO_PIPELINE_LABEL. The value of the environment variable is passed in as a parameter.

tf.exe label %1 $/vsconnector /comment:"applied label" /collection:http://42.42.42.42:8080/tfs/solutions

I hope this helps show a bit of what can now be done using Team Foundation Server with Go. It only scratches the surface.

Agile Development Using Visual Studio–Part I

It is liberating to do big-time development using Visual Studio without Team Foundation Server in the room. I feel 50 pounds lighter. (Those who know me can imagine how good that makes me feel!) This article presents how I develop software for the Microsoft ecosystem using Visual Studio without using Microsoft’s flagship ALM tools. The theme is to find best-of-breed tools and combine them to get something I like, rather than force myself to use TFS, much of which I don’t like.

Let’s start with the desktop tool for editing and testing code. This is where we spend all of our time. I like to Visual Studio when building software for the Microsoft ecosystem since it’s so attuned to working with .NET assemblies. Full stop. Once that decision is made it’s time to look around for how to take advantage of the best  of everything else.

There are excellent choices for integrated source control, task tracking, team collaboration, test-driven development and automated build/deployment using Visual Studio as your integrated development environment (IDE). I use these:

• Visual Studio 2010 Ultimate (IDE)
• Github for source control using the Git Extensions/Mysysgit for Visual Studio
• Mingle from ThoughtWorks Studios (task tracking and team collaboration)
• Mingle Extension for Visual Studio
• ReSharper
• Go from ThoughtWorks Studios for continuous integration/delivery
• IronRuby for some of my automated testing
• Productivity Power Tools for Visual Studio
• PowerCommands for Visual Studio

Visual Studio 2010 was a fabulous integrated gallery of IDE extensions. It’s easy to load up. I try and only install the extensions I need and actually use to make life better. Here’s my list:

My Visual Studio solution has five projects. You’ll find the source for this solution here on Github.

This is the project for our .NET client API for Mingle. The highlights are:

• Class libraries that wrap the the Mingle (and soon Go) REST APIs with XElement wrappers that expose .NET CLR classes for consumption.
• Automated tests primarily written in Ruby using RSpec. Why Ruby, you ask? There’s a nice gem called Sinatra that features a light weight web framework useful for mocking. This makes it easy to take Mingle out of the room when unit testing.
• A setup project to create an installer the binaries.stand-up

 

Task Management and Collaboration

I use the Mingle Extension for Visual Studio connected to a Mingle system hosted internally here at ThoughtWorks. These screen shots are from a sample project showing how I query Mingle “cards” (tasks). We use a number of default queries (favorites). “My Work” lists things assigned to me and conforms to the identity of the logged in user of Visual Studio. Individual cards open into a tabbed UI for editing card details. A row of buttons across he top of the card window gives easy access to pre-defined transitions like “start development”, “start testing” and “complete development”.

Work Flow

Our work flow goes like this:

The backlog is what you’d expect in an agile model; it’s the list of things we want. We use a secondary category called Do, which is the list of things we are committed to deliver soon. Each of us takes things from the Do pile and moves them into a state we call Doing when we are actually working on something. I never have more than a couple of things in the Doing state. We use Ready to denote things that are ready to release – nothing more needs to be done at all. We batch things together before releasing simply because it typically makes sense not to formally release each and every story as it is completed. In our shop Done means the story is in the end-user’s hands and they concur it is what they want. Done really means DONE.

We batch things together into iterations  and releases using properties of cards (tasks/stories/bugs). A nature of our group is that about half the work we do does not need to be batched into releases and Mingle gives us the flexibility to easily accommodate released-based and one-off development in the context of one Mingle project. It is very nice.

One cool feature of Mingle is something we call “murmurs”, a messaging system that can be connected with Jabber instant messaging. Murmurs quite flexible and useful in different ways, all of them important for team communication and collaboration. My team is geographically dispersed in the United States and the United Kingdom. We use murmurs to keep a running “live log” of comments, questions, observations and status. At any given moment any one of us can jump online and review the murmurs history to come up to speed on current status of a project or each other. I do this in the morning when I start work on the west coast of the United States to catch up on what my colleagues have been doing all day in their office near London.

I use the Mingle Extension for Visual Studio to give me access to a lot of Mingle from within the Visual Studio environment. The following diagram shows you the mains elements of the extension, which supports running pre-defined “favorite” queries in Mingle, drilling down into individual cards, editing and composing new cards.

We have exposed murmurs inside Visual Studio as a lightweight instant message window. You can murmur directly into the window OR from comments to individual cards (tasks). My team’s model for using murmurs has become very a easy and thorough way for us to stay current with each other over thousands of miles in real time. Because murmurs are integrated into Jabber our Jabber IM clients light up if Visual Studio isn’t running. Very cool.

This is enough for this post. Next I’ll tear into the joy of using Git with Visual Studio (instead of TFS), automated testing using a combination of MSTest and Ruby (use the best tool for job) and continuous integration using Go.

Mingle Portal Page - Visual Studio 2010

With Visual Studio 2010 you can set a portal page link for any TFS Team Project. When using Mingle I set my portal to be my personal Mingle card wall, making it very easy for me to jump to the wall and saving clicks and keystrokes.

To do this:

- Right click on a team project node in Team Explorer

- Choose Team Project Settings -> Portal Settings

- Past in the link

Mine looks like this:

Card Walls for Organizing Your Work

ThoughtWorks Mingle makes it trivially easy to organize your personal work using 2-D card walls. We introduced them specifically to handle SCRUM projects. Yet they are flexible enough to do lots more. Here’s my personal “to do” list organized by project (rows) and timing (columns). Individual work items are in the cells and color-coded by type (story, task, etc.). It took me less than a minute to produce this. (Apologies for smudging the details, but my employer gets funny about disclosing proprietary information. Smile.)

Sun Java on Ubuntu 11

Took some digging, but this finally worked.

sudo add-apt-repository “deb http://archive.canonical.com/ lucid partner”
sudo apt-get update
apt-get install sun-java6-jre sun-java6-plugin

Visual Studio vNext and Next Gen MSFT ALM

Microsoft presented a bit about their vision for ALM and the next generation of Visual Studio at Tech Ed last month. As usual with these sorts of announcements and white papers the signal to noise ratio of quite low. I'll help with a synopsis for folks who don't want to read the 32-page PDF.  First an editorial comment.
vNext doubles-down on the theme of Microsoft's development environment - automation. Their vision is a world where tools and process are effectively one and the same. They see their mission as providing amazing amounts of automation of routine tasks through the use of tools - their tools. In the Microsoft development ecosystem life is "easy". Microsoft provides you with all the tools you "need". In the process you may be blind to a world of thinking and building software that goes on around you.
 
Resource capacity visualization is nice.




Interactive drag-and-drop Task Board, which looks quite familiar.



Integrated MS Project Server (this is a feature add-on in VS 2010). This is a very cool feature, especially for enterprise resource planning spanning multiple projects. We can also do it with Mingle with a bit of code.



There's something called StoryBoard Assistant, that integrates into PowerPoint. They claim it facilitates better inbuilt graphical design tools, ability to embed other content, ability to create lightweight animations, etc. Personally I prefer things like SketchFlow in Expression Blend, which is a more "design UX in code" tool for WPF.



They've integrated e-mail for collecting stakeholder feedback. Presumably this was done so that it uses the default e-mail provider configured on the Windows client PC rather than being limited to Outlook.  This feature doesn't impress me much without seeing it since my personal bias is that demos are a much more effective feedback collection platform. If viewers are remote then one has tools like Skype, Webex, GoToMeeting and Live Meeting. One nice thing about this is they claim it's integrated into the web browsing environment so that it's easy to create feedback directly from a web page.



Unit testing in vNext is asynchronous to you can code and test at the same time. Sounds iffy to me. Do I really want my in-flight code being constantly tested? I know I want my checked-in code immediately tested using a GO pipeline. I do like the UI treatment of test results here very much.



Integrated code review looks spiffy. (TFS has facilitated code sharing for review using a feature they call "shelf sets" for years. It's one of my favorite features. A shelf set is code on the TFS server that has a name so it can be shared, but it's not checked in.)



They also claim to have added production testing scenarios to facilitate close communication between development and operations.