Mark Ploeh has an interesting post about interfaces in TDD – that interfaces aren’t necessarily abstractions. That’s certainly true. Interfaces don’t guarantee we’re actually following SOLID design principles. In fact,…
For about 3 years, I wrote absolutely horrible UI tests. These tests worked great for a while, but like most people that tried their hands at UI tests, found them to be utterly brittle and unmaintainable in the face of change.
And, like most people, I would mark these tests explicit or ignored, until I ditched most or all of them. I’ve heard this story over and over again, from folks that wrote lots and lots of UI tests, only to see their investment go to waste as the time and effort to maintain these often-breaking tests far outweighs any benefit gained from full, end-to-end UI tests. I gave some presentations on UI testing, but two things I’d like to make absolutely clear:
I know this because our team did this. It was quite a bit of work to get to that point, but once we did, we had a great set of concepts on how we can continue to do so in the future.
In a recent show-and-tell session with another team that I really wish would blog about their efforts (they know who they are 
, it became even more clear to me exactly why we had such success with our UI tests. I didn’t have all the pieces in my head, and lots of our success was completely accidental, but we settled on a set of rules that no matter what our external user interface technology, these rules led to maintainable, effective UI tests.
Whether it’s ASP.NET MVC, WebForms, Silverlight or whatever, the views need to be built from a model type. It doesn’t really matter how that model type is built, CQRS, SQL projection, LINQ projection, AutoMapper or what, but we got the most mileage out of creating models specific for each view. That type represented the information to show on that screen.
Besides being a great source for metadata about the view, a view-specific model, or “view model” becomes a specification for how the view should render. Our views are very convention-based, where things like the presence of a “RequiredField” attribute not only instructs the validation framework to kick in.
There’s the magic string problem with web frameworks such as ASP.NET MVC, that strongly-typed views eliminate. But step 1 – strongly-typed views.
Another big must. Every UI element needs to be generated from a C# expression used to output the display or edit field. With ASP.NET MVC, it means doing something like “Html.DisplayFor” or “Html.EditorFor(m => m.FirstName)”.
We want this because strongly typed views plus expression based UI element generation gives us a very, very powerful tool to build very intelligent views. These intelligent views are critical to easily designing for UI tests. Using the expression gives us a metric ton of metadata information from which we can build convention-based UI element generators, including:
All this information gives us all we need to easily build UIs that don’t break and are easy to test.
When it comes to the mechanics of UI testing, you have to figure out how the UI framework will locate edit and display fields in your specific UI platform. For the web, we chose to use the ID HTML element for both display and edit fields. Display fields were wrapped in <span> tags with an ID value matching the property name of the view model property.
If we coded “Html.DisplayFor(m => m.FirstName)”, this would render “<span id=’FirstName’>Bob</span>”. In the edit fields, the property name shows up in both the NAME and ID attributes of the input field.
Since the HTML generation includes model-based identifiers, this greatly helps locating these elements in UI automation frameworks. Magic strings get eliminated from both your views and your UI tests.
In other frameworks, such as Silverlight and WinForms, element location is done by grabbing the actual field’s name in the form class. In that case, something like “TextBox1” is useless. We want the element identifiers to be driven off of the property name metadata of our view models.
When it comes to building our UI tests, we want to use the EXACT SAME expressions in our UI tests that we find in our views. If I code “Html.EditorFor(m => m.FirstName)” in my view, I need to see something like “Form.TextBox(m => m.FirstName, “Bob”)” in my UI tests. The exact same view model type and the exact same expression are used to generate the UI element, as is used to locate the UI element.
When my UI element generation and location logic are the same, and bound by view model metadata, I no longer have to worry about broken UI tests because of changing my view. If I remove a member from the screen, I delete that property from my view model, and now I need to update my UI test, all before I can commit the change!
I really can’t stress enough how important this rule is. Without it, you’re really spending too much time futzing around with broken UI tests, or just not getting as much mileage as you could have.
If you need to set up data, and you WILL need to set up data, don’t go through any back doors to set things up. Don’t build new data access layers, and don’t go through custom-built back doors to build state that doesn’t go through the normal domain logic.
Whatever your domain layer is, use it to set up your contexts. Don’t design workarounds to get things in a state that isn’t normally possible. If you look for usages of a type or member and it’s only used in tests, you’re asking for brittle tests.
Effective, maintainable UI tests are definitely possible. They can be written in such a way that it doesn’t require lots of babysitting or dedicated personnel to maintain. They can even be automated to run after every commit. They can provide value, catch bugs, and expose all those unintended consequences from changes that complex applications tend to have.
It’s all possible, but only if the UI is designed with tests in mind. I’m sure these aren’t the only ways to create maintainable UI tests, as I know the Dovetail folks also have a lot of innovation in that area, but these rules certainly worked for us.
On a recent large project, we had a goal early on that we didn’t want to have a lot of QA folks manually testing our software. Finding bugs through manual testing is incredibly time consuming and expensive, so we opted to try and build as much quality in to the product. That’s not to say that manual testing doesn’t have its place, as humans are fantastic about using software in ways you didn’t expect.
This was a long project, around 18 months, and will continue to have active development in the future. Very early on we found that a good testing strategy was critical to the success of the project, especially for our team to be able to 1) continue to increase our velocity over time and 2) have the confidence to make both small and large changes in our application.
It took quite a while for us to settle on an effective strategy. This was mostly because we had to learn how to design our application for testability, in all layers of the application. Our team were all experienced in TDD before starting the project, but that wasn’t the only skill we needed to create an effective testing strategy.
Categorizing tests can get annoying. You have functional tests, integration tests, unit tests, acceptance tests, slow tests, fast tests, UI tests, and on and on. We found that our tests belonged in three main categories:
Each of these differs in the scope of what’s being tested. A full-system test exercises the application through the external interface, whether that’s a browser, file drop, queue, WinForms app or whatnot.
Subcutaneous tests work at the layer directly below the external user interface. In the context of a web application, a subcutaneous test in our case would be a form object sent through a command processor, with all the real classes and implementations in place. We bypassed the Controller, which only contained UI layer logic, straight to the domain layer. Send in form object, out pops success/failure.
Finally, we had unit tests. Unit tests are designed to test one class, and can either be fast or slow tests. Fast tests are the normal TDD tests, used to build out class design. Test doubles are used as needed, but strictly interaction-based tests have less value unless we find the interactions very interesting. We also have slow unit tests, which could also be classified as integration tests. These would be things like repository tests, persistence tests, etc.
Our ratio of unit:subcutaneous:full-system tests hovered around something like 10:2:1. We ended the project with something around 5000 unit tests, 1000 subcutaneous tests, and 500 full-system tests that used WatiN and Gallio to drive a browser. The 6000 unit/subcutaneous tests executed in about 10 minutes, while the 500 UI tests completed in about 50 minutes.
Unit tests were developed in a pretty strict TDD manner. We write tests before any implementation is in place, and use the tests to drive the design of the code. These tests help identify design issues, encapsulation problems, code smells and so on.
We strived to not write code that existed solely to enable testing. That often meant that we had a design issue, and responsibilities were misplaced or encapsulation was violated.
As we got further down the pipeline in our project, we started to value interaction tests less and less. Interaction testing through mocking is only really interesting if you’re truly interested about interactions. But more often, we were more interested in side-effects, and interactions were just an implementation detail. What we often did instead is mock out slow or untestable pieces, like repositories, facades over external services, configuration classes, etc. Otherwise, we limited mocking only to places where mocks were the only observation points for what we were interested in.
At some point in large projects, it can become obvious that your design needs a large-level refactoring, to extract out concepts to enable quicker delivery of features. On our last project, some concepts unearthed included:
With each of these, unit tests were actually a barrier to these refactorings. But the barrier only existed because we had relied on these tests to capture all of the interesting behavior in our application. To effectively allow large- and mid-size refactorings, we needed an additional level of testing.
Subcutaneous tests, like their name implies, test everything just below the surface of the user interface. In an MVC application, these would be tests for everything just below the controller. For a web service, everything just below the endpoint. The idea is that the topmost layer in an application does not perform any actual business logic, but just connect the external interfaces with the underlying services.
Subcutaneous tests are important because we want to be able to test business logic with the entirety of the system in play, with the exception of external connection points such as the user interface and external services. While a unit test focuses on small-scale design, a subcutaneous test does not address design, but instead tests basic inputs and outputs of the system as a whole.
To build effective subcutaneous tests, we can try and build uniform pinch points through which common logic flows. For example, we might build a command message handling system, or a common query interface. In a recent project that processed batch files, each row in the file was transformed into a message. We could then craft a message, send it through the system, and then verify all the side effects of processing that message.
Because subcutaneous tests address high-level behavior, rather than design, they are ideal for scenario-based testing strategies such as BDD or the Testcase Class per Fixture pattern. If we want to be able to perform large refactorings, we need these high-level tests to create that wide-cast safety net for business behavior. Subcutaneous tests are also great target points for calling features done, as they focus on more end-to-end logic.
While subcutaneous tests allow us to safely perform larger refactorings, they still do not provide a satisfactory level of confidence that our system will work in production.
Our team originally called these tests “UI tests”, until more and more of our projects entailed integration strategies where the inputs to our system weren’t a browser, but instead messages, a REST endpoint, or FTP drops and batch file processing. UI testing is a subset of full system testing. The idea behind a full system test is that we want to test our software as it might be used in production. For an MVC application, these would be browser-based tests. For batch files, we would use actual files. REST, actual HTTP requests. Messages, real queues and messages.
If we want to know if our application works as expected, before it goes to production, one effective and efficient way to do so is to create an automated test that exercises the full system. If my UI tests logs in to the application, places an order and I can verify that an order request was generated, I’m feeling pretty good about things.
One common misconception about full system tests is that they are black box tests. While these have their merit, full system tests should have intimate knowledge about what’s going on underneath the covers. In fact, full system tests can even take advantage of the domain model to build up data, instead of a back-door system built solely for testing purposes. One big mistake teams run in to is not following the same code paths in testing as they do in production, leading to wacky invalid, impossible states of the system.
In our projects, a full system test is the last code we write before we call a feature/story done, done, done. Manual testing is just too expensive and unreliable for characterizing “done-ness” of a feature, but if I can do the exact same actions as would happen in production through the exact same external interfaces, that’s success.
In an application without tests, we’ve actually found it most valuable to start with full system testing, moving down towards unit tests as a means of a strategy for coverage. We cast the widest net possible, but the simplest assertions first, then slowly move down towards unit-level logic. In new applications, we tend not to focus on any one area, as all of these tests are critical to us for long-term maintainability of a system.
This testing strategy does require a level of investment. We’ve found this holistic approach especially effective when we know that this application is critical to our client’s business. If an application is critical to business, it’s going to require change. If it’s going to require change, we better be able to safely change it without affecting our client’s business.
While driving design with unit tests, I often break behaviors out into separate classes, both to increase cohesion, and as a side effect, increase testability. Occasionally, I run into situations where I have some sort of environmental variable that never changes. Or, if it does change, it would take an act of Congress.
When designing these environmental “astronomical constants”, I take the “JFHCI” approach. These are values that have never changed, and the customer has told us don’t need to change. One case I ran in to recently is a reward limit. If you purchase over X dollars, you earn a reward. The value X does not change, so I made it a constant:
public const decimal GadgetSpendRewardLimit = 200m;
My “testability” hat on says that constants are bad, and that I need to be able to properly set up all environmental variables. However, in this case, allowing this value to change in ANY way produces a design that does not match reality. In reality, this reward limit does not need to change.
Instead of going through a bunch of hoops to allow this value to change solely for testing, I’ll leave the value alone. However, in the off chance that this value DOES change, my tests make the assumption that this value is this constant.
So, I’ve introduced testing assumptions made in my tests with a set of preconditions in the setup portion of the test:
[SetUp] public void SetUp() { Debug.Assert(Customer.GadgetSpendRewardLimit == 200m, "Assumes threshold is $200");
I use a Debug.Assert mostly just to signify that I don’t need to test this every time, so I don’t really need a regular test assertion. Instead, this is just a fail-safe, and matches the intent of assumption validation, rather than behavior validation.
With assumption checking through preconditions, I keep the correct design (an immutable constant), while providing explicit callouts of assumptions made about the “correctness” of my tests. If my assumptions are wrong, I won’t bother trying to continue the test, as the base set of assumptions the test was built around are no longer valid.
Michael Feathers posted a thoughtful piece describing the general problems of UI testing tools and the industry in general. In general, I’d agree here. Automation tool vendors, as with almost every tool vendor out there, are eager to solve perceived software development problems. Unfortunately, these tools usually only address symptoms of larger problems.
With UI testing tools, this is no different. When we started out authoring UI tests for a large MVC application, we had a few large goals in mind. The tests needed to be:
To achieve these goals, we needed to do exactly what we do when we write code using TDD, design for testability. I won’t go into everything we did, there’s already a whole screencast on that. However, Michael does bring up some key points:
That’s what we’ve done, and it’s been very successful. We have UI tests, around 500 of them, and we’re not staying up until all hours of the night keeping them working. We also have subcutaneous tests, that work in a layer right below the UI layer, that allow us to send a UI message into our system and check what comes out on the other side, whether it’s business rule violations or entities modified.
All of our testability enhancements came from design for testability. We use strongly-typed views. We don’t have any business/domain logic in controllers, only UI controller logic. Separating your architecture into layers isn’t for portability, it’s for separation of concerns. Once we started designing for testability in the large, and not just the unit level, we were able to maximize the value of automated tests, both at the UI, integration and unit level.
If anything, this is another example of the need for a developer to work on their spidey sense for vendor product demos. We rail on MS for drag-and-drop demos, but it’s every tool vendor that employs this technique. Just like every other product sold in the world, it’s up to the buyer to be critical of the true value a tool can bring.
As promised during the session at Remix 2009, here’s my example code and slide deck.
Abstract: "Building a Twitter clone in 60 minutes, featuring what’s new in ASP.NET MVC 2 preview 1 and focusing on some of the core ASP.NET MVC features like security and routing."
Example code can be downloaded here: ASP.NET MVC Wisdom – ReMix.zip (8.91 mb)
Thank you for attending!
Early this week we ( Jeffrey Palermo and I ) gave a Continuous Integration Workshop in Austin. We were able to record the workshop and our company Headspring Systems made the recordings available on their website. There is just under 3 hours of video covering the following topics:
These are not professional videos by any means, but there seemed to be some interest in hearing how we do ci. I am personally excited that we have these videos because it will help me improve how I present this information in the future.
You can view the videos and download the slides from here:
http://www.headspringsystems.com/services/agile-training/continuous-integration-boot-camp/
Enjoy!
I am doing a Continuous Integration Workshop next Tuesday. The admission is free and will be held at the Microsoft Office.
To register go here: http://www.headspringsystems.com/services/agile-training/continuous-integration/
This will cover the basics of what continuous integration is as well as the the advanced techniques like Database Configuration Management / Continuous Database Integration. At Headspring we implement CI on all of our projects and we can provide some perspective on applying CI to .Net applications. If there is anything of particular interest you would like to see at the workshop please feel free to comment below.
If you cannot attend this workshop I encourage you to read this book: Continuous Integration: Improving Software Quality and Reducing Risk and have a conversation with me.
See you then.
There were some changes to the nCover xml reports for code coverage which will break your existing integration with cruise control .net. Specifically if you use the Statistics feature of Cruise Control which is one of the best features it provides IMHO, the xml nodes have changed and ncover no longer provides the total percent of coverage of the project as a single xml Node (or attribute). This means that you need to do a little more work..(not much) in order to have your code coverage collected as part of the ongoing statistics inside cruise control.
You can view the statistics through this link on the project page inside the CCNet Web dashboard.
The coverage is not being collected with the new version of ncover…
Here is the ugly side of Cruise Control .Net.. The xml configuration. I know it is ugly and painful to deal with… The line underlined is the new xpath selection for code coverage.
<statistics >
<statisticList>
<statistic name="Coverage"
xpath=" (//trendcoveragedata/stats/@vsp div ( //trendcoveragedata/stats/@vsp + //trendcoveragedata/stats/@usp ) ) * 100 "
generateGraph="true"/>
…
</statisticList>
</statistics>
The old xml node for selecting the coverage was a firstMatch element, that must now be a statistic element. The old xPath must be replaced with this horrible statement which calculates the percentage of code coverage for the project. It helps to know that VSP = Visited Sequence Point and USP= Unvisited Sequence Point. Once you know that the rest falls into place.
Once the data is collected it falls into place and shows up in the trend chart….
