Reading Ayende’s post on integrating Rhino Service Bus and RavenDB, one thing that caught my eye was the use of the ThreadStatic attribute to control the lifecycle of a private field: One of the things that really bothers me about … Continue reading →
When one nurtures a production system for a long period of time, new requirements or examinations of existing behavior typically start with a set of assumptions of how the existing system should behave. However, I’ve often found that my assumptions … Continue reading →
Today I plan on writing a stored procedure. Not because I’m being forced to, but because the problem, as far as I can determine, requires a set-based solution. It’s a once-a-month process, doing bulk import, update and export of up … Continue reading →
I’m looking at a database table this morning, looking to optimize a few queries by adding some indexes. The trouble is, this table already has many indexes, all on different columns. So now I’m thinking, is this a smell that … Continue reading →
One of my favorite posts by Ayende covers the idea of concepts and features. Design occurs at the concept level, whereas features build on top of concepts. This can be a little abstract, but I recently ran into a situation where the presence of a concept in one area allowed for feature development, while the absence of a concept in another area made change difficult.
In this system, we integrate with many 3rd party vendors through file drops. Some data has changed in another system, or users have interacted with the system, or system-level events have occurred. When this happens, our system consumes these files and makes decisions based on what the data in the file represents.
These files come in a variety of formats, CSV, tab-delimited and XML. But one common concept in all of these files was that they represented a set of messages. We already introduced the concept of executing a “batch job”:
public interface IBatchJob { void Execute(); }
We then had a batch job executor that could execute any batch job. When a requirement came up to log exceptions when a batch job executed, this was a trivial task. We already created a system-wide concept of executing a batch job, it was now just a matter of adding that feature.
The next concept we needed to augment was the idea of handling a message. Each entry in the file was transformed into a message object, and a message broker located and executed the appropriate handler:
public interface IMessageHandler<T> where T : IMessage { void Handle(T message); }
We needed to know specifically when a handler failed, to log the specific message and exception so that the exception could be examined and the message potentially re-processed. Again, because all control for executing message handlers went to a single message broker, this is trivial to implement. Applying the Single-Responsibility Principle and concepts of orthogonal design meant that no implementation needed to change to add this concept.
The tipping point for identifying concepts comes when adding of a kind of feature becomes difficult, and needs architectural design work. In our case, it was the need to record the history of processing files.
We already have the concept of executing batch jobs and handling messages. The missing concept was processing a file. That is, locating a file, parsing it and converting to messages, and actually executing those messages.
There are about five different ways of doing this at the moment. In some cases we use a persistent batch object to represent the workflow of processing the file. In other cases, we just have some parser class that then invokes the message broker.
But in no place in our system architecture was the concept of processing a file. When it came time to add features to this concept, and the concept didn’t exist in our system, it became pretty daunting to augment a non-existent design.
In this case, we waited too long to formalize the concept of parsing files. We hadn’t needed to add new features to this absent concept, but building out the concept itself with the existing shape of disparate features would have made adding subsequent parsers much easier to think about and develop.
Concepts are quite similar to building blocks. Building on top of a concept means I already have a set of building blocks to start from. Without concepts, I have a blank canvas. This can be appropriate sometimes, but other times it leads to wildly different implementations.
Even though each implementation of the file parsing was self contained, overall ease of maintainability suffered because there was no common theme, no unifying concept, no design.
It’s easy to put off concept design with the excuse of YAGNI. We waited until we needed the feature of recording the parsing outcome before we went down the road of designing the concept. Unfortunately, that left quite a bit of rework and redesign, as we had so many different ways of parsing files.
Instead, recognizing the concept of parsing files early would have made subsequent file parsing easier to understand. We wouldn’t have had to think about what pieces we needed to invent before putting together the puzzle. The outlines would have already been there, and we would have just needed to fill in the blanks.
I’m all for starting with blank canvases. But on a regular basis, introspection is needed to search for common concepts to build upon. Otherwise, we lead ourselves into wildly heterogeneous architecture.
A popular cause the Agile folks like to rally against is the idea of a Big Design Up Front (BDUF). But much like Waterfall, the people doing BDUF will hardly admit that it’s BDUF that they’re doing. Instead, you’re much more likely to get a bevy of really convincing cautionary tales of what might happen if you don’t do a certain design. With enough experience under their belt, the architect slips into a “Just-In-Case” design mode, where the myriad of pain points encountered in the past leave a design that’s hardened against every possible difficulty that might come up.
Instead of a well architected design, you’ll have one that’s over-abstracted, over-engineered, over-complicated and very difficult for the next developer to come on board and understand. But there’s an alternative approach. The better choice instead of BDUF is a collection of design and architecture techniques that both delay decisions until they’re absolutely necessary, and refactoring practices to evolve our design when assumptions are proved incorrect.
YAGNI stands for “You Aren’t Gonna Need It”. It’s a generally philosophy of waiting until code actually needs to be written before writing it. Instead of guessing or projecting on future needs, abstractions or features, the need must present itself in a concrete manner. This idea can be abused as a way of dismissing true needs for abstractions, so it can take some trust and negotiation in a team when disagreements arise on whether a need has arisen.
The benefit of YAGNI is that you don’t develop features that you might not need. If you only design based on needs currently evident in the codebase, you’ll never fall into the trap of over-design. After a design has changed passed the YAGNI tipping point, it can be tempting to castigate yourself, asking “why didn’t we do that sooner”? Because you didn’t need it sooner. The trick is to find the right tipping point to guide the next step in design.
Another common over-engineering technique is refactoring areas that may be ugly, but don’t actually present any problems. The antidote for refactorbation and speculative design is pain-driven development. The concept is simple: if it doesn’t hurt, it doesn’t need fixing. If it hurts, fix it! The bottom line is that not all areas of your application will have the same sophistication and attention to detail in its design. But that’s normal, every project works under constraints and you have to focus your time on the most critical areas.
The most critical areas tend to be the ones that change the most. In Feathers’ Legacy Code book, he recommends refactoring code only when it requires change. If we only refactor when code needs to change, it naturally leads to the most-changed areas having the best design. If the most-changed areas are where the most features are added, it tends to follow that the design needs to be more robust in those areas. By paying attention to pain, we can gauge how much time and effort we need to address in changing a design. A design may not be ideal, but if it’s not causing much pain, there’s not much ROI on “fixing” the problem.
If you view pain as a cost or a debt, then a design improvement must be measured on how much pain the fix solves, rather than how much better it makes us feel or lets us flex our intellectual muscles. For long-lasting success, it’s important that we spend our time improving areas that will have the most long-term impact. Paying attention to pain is a great technique for doing so.
Sometimes a pain can be so slight, but so endemic, that you become numb to pain. An alternative solution doesn’t seem like it will be that much better, since you’ve become adjusted to the more difficult design or tool. This tends to close your mind to opportunities that the new design or tool provides. For example, we used the HBM NHibernate mapping files for a long time because we were all quite familiar with using them, and didn’t cause that much pain. However, Fluent NHibernate opened new doors that our numbness to the HBM pain was actually an opportunity cost because we couldn’t take advantage of things like conventions to drastically reduce the amount of manual mapping code we needed.
So while the current design may not incur too much pain, we still have to be aware of the opportunities alternative designs might afford us.
Another big issue I’ve seen with long-lived codebases is that although good design concepts are introduced, they are sparsely, almost randomly applied to the outside observer. Improving the architecture of a system is good, but leaving a trail of decaying, older designs still in the codebase can cripple further innovation. When we encounter a system-wide pain, we don’t just fix it in one or two spots. Instead, we estimate the cost to apply the fix system-wide. Only once we’ve applied the design system-wide will we try to apply the next iteration in our design.
The benefits to this approach are several. With several older designs in our system in varying degrees in the evolution of our system, it becomes harder and harder to determine what the “right” design is. If we have a system-wide concept that we only improve one spot at a time, we wind up with a looooong trail of various designs, and it becomes that much harder to find what the next step is. When it comes to system-wide architecture and design, the more examples we have of the current design, the better our next choice will be.
I could go on forever on other techniques, such as DRY, Responsibility-Driven Design, Convention over Configuration, Last Responsible Moment, Simplest Thing that Could Possibly Work and more. Since some decisions are not easily reversible, we would like to wait until we have as much information as possible before making a long-lasting, costly to reverse decision. When it comes to choosing between something like Rails or ASP.NET MVC, it would be only responsible to ask as many questions as possible before we make our final decision. Or choosing between Active Record or the Domain Model pattern. Are we in that 5% slice that actually needs DDD? Or is it just 5% of our current application that might require DDD? These are tough questions to answer, and there’s not always all the information you need to know where to go.
In my experience, I’ve found that choosing the simplest thing that could possibly work is a great starting point, and applying iterative design as we go tends to produce the best final result.
While prepping for the Headspring MVC Boot Camp last week, I had a couple of choices for getting the examples project up and going. I wanted the examples to use an actual domain model, a real IoC tool, and a real ORM underneath the covers. Getting these pieces up and going isn’t necessarily a trivial affair, and time spent doing that is time taken away from preparing the other parts of the course. So I had two options: The first option would be to take an existing application and pare away the pieces I don’t need. The second option would be to bite the bullet and just start from scratch.
So what did I wind up doing? (Drumroll please) Start from scratch.
And it was sooooooo liberating. With our existing applications, there’s always one piece or another that doesn’t fit quite right, but you don’t change because “it’s how things are done” or “it already works”. That’s great for a lot of solutions, but I’m now seeing is a poor choice for micro-design solutions. For example, I don’t really want to design templated HTML builders for every project, nor do I want to design an object-object mapper every time either. Those types of pieces can be abstracted into mini-frameworks to consume.
But then you have things like “how to use NHibernate”, the default Repository design, the “right way” to read from files and so on. For those pieces, I just tossed out everything and started over. As in, let’s completely re-evaluate how I use these 3rd-party libraries and how I’ve solved these problems in the past. I wound up with a completely different design than I’ve used before, and one that I like better than ones I’ve used in the past. Had I merely incrementally improved my existing approach, I wouldn’t have arrived at the point I used in the class.
Incremental design doesn’t just mean making small changes to one class. Incremental design isn’t merely incremental change. Sometimes, you just need to throw everything out, build on what you’ve learned, and take a clean approach to get to a better final design.
Existing code is baggage, be careful how much you pull from previous projects and existing work. While it can be tempting to try and set a high starting point, it may set you back in the long run. On our current project, we held ourselves back for quite a while by sticking with designs we did not consciously choose in the first place. Once we got over our fear of change, we landed in a much better place than where we started.
However liberating this was, it’s still not one I’d do on every project. For smaller projects, it’s great to have an existing infrastructure in place so you can start delivering value on day 1. But for longer-term projects, where only innovation will keep your velocity from stagnating or even dropping over time, we found that only by throwing out our existing prejudices and assumptions could we have the right state of mind to truly embrace change.
Surprise surprise, but Uncle Bob got the twitterverse all riled up with another opinionated post, “Dependency Injection Inversion”. His basic advice from the post on DI tools is:
I think these frameworks are great tools. But I also think you should carefully restrict how and where you use them.
Couldn’t agree more! Every call to Container.GetInstance should be carefully, carefully thought out. In fact, our goal should be to reduce the number of calls to the container to hopefully exactly one. But a slight problem with the example he gives to show that DI is not necessary in all cases: It’s so simplistic that it no longer represents any real-world code using DI.
Uncle Bob also prefers hand-rolled mocks, which may be an artifact of the crappiness of Java mocking tools, which in turn is a reflection of how far behind Java the language is behind C#. He also points out:
I don’t want lots of concrete [Container] dependencies scattered through my code.
Couldn’t agree more! That’s why we limit the container calls in our application as much as possible. If I have a bunch of container calls in my code, I’m doing something wrong. When I read his post, I thought that they must have been looking at the MVC framework source code, one which was built with IoC hooks, but not with the Dependency Inversion Principle in mind. If I have to tell MVC that I’m using an IoC tool, it takes quite a bit of work to configure all of the places where things get instantiated manually to work. I often have to write new classes just to support IoC. Whether he meant to or not, Uncle Bob’s entire post was a straw man argument. He argues against bad use of IoC, but never shows good use of IoC. Which may mean that he’s never built or seen an application built well with IoC.
If you want to see an application written with DIP, IoC and Dependency Injection in mind, check out FubuMVC. One of its core principles from the get-go was “turtles, all the way down”, basically meaning that a container will be used, and you cannot use the application without an IoC tool. Guess what? The design is much cleaner for it.
If I want to use an IoC tool in my application, this is something that should be configured once and only once. If I have to plug a bunch of hooks just to say, “Yes, I am using an IoC tool”, then I haven’t really improved anything for the end user, have I? Instead, I’ve made my application favor configuration over conventions, forcing more coding and more moving parts to figure out, making it more difficult to configure as needed.
So while it’s a great read and has a lot of great points, all that post showed me was that poor use of DI leads to a poor DI experience.
In my recent adventures with controller-less actions, and trying to solve the issue of the crazy CRUDController mess I had put myself (and our team/project) into. While some gravitate towards the Singleton pattern to abuse after they learn the GoF patterns, that wasn’t the case for me. Instead, I fell in love with the Template Pattern. But there’s a problem with the Template pattern as the golden hammer for every incidence of duplication we find in our application. The Template Method favors inheritance over composition.
Template method has its place, but that place is for procedural, algorithmic duplication. If you jump to Template method, you reinforce a procedural, rather than Object-Oriented design. Instead of template method, I’ve forced myself to look at the more OO patterns (though I haven’t gone as far as join the Anti-If Campaign) These patterns include:
All of these lead towards a more OO approach, which I have surprised myself lately on how far I really have to go in that area. What’s held me back is my knee-jerk reaction to duplication to put in a Template pattern. This pattern has its place, but like any over-applied pattern, can stink up your design and architecture when you push it into places it shouldn’t go.
