(This part III of a series -see Part I, Part II)

We need to face it, there are no absolute truths when it comes to software architecture( I guess that's part of the reason the term always looks so fuzzy. ) Should we use REST? it depends. Should we use OR/M or direct database access? it depends. Sometimes even a big ball of mud can be a good option. The good news is that we can always answer "It depends" to any architectural question and always be correct. The bad news is that it is our role to figure out what does it depend on and come up with a viable trade off.

The fact everything is a tradeoff doesn't mean that there aren't any cases where the trade-off is simple. E.g. if you just have to build a couple of data entry screens and a simple database you probably shouldn't spend a couple of months evaluating your options, just write the damn thing in your spare time. Nevertheless, just hacking it can mean it wouldn't be extensible, it that's what you needed then cool. if not, maybe you should have considered that.

This is one of the reasons I don't like the term "best practices" - it sends out a "you don't have to think anymore" message which is oh-so-incorrect. Patterns on the other hand, don't send  out this message, as they also include discussion on where to use them as well as their limitations and pointers to other patterns. Unless of course, patterns are seen as an end-goal rather than a means, in which case, they look pretty close to "best practices"

To sum this post - everything is a tradeoff. The most important bit here is to keep that in mind, even when the choices seems obvious. Awareness  is the key to better decisions.

Powered by ScribeFire.

 

As I was going through my RSS list (which I publish as a linkblog) I noticed Reginald Braithwaite's post on the metaobject protocol. Reginald says

The bottom line: Languages like Java give you an object protocol: There is one way to do things with objects and classes and interfaces, period. Anything else, like adding generics or annotations, must be done outside of the language, it must be done by the creators of the language.

Languages like Common Lisp and Smalltalk give you a meta object protocol: You can decide for yourself how to do things with objects, classes, interfaces, generic functions, whatever you want. You don’t need to wait for a committee to try something different.

What's nice is that a couple of posts later I noticed a link by Harry Pierson to an attempt by Jeff Moser to port OMeta to the .NET world and to enable the very "meta object protocol" Reginald mentios. Here are the opening bits from Jeff's post (go read the rest):

What if programming language implementations were object-oriented? What if you wanted to change one itty-bitty part of your favorite language? Say you wanted to add an exponentiation operator (^^). How hard would it be?

Wouldn't it be nice if you could "subclass" a language like C# and then add 5 lines of code to make it work and then use it? What if you could add Ruby-like method_missing support in 20 lines?

What if you could conceive of a new language and start experimenting with it in production code in an hour? What if it leveraged your knowledge of existing frameworks like .NET?

Also check out the code Jeff posted so far (on CodePlex)

 

I saw a post (via Dzone) by Keith Elder called  "How to not screw up your application object model - Don't go all OOP on me!"
Its a long read, so I'll give you the skinny. Keith tries to make his point by an example. He designs an elaborate initial object model for a veterinary clinic (IPet, pet, cat, dog, spaniel, owner, address, work address ..), then takes the search screen from a real vet. app, maps the model to the information needed for that screen and explain that the initial model sucks and that that particular screen needs two UI objects (search and searchresults) that would better handle the problem.
He sums the post with

"The big takeaway from this litany of verbal spillage is when you design
applications you have to know how the end-user is going to use the
application and what the business requirements are.  Sitting down and
building an elaborate object model, while really fun and educational,
it is a big waste of time.  Don't do it.  Sit down and ask questions so
the behavior and business requirements can be accounted for.  Whether
you are using CSLA.Net or not the same thoughts must go into building
the application.  I hope this helps cements these concepts because I
know there is a large amount of disconnect with this out there among
developers who think everything is a perfect object graph like they
learned it in school. "

There's nothing really wrong in the conclusion on its own but in the light of the post itself there are few things to notice

1. The "elaborate object model" Keith talks about has nothing to do with OOP.
   
2. Keith just drafts an "elaborate object model" but fails to note that it is a "Problem Domain Object Model" (PDOM) and a bad one at that (since it wasn't built with a domain expert involvement). The role of a PDOM is to form an initial working vocabulary with domain experts. It is usually a big mistake to use it as a foundation for development
3. Even as a PDOM and even considering it was built for the sake of argument, it still doesn't make sense to create a class for each type of dog. When you create a sub-class you need to have a new behavior not a different value in a property.
   
4. The example of the client side objects which do not match business side objects doesn't prove anything. client object (esp. search dialogs) tend to cross or aggregate domain boundaries. You can still have a meaningful OO domain model running in a backend.
5. The fact that the search screen aggregates data from multiple domain object doesn't mean that the other processes and screen don't use that but from the application type at hand I would bet that most of the screens only deal with CRUD which brings me to another point:
   
6. Another consideration Keith fails to mention is the type of application at hand. For the example given. It is probably a viable option to develop the whole thing as an Access application with embedded SQL in the controls. no OO and and no nothing.
7. In any event. It seems to me Keith doesn't provide enough requirements to justify either of the solutions

In summary, yes it is completely wasteful to draw an off-hand solution that doesn't support any concrete requirements but that's an absolute truth whether you are using OO or no.

Edit 5/5: After looking at Evan Hoff's post on this. I understand that the point is  better delivered if you can also see the object model in question. So here it is:

Figure 1: Poor OO model - don't use

 

Among the reactions I got for my previous post on the Singleton pattern in .NET were a couple that talked about the design rationale behind the solution I posted:

Adi Avnit posted on the risk of using a "generic singleton":

"However, there is one possible pitfall to this approach, as it makes this code possible:
Singleton obj = Singleton.Instance;
MyClass obj2 = new MyClass();

While I personally like the idea of having the freedom to use the same class in two different ways throughout the application, I know some people like their Singletons - well, single.
On the other hand, if you write a class from the start as a Singleton this is not an issue.

There is an inherit risk in decoupling a class from it's expected behavior, so take this into consideration before using this pattern."

And Cade Roux wrote in a comment:

"don't believe discoverability was a motivation for Singleton. The purpose was to ensure only one instance existed (a print spooler, a file system). You can get discoverability through the technique you explain but the primary purpose of singleton was a pattern of prohibition - to stop a programmer doing something they shouldn't do by accident by enforcing the primary goals of single point of access.

It is more than a global, and the OP, while comparing them to globals, does not acknowledge that this solves many of the problems which made "global" a 4-letter word. Knee-jerk reaction against globals is not healthy once you've mitigated their drawbacks.

It is true that usage of singleton can be misguided and cause coupling - which is why you need to ensure that the usage of the pattern matches the motivation in the first place.

This is a key point of design patterns which Alexander should have made clear in http://en.wikipedia.org/wiki/N...is_of_Form - the pattern is a result of a resolution of system of forces. It only satisfies that system - moving it to another different system will result in poor fitness. "

I guess that's a good opportunity  to talk about design issues regarding Design Patterns in general and the GoF ones in particular.

Aristotle Pagaltzis noted (in a comment on Cedric's blog) that

"design patterns are a sign of a deficiency of a language for the purpose that the design pattern addresses. In other words, the Visitor pattern used in Java points to the fact that Java is deficient in terms of list processing: the `map` and `filter` constructs need to be emulated with lengthy OO incantations."

In other words Concrete patterns* (like GoF's) are tied to implementation which means that the implementation language is  part of their "context". Thus when you come to apply a pattern in a different language you need to consider the language in use and make sure that

1. The pattern is needed
2. The solution in the pattern is the best one for the language.
   

Let's look at a few examples.
If we take the Visitor pattern mentioned above - The intent of the visitor pattern is to "represent an operation to be performed on the elements of an object structure without changing the classes of the elements". In .NET 3.5 you can do that with the use of LINQ (to find relevant items) and extension methods (to add external functionality) - same intent, completely different implementation.

In the case of the Singleton pattern (mentioned in the previous post). The intent is to "Ensure a class only has one instance, and provide a global point of access to it". An implementation using templates (or generics) is superior since it provides the same intent but also adds better compliance with the Single Responsibility Principle. While it is possible to create the class as a non-singleton on the one hand it solves the multi-threading issue in one place preventing both duplication of code and mistakes (again this can be especially important in non-forgiving environments like C++). It also lets you (not in the implementation I provided) add flexibility and e.g. let singletons die and (if needed) resurrect themselves etc. Note that even if you are not convinced that using generics is better, there's no doubt that using .NET's thread-safe static readonly variable (public static readonly MySingleton = new MySingleton();)  is a much simpler solution than the GoF one and again, answers the original intent with a different solution.

Another example would be the Template method pattern. The idea behind the template method is to support the Open closed principle (classes should be open for extension but closed for modification) or as the GoF defines the intent:

"Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template method lets subclasses redefine certain steps of an algorithm without changing the algorithm strucuture"

.NET 3.5 supports the notion of Generic Delegates so you can define functions as parameters and pass them along. This is a better implementation than the template method as now you don't have to subclass when you want to extend an algorithm you can just pass a function that matches the signature
e.g.

    class Program
    {
        static void Main(string[] args)
        {
            var alg = new Algorithm();
            alg.DoSomething(Funky);
        }

        static bool Funky(string value)
        {
            return value == "somevalue";
        }
    }


    class Algorithm
    {
        public void DoSomething(Func<string,bool>CanGoForward)
        {
            string someVariable = "somevalue";

            if (null!=CanGoForward && CanGoForward(someVariable))
            {
                //do one step
            }
            // whatever
        }
    }

You should note that this "refactoring to fit the language" phenomena isn't limited to GoF patterns (or .NET :)) , for instance you can consider the use (or lack thereof) of Dependency Injection in languages such as Ruby which I wrote about a few months ago.

To sum up - when you talk about design patterns you need to consider the implementation language, design pattern deal with deficiencies in the language and different languages have different deficiencies and may have better solutions to the problems solved by the patterns

---

* As opposed to architectural patterns which are more abstract and thus more reusable (e.g. Hohpe & Wolf Enterprise integration patterns, or Martin fowler Enterprise Architecture patterns)

 

Awhile ago I asked "who tests the test":

" One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?"

I would add that this is especially true considering that the same person that writes the code also writes its unit tests

One answer I had was making sure to keep the triad of tests->code->acceptance tests. so that each two test the third. This is especially good since, usually, the acceptance/integration tests are not written by the developer who wrote the code.

One answer I got was that tests are usually simple enough to be self evident and when tests contain complex constructs you should probably test them as well.

Basically, tests can have two types of problems

• Not testing the right thing
  
• Not testing the thing right

Code+unit tests +acceptance seem to take care of the first type of problems. The claim that tests are simple should take care of the second type of problems.
However I think that it isn't always true, especially when it comes to using the tests for regression. consider for example, the following two incidents I had  where the the problems were actually with the tests:

1. We have a component that interacts with resources in different URIs, so when resources wake up they register themselves with that component. Resources register two URIs -> one for control and one for interactions. We've added a business rule that both URIs should be on the same host (naturally a new test for that was also added but that not the point) suddenly a bunch of the old tests started failing. Sure enough, I took a look at the new code, found a bug, fixed it, but the tests were still in the  red. In the end it turned out one of the Uris that is registered during the tests had an extra "L" (locallhost instead of localhost) so the tests failed with the new rule - as they should..

2. After writing a piece of code to "just work" it had to be made multi-threaded. I needed to write a test that would make sure the component can handle multiple concurrent requests. Sure, I thought, I'll just run few or even all of the other tests in parallel and be done with it. The only problem was that the other tests were written to be isolated (i.e. new instance per test) running them in parallel had no effect what-so-ever.

What can we do to help us locate problems and bugs with tests?

• remeber that Test code is code - so as you write more tests you should also refine the tests design and refactor the code accordingly. For instance in the first example above, it was easier to diagnose the problem since the faulty setup was done by a single method used by alll the failing  tests (Single Responsibility Principle @ the method level).
• Test-First - One of the reasons test-first is beneficial is because you can actually see the test fail before you implement the code to make it work. This helps you make sure that the test actually tests the behavior you want. When the code is already in place it is harder to make sure if the test works because of the change or not. In the second example running all the tests in parallel everything was still green, which triggered me to write specific tests for checking for multi-threading issues.
  

What do you do?

 

While I personally don't like the Singleton pattern too much (it is essentially an OO mask for Global Variables, it makes it harder to unit test etc.), I still need to use them now and then. Anyway, I saw a post by Jack Altiere about "Using the Singleton Pattern" in .NET and since it presents an implementation that leaves a lot to be desired I decided to comment on that

For one, he presents a solution which locks every time the singleton is accessed (see below). The reason this is flawed is that singletons are instantiated once, but they are accessed a lot of times and with this implementation you pay every time you try to access the singleton

//bad implementation - don't use
//also note some initialization etc. is omitted for brevity
 public static Foo Instance
{
      get
       {
        // This lock allows thread safety.
             lock (_mutex)
             {
                   if (_instance == null)
                        _instance = new Foo();
                  return _instance;
            }
      }
   }

There's a better "standard" way to do that which basically means you check if the the instance is null, lock, check again (in case more than one thread got past the first check ) and only then instantiate (you'd also need to mark the instance volatile in this case)
.NET has a better way to create thread safe singletons by merely using a static readonly declaration as in:

public  sealed class singleton
{
        public static readonly MySigleton Instance = new MySingleton()
}

This implementation is not as lazy as the implementations above - but it is thread safe an efficient (you can get more laziness if you use a nested class as described here)

But that's not all- see, one of the problems of the Singleton pattern (besides the ones mentioned above) is that it is also a violation of the Single Responsibility Principle (SRP see OO primer). You can solve this problem if you use Generics and create something like C++'s template based Singleton class:

namespace ConsoleApplication5
{
    class Singleton<T> where T : new()
    {
        private static readonly T inst = new T();

        public static T Instance
        {
            get { return inst; }
        }
    }
}


and then use that simply by doing something like Singleton<MyClass>.Instance

[update]
Eran points out that Jack's code is not only inefficient but also not thread safe (see Eran's post on the subject) - basically you need to make the instance variable volatile just like in the double check scenario

Reshef points to the "Static Gateway Pattern" as an alternate  way to get the singleton effect. While we are talking about other possibilities, I should have probably mentioned that the usual way I handle situation where I need a single instance is instantiating one instance on the main/loader thread and then just using dependency injection... :)

 

If you recall what I currently work on is a type of a visual search engine. In a nutshell when we get a request (image) we allocate a bunch of algorithmic engines in a grid like manner to process the image  (e.g. try to perform OCR or whatever). As it happens, we are developing the different components using several different environments(*) - e.g. the control bits run on windows (.NET) and most algorithms run on Linux (mostly C++).
The need for easy cross-platform communications and extensibility, the resource nature of the solution and a few other tidbits led us to design our solution in a RESTful manner.

If you are a .NET developer/architect and wanted you may know that to implement a RESTful application in Windows Communication Foundation (WCF) you really have to jump through hoops.For instance  you have to go back to basics and use the HttpRequest and HttpResponse, handle the breakdown and parsing of URI hierarchies yourself not to mention  fight  with the  bindings .

Fortunetly this all changed with WCF 3.5. True, .Net doesn't have (to my knowledge anyway) something like RESTlets, but at least building REST on http is pretty straightforward.

Consider for example the following excerpt:

    [ServiceContract(Namespace = "http://paperlnx.Contracts/2007/12", Name = "ISessions")]
    public interface ISessions
    {
        [OperationContract]
        [WebGet(UriTemplate = "/Sessions/{sessionId}")]
        [ServiceKnownType(typeof(Atom10FeedFormatter))]
        SyndicationFeedFormatter ListSessionStatus(string sessionId);
	.
	.
	.

With these 6 lines of code you see the essence of  the .NET 3.5 REST goodies

1. Integrated support for HTTP verbs  - The sample above shows the support for GET. You can get the other verbs almost as easy with the WebInvoke Attribute. To do that simply specify the verb you want e.g.   [WebInvoke(Method = "PUT")] , [WebInvoke(Method="DELETE")] etc.


2. Support for URI templates -  In a way not too far from Joe Gregorio's IETF draft , WCF supports the notion of providing a way to describe families of URIs. This is done using the UriTemplate class. The WebGet and WebInvoke attributes also accept URI templates as variables and map the variable values (the curly brackets ones {}) to parameters of methods.
   
3. Support for standard  formats - you can use plain XML or you can choose to use RSS and ATOM syndication formats. In its most basic form you just create a syndicationfeed and format it to atom feed. Which is what we do for error messages:
   

   
           public static SyndicationFeedFormatter GenerateAtomError(string errormessage, string description,Uri location)
           {
               SyndicationFeed feed = new SyndicationFeed(errormessage, description, location);
               return new Atom10FeedFormatter(feed);
           }
   

   Naturally you can also add items and element extensions to all elements (e.g. the feed or items)
   

All in all, I am a happy camper :) After all, when you make an architectural decision, you always need to review it once you opted for an underlying technology. Even when a decision is right. The friction caused by a  technology which doesn't accommodate it well can both make your life miserable and make a good decision bad. .NET 3.5 with its newly added support for REST increases the architectural freedom and that's always a good thing

---

* Among other things, it helps us avoid the "Network is homogeneous" fallacy - but that's another story :)

 

Following the third part of my "Defining SOA" posts Udi Dahan left the following two questions:

How does this [layers - ARGO] play with two services talking with each other? One pubs
to the other's subs?The other requests to the first's response?
How valuable is the layered abstraction?

Considering Udi and me usually see things eye to eye. I guess that if I managed to get him confused, more clarification is warrant :) I'll do that in two posts, this one which will explain the concept of layers and the next which will explain why it is paramount  to SOA (and answer the two questions)

Usually when I review an architecture one of the first (sometimes the only) architectural artifact I am shown is a "layered diagram" of the architecture e.g. something like the following:

These sort of half layers/half block diagrams with or without the common 3 layers (which also appear in the diagram above) of "UI" "Business" and "Data" give the whole idea of layers a bad name

The key differentiator between layers and just a bunch of blocks is the limitations on the allowed communication paths between the components (layers vs. blocks). In the previous post I quoted an old (2005) definition I had for layers where I said the following

"Typically a layered is allowed to call only the layer below it and be called only by the layer above it (but there are variants e.g. a layer can call to any layer below it; vertical layers that can call multiple layers; etc. -- all is fine as long as the layers communication paths are limited by some rules)."

Alas, I was too quick on the Copy/past and everything in the brackets (bold) is wrong - it should actually say - "but there's another variant where layers are allowed to call the layer above it or below it". The other variants (like  a layer that can all any below it) just muddy the water, makes it hard to distinguish between layers and regular components and thus make layers seem unimportant. consider the following diagram:


So, in the above diagram the relations are the Component D and Component C know each other. Component D is made of two layers (A and B). Note that a more proper representation should also explain the relations allowed between the layers i.e. is it unidirectional or bidirectional (unless there's a common convention in the project)

Why is the distinction between layers and other type of components important? because Layering gives you some benefits which "just components" don't:
1. Layers allow information hiding. Since we don't know the inner working of what's beyond the layer
2. Layers allow separation  - Things beyond the immediate layer  are hidden from each other. This means that things which are beyond the layers are loosely coupled in a way that allows for  flexibility and the addition of capabilities. for instance adding a firewall between your computer and the internet.
3. Layers allows changing the abstraction level - since layers are hierarchical in nature, moving through the layer "stack" you can increase or decrease the level of abstraction you use. This allows expressing complex ideas with simple building blocks. The best known example for this is the TCP/IP stack moving from an abstraction level close to the hardware of the network interface layer to the application level protocols such as HTTP


On the downsides - layers hurt performance by adding latency. Also too many layers introduce added complexity to the overall solution (e.g. it is harder to debug).

It it interesting to note that Interfaces are in fact leaky layer abstractions (vs. for example SOA contracts which are not leaky) - since when you use an interface you still need to instantiate the object which is otherwise hidden behind the "layer" (interface). This is basically the reason we want something like dependency injection (DI) - to help make the abstraction complete and why languages like Ruby where the contract abstraction is complete - you don't need DI (I discussed Ruby and DI in another post)

Another issue which I mentioned here is the difference between logical layers and physical (or potentially  physical) layers. I usually call the first kind layers and the latter tiers. logical layers are local and can assume a lot about their neighboring layers. Tiers or physical layers can be distributed, which carries a lot of implications (something I discussed here recently in relation to MS Volta)





 

Wes Dyer, one of the principal people behind the Volta tier-splitting was kind enough to leave a comment on my previous post. Here is one quote from that comment

"I do want to clear up a few things about Volta that we apparently
didn't make clear enough. We do not believe that you can develop an
application as if it will run on a single tier and then just sprinkle a
few custom attributes here and there and be done with it. More than
anything else, programmers need brains. Volta does not claim that
programmer brains can be checked at the door. When the programmer wants
to divide the application across a particular boundary then things like
network latency, new failure modes, concurrency, etc. need to be
considered at that boundary. What Volta does is make expressing the
transition between boundaries easier. It reduces the accidental
complexity of writing all of the boilerplate code to express the
programmer's intention. This allows the programmer to focus on the
essential complexity of his problem domain -- figuring out how to write
effective code for that particular tier boundary."

For one, it is good to hear that the architects behind Volta have a deeper understanding of distributed computing challenges - even if the first version doesn't seem to show it. I didn't use MS Volta enough to say that indeed the problem is not with the inherent capabilities and design  (let's just take Wes words for that). I am also not against saving the boilerplate code (though I would personally favor libraries rather than code generation and try to keep the "generation" gap to a minimum (i.e. the amount of generated code or the distance between the abstraction and the next concrete level)).
Lastly I am also in favor of trusting developers have brains and that it is ok to provide developer "sharp tools". So if all is good, where's the problem?

The problem is that you have to make it "easy to do the right thing" and provide the means to do the more complicated, less safe things. When I teach my young kids (and I can objectively say they are very smart :) ) to use a knife, I don't hand them the razor sharp, butcher knife first. They start with the plastic ones. When they've mastered that they can try something more dangerous. When you allow distributing something at a flick of an attribute and put marketing blurb on the site that makes it compelling to use it you create the wrong impression to the less experienced folks.

In one project which architecture I reviewed, the (very talented) architect/developer designed his own distributed transactions system (he shouldn't have been doing that in the first place - but that's for another post). When designing this he built in a lot ways to control the transaction behavior including the option to allow transaction participants to prevent rollback without failing the transaction.  Circumventing the transaction was as easy as making it work properly. Are there edge cases where you may need to have one participant violate the ACIDness of the transaction ? I guess  so - but that is not the general rule. Most of the time when you commit a transaction you expect it to be ACID. if for some reason it didn't behave that way - you want to know about it, even if it didn't actually failed. When you don't make it easy to do the right you get unexpected behaviors, you get hard to explain bugs, you get slow performance etc..
Developers using tools, smart as they may be, don't usually go and read all the source code of the tool/framework they are using (maybe they should). If two options are just as easy to use, it seems safe to assume they are just equally right. Things which are unsafe should be clearly marked as such to prevent mis-use by unexperienced users. This is especially true for tools that are targeted for common use and to ease the life of inexperienced developers

 

I got a couple of emails with questions reagarding  my previous post on Volta. So here's another go at explaining why dynamic-tiering is not a good move - this time in technicolor.

Let's start with a simple illustration. The diagram below represents a typical local component(A) in its environment. As a component that works locally, it has access to other local components which it interacts with. These can be objects it created by itself or objects that where injected to it. The likely design for local components is to have a chatty interaction - After all objects can talk to instances of other objects quite easily.

 

Jeff Atwood (of Coding Horror) writes about Brian Foote and Joseph Yoder's "Big Ball of Mud" paper but completely misses the ball (pardon the pun). Jeff says that the paper describes

"classic architectural mistakes in software development."

while the paper describes the exact opposite. with the exception of the "Big Ball of Mud" pattern itself which can be seen as an anti-pattern (as,by the way, the authors explain) the rest contain exceptionally good advice on how to prevent problems. Let's look at them one by one

• Throwaway Code -  when you just want to make sure something works (prototype, spike etc.) or even a quick fix - you don't write elaborate and heavy code - instead you write something simple to solve the problem. Throwaway code only becomes a problem if you don't actually throw it away...
• Piecemeal Growth - The waterfallish "big plan" in advance has failed numerous times - so instead we should build incrementally, don't over plan or over design, build things into libraries only when it is proven to be recurrent problem, refactor etc.
• Keep it working - The pattern is basically about building often and keeping iterations short, having tests that always prove your code still works as you make changes to it
• Shearing Layers - This pattern is about identifying related components. We all know change is inevitable - however things don't change at the same rate. For instance we can probably get interfaces to be a little more stable than the implementations behind them. Frameworks evolve at a different rate than the business they support etc. The pattern is about dividing the system so that things that change at similar rates are together i.e. in the same package,  CSCI  or whatever you use to partition your system.
• Sweeping It Under the Rug - When you find that you do have badly designed or badly implemented code - the pattern suggest you localize and isolate it to a fixed area (e.g. behind a facade) to prevent it from propagating into the rest of the system.
• Reconstruction - This pattern is about understanding when the code base is so bad that it is better to start-over and rebuild from scratch rather then try to patch it. In many  way reconstruction is not a good or easy thing to do however the point here is to identify when it  is the lesser evil. 

Oh, and what about "Big Ball of Mud" itself - essentially from the architectural perspective it is indeed an anti-pattern - you have something that is not very maintainable, hard to understand and what not. However we should keep in mind that the idea behind designing an architecture is not to get the best, cleanest architecture. The idea is to make the right tradeoffs so that you'd be able to deliver the best overall solution under the constraints you face (budget, time, the team's skill and what not). If you're biggest constraint is time-to-market and your architect spends all eternity planning the 8th wonder of the world, fire his ass. I'd rather live with a Big Ball of Mud for the first release than not ever make a release...
Big Ball of Mud can be considered a pattern for pragmatic approach to building working software. This is probably not acceptable in the long term  - but it can be a good option for short term if you are aware that that's what you are doing and willing to treat what you get as "Throwaway code".

So again, except maybe big ball of mud, these patterns are not "project pathologies" as Jeff calls them - these are very good ways to keep delivering business value and working softwares

 

Well it seems setting up a new company can keep you busy at times - which is my official excuse :) for the quiet last week. Hopefully I'll have a little more free time this week

Note: I am talking in this post about roles and capabilities. i.e. when I say architect I mean someone who has the capacity to be an architect (e.g. as demonstrated in the architect training suggestion I made) and takes that role. On a specific project you may have a person that performs multiple roles such as an architect and project manger or an architect and developer while  other project warrant one or more full time architects.

Not all projects need architects. There, I've said it. Not all projects need architects and I am not talking here just about trivial projects. There are cases (maybe even many cases) where you can get by with what I call "off-the-shelf" architecture - maybe with a few adjustments that any master developer (i.e. seasoned and experienced developer) can handle. For instance a lot of web-sites can do pretty well by using Model-View-Controller (or a derivative of that) along with a simple O/R mapper such as active-record. In fact a lot of them do when they use a framework like Rails that made these architectural choices for them*. Another example is the vanilla 3-tier architecture provided by vendors (such as this one by Microsoft). Yes, when you take something off-the-shelf the result might not be optimal but that doesn't mean it isn't sufficient. You just have to be aware for the tradeoffs...

Another point is that "design" is not an exclusive architect thing. A developer is not a good developer unless she also known about  proper design. mastery of technical details of the language without understanding the wider context of design will just help you code a lot of crap faster.

Having said that we need to consider a few issues - When do you need architects? What do they Do? What's their relation and interaction with the developers ?

When do you need architects?

It is sometime a fine line between a project that can get by with an "off-the-shelf" architecture and one that needs an architect. It would be nice if we could have something like a litmus test that would tell us if architects are needed or not. I don't have one. The closest thing I could get to this is something I call the SCLR test (pronounced scaler). SCLR stands for Size, Complexity & Limited  Resources.

• Size - well if you are going to have something  estimated at 1000 man-years or dozens of teams. I think it is pretty obvious that you can't just use something  which isn't made to fit. If anything there's a need to divide the work between the teams in a way that would make sense so that you wouldn't get a big-ball of mud. There's also a lot of need to coordinate the efforts and keep the big-picture inline. Personally I think that it doesn't  have to be a huge project to warrent some architects involvement. Since as Fred Brooks notes the number of interactions grows exponentially as we add more people.  In my experience trouble starts even with more modest numbers - more than 4 or even 3 different teams working concurrently is probably a good number to start thinking  about architects
• Complexity - There are many signs for complexity in a project. The vision statement can provide a hint. "Let's design the software to support the next mars mission", "best CRM platform ever"  - an ambitions project will not make-do with "average" architecture. Size (which  I already mentioned) is also a sign of complexity, while previously I talked about size of the project, the size of data, users etc. is also relevant when we're thinking about complexity . A lot of external interfaces is another sign. Integration doesn't seem very complicated, until you actually try to pull it off. When you have to do a lot of that in a project that's complex. And there are many other signs
• Limited Resources - Naturally every project has limited resources, but limited resources should be considered as a sign for architect involvement if the resources are extreme. When resources are extremely limited the tradeoffs that have to be done are more meaningful, which is why wed want people who can help with that (i.e. architects). For instance in a projects I worked on in the past we had a lot of availability and performance requirements on one hand but only so many "U"s in the rack and even limited electricity to make all this magic happen. This turned something which otherwise was a relatively standard IT project into something a lot more challenging.
  

Assuming I manage to convince you that some projects can't just choose one of the available blue-prints and need some more work - the next step would be to convince you that architects are better suited to solve this than developers. I'll try to do that in the next post on this series where I'll explain what (I think) architects do and their place in the development team

---

* Rails has more than just MVC and Active-Record but that isn't an important point for this discussion

 

A side effect of my decision not to become an independent consultant at this time means that I have to shelve some of the projects I was considering. One of these projects was to create a training program for software architects which I was discussing with a couple of training centers here in Israel.
Since It seems I am not going to promote it, I thought I'd share what I think a training program for .NET/Java architects should look like in the hope that someone would find it useful and do promote it (or parts of it)

Soft Skills
The way I see it architects needs a bunch of soft skills to be able to perform their roles.
Here is the list I identified in the past (by the way, I began a series of posts on each of these skills and never got to finish it - maybe it is time that I will :) )

• Leadership. Influencing others to accomplish tasks and following your guidance
  
• System thinking. Understand decisions and constrains in the
  wide scope pertaining to whole of the solution at hand. This includes
  the ability to abstract problems.
  
• Strategic thinking. Understanding decisions and constrains and their alighments to the overall business of the company.
  
• Organizational politics. Understand the environment you operate in and how it influences you.
  
• Communications. Making sure you get your point across.
  
• Human relations. Understand the "people" aspects or human
  factors and dynamics. This includes things like negotiation, pragmatism
  etc
  

I am not sure if you can teach all of them, but few courses that can help (in my opinion) include:

• Negotiation/Persuation - Anything based on "Getting to yes","Getting past no" and "The power of a positive no" should be fine.

• Presentation Skills - While getting the architecture and technology right is what matters, if you can't explain it to the different stakeholders you're toast.

• Strategic Planning - This has to do with the vision thing I expect architects to manifest. Note that having a vision should not be confused with future-proofing a solution. future-proof means excess work not needed. Having a vision is knowing where you want to end - it can still be perfectly valid to completely re-write your applications along the way

Project Management
While the architect is not the project manager (mostly anyway), I think understanding the constraints coming from the project management point of view is very important. Since most environments call for a mix of agile and formal disciplines (hey, you've got to be pragmatic). I would train architect both in SCRUM and RUP (or some other formal methodology)

Also while not all environments needs this I would give an 2 days overview of important standards. The first would be IEEE 1471, which defines a standard for documenting software architectures. I would also teach ISO 90003 and CMMI.

It should be noted that the ISO 90003 is much better than the previous incarnation (ISO 9003) as it basically lets you define what you want to do to cover the different areas. The standard just helps you make sure you think about the various parts of project management (requirements, environment etc.). For instance I demonstrated how key areas of 90003 can be mapped to SCRUM to get it approved on my last project.


Languages, Design  and Patterns

I would want the .Net/Java architect training program to include at least 2 of the following languages:
Ruby, Scala, Erlang, F#, Python , Groovy ,OCaml
The reason for this is that these languages have different design goals than .NET and Java so learning them gives you additional perspectives and broaden your horizons for other ways of thinking (even if you don't use them in your project directly). You might have noticed that there's no .NET or Java training here. The reason for that is that's a prerequisite as far as I am concerned. You should master at least one

Object Oriented principles - hopefully aspiring architects already know this. However, I often see people who discovered some of the principles by themselves but haven't heard about all of them.
I am talking about principles such as Liskov Substitution  Principle , Open Closed Principle, Single Responsibility Principle , IoC containers, Don't Repeat Yourself  and YAGNI (I summarized my opinion on most of them in this paper)

The next step is to cover some design issues like Domain Driven Design, UI Design, Database modeling, Database alternatives l(after all the database is dead, right? )

Advanced design patterns - When most people hear the term design patterns they think about the GoF patterns. There are however literally hundreds of design patterns. Some of them are even worth learning :) . For instance there are patterns for concurrent and parallel systems like Proactor, Reactor Half-sync/Half-Async etc; Workflow patterns like Cancelling partial Join, Recovery Action etc; SOA patterns (ok, so I am still working on that :) )



Architecture Workshops
Another important part of the training, in my opinion, is to do some workshops and actually try to apply some of the material covered.

• Architecture Evaluation - workshop 2-3 days - It is probably worthwhile to delve a little on scenario based evaluation techniques such as LAAAM and ATAM. While I prefer evaluating architecture in code, the scenario based thinking is very valuable for eliciting architectural requirements
• Distributed Systems Architectures workshop - I'll expand on this in the next post

Lastly, there are also a few miscellaneous subjects like architect 101, the SPAMMED architecture framework , Agile architecture, Behavior Driven Design , common frameworks (though hopefully this would  not be needed ) like Spring/Spring.Net, Hibernate/NHibernate, iBatis  etc.


PS

Note that there are a few architect training programs available out there
One is offered by the Software Engineering Institute (SEI) and includes a 6 courses. SEI program seems to be focuses on formal sides of architecture as it includes courses on documenting software architecture and ATAM (You can see an old presentation I have on ATAM here)
Dana Bredemeyer  also offer architect training. Dana offers several workshops that cover the software architecture profession.
TOGAF (which is more of an enterprise architecture framework) offers both a certification and courses
Lastly, IASA is considering creating a software architect program and has a few courses in development
If you know any others I'd be happy to hear about them

 

You may have read about the guy who after spending 2 years on a Ruby on Rails project switched back to PHP. Without getting into the debate of whether Ruby on Rails is better than PHP or wether Ruby is overhyped (probably - but that doesn't mean it isn't any good either.By the way  it is the same with SOA, but I digress)

Reading his post I saw a few quotes that raised a red flag for me such as:

"But at every step, it seemed our needs clashed with Rails’ preferences. (Like trying to turn a train into a boat. It’s do-able with a lot of glue. But it’s damn hard. And certainly makes you ask why you’re really doing this.)"

and

#2 - OUR ENTIRE COMPANY’S STUFF WAS IN PHP: DON’T UNDERESTIMATE INTEGRATION
By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change..

and

Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables. I was always fighting against Rails and its migrations hiding my beloved SQL from me.

What these quotes say really is that this guy doesn't know about "Technology Mapping 101". Here is what I wrote about  technology mapping*  about 2 years ago (incidentally that's about the same time this guy started his Ruby adventure :) )

"Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible."

Every technology or framework has its own architecture. This architecture poses constrains and makes certain things easy (like using the ActiveRecord pattern in Rails) and certain things harder (like not using O/R Mapping ) so, for instance, on the case mentioned above a better technology mapping might have been RBatis (iBatis for Ruby/Rails) which lets you map SQL statements to objects. It is important to note (in Rails case) that one of the core tenets for Rails is preferring convention of configuration when you don't do that you have to work hard(er) - as you are working against the framework

Another problem with the technology mapping here was his point #2. It is a pity he only saw it in after the fact. It can be justifiable to switch everything but you've got to allow this change to occur iteratively. While I generally dislike the software architecture = building architecture metaphor, using it here does make sense: building software for an existing business (vs. greenfield development) is like building a new intersection. You have got to think about building all the detours that would allow the traffic (business) to continue to operate, sure it might run slower in the intermediate phase but it can't stop altogether.

So again, Once you have an architecture that fits your business, take a look at the technology options you have. try to choose the best fit. Whatever you choose - take a look at the implications of that technology and think about the tradeoffs you may find that you either have to adjust your architecture or change the technology altogether - if you don't you might find you waited 2 years of development effort or even more..

---

* Technology Mapping is one of the steps of a set of activities I identified as needed to make sure your have a solid architecture. The activities goes by the acronym SPAMMED and you can read about them more in this article and/or this presentation

 

Another great presentation at Architecture & Design world was Neal Ford's presentation on Domain Specific Languages (DSLs) . As the title suggests Neal gave examples both in static languages (Java) and Dynamic ones (Groovy, Ruby).

One interesting observation Neal made was that humans tend to create DSLs in real life whenever they (ok, we :)) have any non-trivial interaction or behavior. Neal gave sevaral examples such as the Starbuck's order taking ("Venti Iced Decaf with whip...") , musicians and a few others.

The next important point was contrasting ("classic") APIs and DSLs. The main difference is that the context is implicit and not repeated

A key  technique for building DSLs Neal mentioned was Fluent Interfaces. Fluent Interfaces means modeling the API so that lines of code are readable English-like sentences. The fluency comes from the easier readability by the interface user.

Fluent Interfaces, now that's a novel idea - what would a fluent interface look like, hmm, wait, I have an idea. Here are 3 samples that come to mind

DIVIDE x BY z GIVING y ROUNDED

INSPECT data REPLACING ALL "foo" BY "bar

READ someFile AT END SET eof TO TRUE

If you haven't guessed  the statements above  are in ...Cobol (by the way pardon the caps that's Cobol conventions..)
So ok, it isn't a new idea, but it is interesting to see it is making a comeback
Anyway one area where we see a lot of fluent interfaces emerging is configuration (mainly as an alternative to those lengthy XML files). For instance the following is an excerpt of configuring  Restlet components  (taken from my Edge Component pattern paper):

Builders.buildContainer()
            .addServer(Protocol.HTTP, portNumeber)
            .attachLog("Log Entry")
            .attachStatus(true, "webmaster@mysite.org", "http://www.mysite.org")
            .attachHost(portNumber)
            .attachRouter("/orders/[+")
            .attach("/getAll$", getAllRestlet).owner().start();
            .attach("/getLast$", getLastOrderRestlet).owner().start();


Note that this example also uses another fluent interface/DSL technique which is method chaining.

Dynamic languages make it even easier to write DSLs since they provide a lot of extension capabilities (see my previous post on OCP in Ruby), are less strict about types, allow reopening classes etc. 
Oneexample for a Ruby DSL is RSpec which is a framework to support Behavior Driven Development (BDD) in Ruby - The example below shows an excerpt for defining specifications for an eight-ball game

require 'eight_ball'

describe Eight_ball do
    before(:each) do
        @eight_ball=Eight_ball.new
    end
    .
    .
    .
    it "should lose if 8-ball sinked in pocket other than called" do
        [1,2,3,4,5,6,7].each ( | val | @eight_ball.sink(:player =>"Player1", :Ball=> val)
        @eight_ball.call(:player => "Player1", :pocket => :upper_left)
        @eight_ball.sink(:player => "Player1", :Ball => 8, :pocket =>:middle_left)
        @eight_ball.game_status.should == :ended
        @eight_ball.player_status("Player1").should == :lost
    end
end
 
By the way Joe Ocampo built a very nice port for rbehave (another Ruby BDD framework) to .NET 3.5 by extending NUnit which has a very Ruby-like syntax
      

Anyway, the DSLs demonstrated by Neal provide a very good example of the difference between dreaming big and actually doing stuff in the small. The counter example for that are "Software Factories". As I wrote here about a year and half ago

Software Factories is not a new idea  - see for example "Software reuse: From Library to Factory" by M. L. Griss  (published in 1993(!)) which talks about "Software Factories" and "Domain Specific kits": components, frameworks, glue languages etc.  The current Microsoft  incarnation of Software Factories takes a similar approach focusing on Domain Specific Languages, Frameworks but also adding important aspects like multiple viewpoints, patterns and designers. The idea is that  building on modern technologies, as well as learning from the mistakes from sister approaches to code generation (OMG's MDA, in case you are wondering) will enable us to build something that is useable.

Microsoft seems to be taking some steps in the right direction (GAT is probably the best example). Nevertheless there is still a long way to go before we can realize the dream of "factories" for vertical applications

Unlike small code based DSLs - the modeling based approaches of software factories, MDA etc. aim too high and thus provide much less value or suffer too much from the generation gap (the code generated is too generalized or far off from the actual need of the solution). Another problem with software factories/ MDA DSLs is the modeling (i.e. diagrams) - they say a picture is worth a thousand words. This is true if you treat models as sketches you can raise the level of abstraction by as much as you want and convey ideas with less clutter. However when you need to make the model very specific so it would allow code generation - you get to a stage where it is more convenient to do it in code and rely on generated or pre-built DSL or framework

Lastly you can  download Neal's presentation (in PDF form) from his site

 

A few weeks ago I wrote about Dependency Injection and how it doesn't really matter in Ruby. On one of the comments for this post Yoni said (in regard to extending classed - both in .Net 3.5 and Ruby):

"Regarding the issue of extending classes (and partial classes) I am very sorry it was ever invented. Developers should be encouraged to divide large classes by decomposing them into decoupled sub-components using design patterns and not by simply splitting them between files. Even though a feature is called "extending" a class it is in violation of the Open Closed Principle..."

that sounds like a serious charge :) So here's another round of  "Ruby does it better" the time featuring Bertrand Meyer's "Open Closed Principle" or OCP for short.
OCP was defined in 1988 in Bertrand's book "Object Oriented Software Construction"  as follows:

Modules should be both open (for extension and adaptation) and closed (to avoid modification that affect clients)

When we work with a language like C# or Java we have several ways to do that - here's what I had to say about it :

It is easy to see the benefit of having a class that answers this principle: When you need to add a requirement, instead of breaking dependent code (and tests) you just extend it somehow and everything is nice and dandy. Furthermore, violating OCP can result in Rigidity,Fragility, and Immobility.

But how do you do that? The obvious (and naïve) answer is inheritance. Every time something needs to change just add a sub-class. The parent class is not changes and voilà. However, if you add sub-classes all the time you'd get "lazy classes" or freeloaders--sub-classes without a real reason for existence not to mention a maintenance nightmare.

Thus, sub-classing is an option but we need to consider carefully where to apply it. Other (more practical ) OCP preserving steps include:

• Marking class fields as private


• Not adding getters/setters automatically. You may want to read "Why getter and setter methods are evil" by Allen Holub, and Martin Fowler's rebuttal/refinement in "Getter Eradicator".
  
• Inversion of Control strategies
  
  
  
  • TemplateMethod pattern
  
  
  • Strategy Pattern
  
  
  •  Anonymous delegates (in C# 2.0) and Closures in dynamic languages (also in C# 3.5)


• And don't forget to hide implementation details behind (stable) interfaces

The way I see it Ruby (and other dynamic languages for that matter) just allows us to extend this repertoire by adding a few other options such as:

Singleton methods (that's not a very good name - but the more appropriate name "instance methods" was already taken by another poor naming choice...) which are methods added to a specific instance of a class:

class Foo
  def bar
    puts "foo.bar"
  end
end
   
obj = Foo.new
obj2 = Foo.new
def obj.bar  # redefining bar just for obj
  puts "foo.newbar"
end
obj.bar   # prints foo.newbar
obj2.bar # prints foo.bar

Closures - I already mentioned closures for C# but it existed in dynamic lanaguages for a long time now. closures are sort of like injecting a procedure. Here's a quick sample:

class Foo
  def bar(myProc)   # accepts a callable object
    foobar = "foo.bar"
    myProc.call(foobar) # call the object with a parameter
  end
end
   
obj=Foo.new
printer=Proc.new {|msg| puts msg}

obj.bar(printer)

Meta programming - Ruby has a lot of ways to add and change classes. methods and whatnot. Again. here is a simple example:

class Foo
end
   
obj=Foo.new

# obj.bar - error
Foo.class_eval do           # this simplistic example can also use regular def
    define_method :bar do   # but class_eval can also evaluate strings to create
      puts "foo.bar"        # dynamic methods like setters etc.
    end
end

obj.bar

And there are a few other similar ways.

All of them  are not violating OCP!

OCP is kept since we do not alter the original class. Anyone using the original class not in our context (i.e. not in the modified context) will not be affected by the change. The interface of the class, in the sense that was originally defined is not changed either, and remains stable. Any client that uses the modified or the original class will not have to change it syntax because of the changes we've maid. This is also in-line with the "protected variation" way of looking at OCP :

"Identify points of predicted variation and create a stable interface around them."

While in C# your would maybe want to add a template method or a specialized interface in Ruby you can apply YAGNI and only change the behavior if the need arise

Lastly, we can also extend classes - but OCP lets us do that so again, we are OK.

Nevertheless, we should be careful not to violate Liskov Substitution Principle when we do all that, which I guess is relatively tempting to do if you using Ruby - but that's for another post ...

 

Ok now, that I got your attention, that it isn't dead yet - but we can see a whole class of applications (maybe a couple of classes) where the importance of the RDBMS as we know it today is great