I read this blog by Joel Spolsky on the “Duct Tape Programmer”, where he praises dropping any libraries or testing or proper coding in favor of getting something out there:

“He is the guy you want on your team building go-carts, because he has two favorite tools: duct tape and WD-40. And he will wield them elegantly even as your go-cart is careening down the hill at a mile a minute. This will happen while other programmers are still at the starting line arguing over whether to use titanium or some kind of space-age composite material that Boeing is using in the 787 Dreamliner.”

While I suspect the main reason Jo wrote the blog a little provocatively is to promote the book mentioned there (Hey, it worked, I am writing on that..), I guess some people will take this at face value so I’ll respond anyway.

Let’s start with the good stuff

• Focus on shipping software – Tools, pretty code, fancy frameworks, patterns etc. are no excuse for not shipping your product. 100% agree
• Overengineering is bad for your (project’s) health – We’ve been through that before – complex frameworks , big up front design, are bad all bad
• Don’t use a technology/framework just because it is there – if you don’t have a real problem then KISS (keep it simple stupid)
• Simple design is, well, less complex than complex design – so it is usually preferable

In a similar fashion to Joel’s post, I wrote that even the “Big Ball of Mud” which is an architectural nightmare can be considered a pattern for a pragmatic approach to building working software.

However, I also added an important caveat which is missing from Joel’s post:

“This is probably not acceptable in the long term  - but it can be a good option for short term if you are aware that that's what you are doing and willing to treat what you get as "Throwaway code".”

If you can’t afford to throwaway the code (or not sure you will be able to)  it is still ok to cut some corners in order to get things done. However you should keep in mind something Uncle bob said a few days ago : “A Mess is not a Technical Debt.”

Oh, and by the way, duct tape isn’t a good analogy anyway :) – in the first gulf war, the government in Israel recommended people to prepare their houses for Sadaam’s future missile attacks by duct-taping the windows and other openings. The only reason it actually did work was that there were hardly any missiles shot. So yeah duct tape may seem to be working and everything but you better not put that into a real test

 

I am currently implementing Reservation (a pattern for  “partial temporary commitment” when using Sagas).The  implementation calls for calling a few of the saga members asking them to reserve an asset. Since network calls can take some time I wanted to make each call on its own thread, collect any errors (failures to reserve or communication exceptions) and then have all the threads reconverge to the initiating thread  where any problems will be sorted out (e.g. by retrying the reservation). It basically looks something like the figure below:

As you can probably guess this type of synchronization is called Rendezvous (as the threads “meet”). It is also actually fairly easy to implement in .NET I usually do something along the lines of the code snippet below and things like Parallel.For in C# 4.0  will give a similar effect

However, that’s not why I am writing this post. It turns out I have a more complicated situation. In my setup the code of “Main Thread” (in the figure above) can be run by multiple different threads which are initiated by external events (i.e. I don’t control when they run) – What I needed to to is to make sure that if there are several threads that run in parallel they need to wait for the last of them to finish one section of the code before they can all move on. To explain this better consider the figure below:Thread 1 starts, sometime after that Thread 2 starts and then Thread 3 starts as well – Thread 1 and Thread 2 have to hold and wait until Thread 3 finished. Thread 4 starts beyond the Rendezvous so none of the other threads care about it. Had it started just before Thread 3 finished all the threads would have had to wait for it as well. Basically I need a similar effect to chords in polyphonic C#

I couldn’t find anything that .NET 3.5 for this - It is kind of what ManualResetEvent – however, it not quite that, since we need to wait for the event only when we’re done. Also we need to let other threads “know” we are running so that they’d wait for us. So indeed I wrapped a ManualResetEvent with a few methods to do just that and ended up with the following.

As long as all the threads have access to the same instance of the Rendezvous class they can synchronize their end times.

Rendezvous is a common pattern for workflow systems (also known as Join-And). It is the first time I had to do that in C# with uncontrolled/uncoordinated thread starts so I don’t know how useful it would be to you. However I thought it is also interesting reading so I published it anyway :)

 

Earlier today I read a post by Michael Feathers Called "10 Papers Every Developer Should Read (At  Least Twice). I knew some of the articles mentioned there and learnt about few interesting ones.I liked it so much,  I thought I'd compile a similar list for software architects - based on stuff I read over the years.

1. The Byzantine Generals Problem (1982) by Leslie Lamport, Robert Shostak and Marshall Pease - The problem with distributed consensus
2. Go To statements considered harmfull (1968) - by Edsger W. Dijkstra - Didn't you always want to know why ? :)
3. A Note on Distributed Computing (1994) - by Samuel C. Kendall, Jim Waldo, Ann Wollrath and Geoff Wyant - Also on Michael's list but it is one of the foundation papers on distributed computing
4. Big Ball of Mud (1999) - Brian Foote and Joseph Yoder - patterns or anti-patterns?
5. No Silver Bullet Essence and Accidents of Software Engineering (1987) - Frederick P. Brooks - On the limitations of Technology and Technological innovations.
6. The Open Closed Principle (1996) - Robert C. Martin (Uncle Bob) - The first in a series of articles on Object Oriented Principles (you remember the debate on SOLID...)
7. IEEE1471-2000 A recommended practice for architectural description of software intensive systems (2000) various- It is a standard and not a paper but it is the best foundation for describing a software architecture I know.
8. Harvest, Yield, and Scalable Tolerant Systems (1999) Armando Fox, Eric A. Brewer - That's where the CAP theorem was first defined
9. An Introduction to Software Architecture (1993) - David Garlan and Mary Shaw - one of the foundation articles of software architecture field (although based on earlier work by the two)
10. Who Needs an Architect? (2003) Martin Fowler - Do we or don't we?

I could come up with quite a few more articles not to mention books that aren't in this list. However these are definitely some of the most influential papers I read.

 

Returning from a short blogging vacation (both personal and work related) I found my RSS reader is getting filled with posts relating to "Uncle Bob vs. Joal & Jeff or the case of the SOLID principles (OO principles by various people including Uncle Bob himself - you can also read the "OO Primer" paper (pdf/ppt) I worte on these and few related principles).

Codeclimber has the complete timeline, but in essence, Unclue bob talked about the SOLID principles in a Hanselminutes podcast; Joel Spolsky commented that "quality doesn't matter that much" on his and Jeff's StackOverflow podcast, which was followed by a few blog posts on both side like Uncle Bob's "Open letter to Joel Spolsky and Jeff Atwood" and Jeff Atwood's "The Ferengi Programmer".

I don't know Uncle Bob personally (I've seen him talk a couple of times and I read his blog/papers), so I might be off here. However (to me) Uncle bob usually comes off  as  being too religious about following practices to the letter. e.g. see the debate  between Uncle Bob and Jim Coplien on TDD where Bob says "nowadays it is irresponsible for a developer to ship a line of code he has not executed in a unit test."

Nevertheless, I am fully with Unclue Bob on this one. I especially like the following qoute from his "Getting a SOLID start"

"The SOLID principles are not rules. They are not laws. They are not perfect truths. The are statements on the order of “An apple a day keeps the doctor away.” This is a good principle, it is good advice, but it’s not a pure truth, nor is it a rule.

The principles are mental cubby-holes. They give a name to a concept so that you can talk and reason about that concept. They provide a place to hang the feelings we have about good and bad code. They attempt to categorize those feelings into concrete advice. In that sense, the principles are a kind of anodyne. Given some code or design that you feel bad about, you may be able to find a principle that explains that bad feeling and advises you about how to feel better."

You see, the point about these OO principles (and patterns and even so called "best practices") are an attempt to distill experience. Assuming these principles are worth anything (and the SOLID ones do) - You can either  ignore that knowledge and learn these lessons the hard way, i.e. through your mistakes or internalize them and apply them.

How do you apply principles - by applying critical thinking. I wrote here before that I don't like the term "best practices" since it implies you don't have to think. Sorry, being a developer means you have to think, deal with it. There's no "deus ex machina" - it is all up to you.
Any good idea can be abused, e.g. let's look at the Single Responsibility Principle. SRP helps bring cohesion into classes by  lowering the "reasons to change. It is very easy to apply it ad absurdum and get a single method classes that really only do one thing but that's not the way to go.
The thinking part means that when we implement some class, and we happen to know about SRP, we can say to ourselves, hey this looks like an SRP violation. Why am i doing this? Do I have a good reason for that etc. knowledge of principles and patterns promote making your design decisions explicit and thus having you control the design (and architecture) rather than the other way around.

One claim I read is that principles such as SOLID preach to the choir. The so called "20% of developers" who already read blogs, enrich themselves and generally know what they are talking about. Well, I don't know, but the way I see it, not all those developers are black belt gurus who feel good design by the direction of the wind or get perfect code by mumbling a few incantations. I know I am not one. That means I am always learning. Ok, so I already know about SOLID and a couple of other nifty patterns so I need to learn other stuff like good principles for functional programming or whatnot . If you are starting out on OOP though, I suggest you build yourself on a SOLID foundation and go form there

 

 A couple of days ago, one of our team members saw in the log that we are getting index out of bound exceptions in some of the code of the a piece of code I wrote (part of the EventBroker). Taking a look he saw the following:

       private void EventSender(object state)
        {
            var id = (Guid) state;
            try
            {
                    sagas[id].Dispatch();
            }
            catch (Exception e)
            {
               
                Logger.DebugFormat("EventSender: Exeption Details={0}",e);
                rwl.TryRLock();
                var isClosed = true;
                if (sagas.ContainsKey(id))
                    isClosed = sagas[id].IsClosed;
                rwl.ExitReadLock();

                if (!isClosed)
                    HandleSagaFault(id);
                
            }
        }

It is well known that one of the guidelines for exception throwing is "Do not use exceptions for normal or expected errors, or for normal flow of control".So,naturally, he raised an eyebrow and asked me why don't I check that the id exist before I try to call the Dispatch method.
e.g. something like:

            try
            {
                rwl.TryRLock();
                var isClosed = true;
                if (sagas.ContainsKey(id))
                    sagas[id].Dispatch();
                rwl.ExitReadLock();
            }

Well, as it happens the EventSender method runs on a thread pool thread (something you could have guessed from the object State parameter). It's role is to get the saga to dissipate the event. i.e. the Dispatch method on the Saga object handles actually sending   events to event's subscribers.And while it does its work in parallel it also waits for all the events to arrive before returning (collecting any communications exceptions etc.) What this means is that the Dispatch method takes awhile to return.

Ahh, there's the rub - Contrary to the guideline mentioned above, it is actually more worthwhile to pay the performance penalty of throwing an exception rather than pay the more severe penalty of holding a lock for a long time. The lock affects all the threads running whereas the exception only affects the isolated thread.

Indeed - on another MSDN page there a better version of the guideline "Do not use exceptions for normal flow of control, if possible. Except for system failures and operations with potential race conditions"

The lesson here is (again) is that we need to think before blindly following guidelines.Guidelines are, well er, guidelines not commandments

PS
if you release the lock before calling Dispatch you don't gain anything, since it is multi-threaded code and the id can still be collected between the check and the call itself.

PPS
in case you are wondering TryRLock() is an extension method which try to obtain a lock with a given (short) timeout and throw (instead of deadlock) if the timeout is reached.

 

As I mentioned in a couple of previous posts (like "Using REST along with other architectural ), I've been spending the last few weeks writing an Event system over WCF (probably also explains posts on  WCF gotchas like this;) ). Being a communication infrastructure it is still a long way from being completed, but it seems to be stabilizing and I think it turned out nicely so I thought I'd share a few details.

Let's start with the simple part - the usage.
The eventing is built on the idea of a bus (i.e. no centralized components) and the resources/services that want to use eventing have to use a library which I call EventBroker.  There are two modes for using the EventBroker. one is "regular" events which are contexless. This means that consecutive events can reach different services, and there is no context that flows from event to event:

bool raisedEvent = eb.RaiseEvent<SampleEvent>(new SampleEvent());

The second type of events are Sagas, which represent long running interactions. Sagas does have a "best effort" guarantee to reach the same recipients over consecutive calls. Also you can also End sagas (sucessful termination), Force End Saga (successful termination by a service that didn't initiate the saga) and Abot Saga (unsuccessful termination): Here is how you raise a saga event.

var evnt = new SampleEvent { data = somevalue};
var SagaId = Guid.NewGuid();
eb.RaiseSagaEvent<SampleEvent>(SagaId, evnt);

if you use the same Saga Id, the events are handled as part of the same saga, if you use a Saga Id that wasn't previously defined it will initialize a new saga.
The eventbroker translates events to the relevant contract and dispatches the events over to the different subscribers. Which brings us to to the next part which I  guess,   is also a little more interesting. How subscriptions are defined.

The first thing to do is to define the event itself.

    public class SampleClassEvent :ImEvent
    {
        public string DataMember1 {set;get;}
        public int DataMember2 { set; get;}
    }

There aren't any real constraints on the event, except that it has to "implement" the ImEvent interface. Which is really an empty interface but it marks the event as one for the event broker.
Then you have to define an interface for handling the event. The event broker, builds on the idea of convention rather than configuration (an idea popularized by the rails framework) so it is easier to generate the interface (something I do with a resharper template)

    [ServiceContract]
    public  interface IHandleSampleClass
    {
        [OperationContract]
        int  SampleClass(SampleClassEvent eventOccured);
        
    }

The convention is that the interface will have a IHandle prefix followed by the name of the event. It will hold a single operation named like the event (without the Event suffix) and will recieve a single parameter which is the event data. Currently  events do return a value (int) but I am thinking about changing it to void and have everything marked as OneWay for added performance

Now, when we create a service which needs to handle events it will do that by specifing which events it handles. E.g.

    [ServiceContract]
    public interface ImSampelResource : ImContract, IHandleSampleClass, IHandleSomeOtherThing
    {
    }

So each contract declares all its subscriptions (by a list of IHandleXXX). It should also include the ImContract interface which holds all the service operation used by the eventbroker (e.g. ending sagas etc.).
Services that want to raise events should inherit from a ControlEdge class (base class Edge component that delegates control events to the event broker)

There's still the question of how does the event broker knows where to find other services. There are several ways this can be done (e.g. a service repository) but since we have  blogjecting watchdogs in place anyway, we use them to propagate liveliness (and location ) of services.

This sums up this post. It is basically just a little context for several planned posts where I hope to talk about some of the challenges, alternatives and design decisions that led me to the current design. Meanwhile, I'd also be happy to hear any comments, ideas or reactions you may have
 

(This part III of a series -see Part I, Part II)

We need to face it, there are no absolute truths when it comes to software architecture( I guess that's part of the reason the term always looks so fuzzy. ) Should we use REST? it depends. Should we use OR/M or direct database access? it depends. Sometimes even a big ball of mud can be a good option. The good news is that we can always answer "It depends" to any architectural question and always be correct. The bad news is that it is our role to figure out what does it depend on and come up with a viable trade off.

The fact everything is a tradeoff doesn't mean that there aren't any cases where the trade-off is simple. E.g. if you just have to build a couple of data entry screens and a simple database you probably shouldn't spend a couple of months evaluating your options, just write the damn thing in your spare time. Nevertheless, just hacking it can mean it wouldn't be extensible, it that's what you needed then cool. if not, maybe you should have considered that.

This is one of the reasons I don't like the term "best practices" - it sends out a "you don't have to think anymore" message which is oh-so-incorrect. Patterns on the other hand, don't send  out this message, as they also include discussion on where to use them as well as their limitations and pointers to other patterns. Unless of course, patterns are seen as an end-goal rather than a means, in which case, they look pretty close to "best practices"

To sum this post - everything is a tradeoff. The most important bit here is to keep that in mind, even when the choices seems obvious. Awareness  is the key to better decisions.

Powered by ScribeFire.

 

As I was going through my RSS list (which I publish as a linkblog) I noticed Reginald Braithwaite's post on the metaobject protocol. Reginald says

The bottom line: Languages like Java give you an object protocol: There is one way to do things with objects and classes and interfaces, period. Anything else, like adding generics or annotations, must be done outside of the language, it must be done by the creators of the language.

Languages like Common Lisp and Smalltalk give you a meta object protocol: You can decide for yourself how to do things with objects, classes, interfaces, generic functions, whatever you want. You don’t need to wait for a committee to try something different.

What's nice is that a couple of posts later I noticed a link by Harry Pierson to an attempt by Jeff Moser to port OMeta to the .NET world and to enable the very "meta object protocol" Reginald mentios. Here are the opening bits from Jeff's post (go read the rest):

What if programming language implementations were object-oriented? What if you wanted to change one itty-bitty part of your favorite language? Say you wanted to add an exponentiation operator (^^). How hard would it be?

Wouldn't it be nice if you could "subclass" a language like C# and then add 5 lines of code to make it work and then use it? What if you could add Ruby-like method_missing support in 20 lines?

What if you could conceive of a new language and start experimenting with it in production code in an hour? What if it leveraged your knowledge of existing frameworks like .NET?

Also check out the code Jeff posted so far (on CodePlex)

 

I saw a post (via Dzone) by Keith Elder called  "How to not screw up your application object model - Don't go all OOP on me!"
Its a long read, so I'll give you the skinny. Keith tries to make his point by an example. He designs an elaborate initial object model for a veterinary clinic (IPet, pet, cat, dog, spaniel, owner, address, work address ..), then takes the search screen from a real vet. app, maps the model to the information needed for that screen and explain that the initial model sucks and that that particular screen needs two UI objects (search and searchresults) that would better handle the problem.
He sums the post with

"The big takeaway from this litany of verbal spillage is when you design
applications you have to know how the end-user is going to use the
application and what the business requirements are.  Sitting down and
building an elaborate object model, while really fun and educational,
it is a big waste of time.  Don't do it.  Sit down and ask questions so
the behavior and business requirements can be accounted for.  Whether
you are using CSLA.Net or not the same thoughts must go into building
the application.  I hope this helps cements these concepts because I
know there is a large amount of disconnect with this out there among
developers who think everything is a perfect object graph like they
learned it in school. "

There's nothing really wrong in the conclusion on its own but in the light of the post itself there are few things to notice

1. The "elaborate object model" Keith talks about has nothing to do with OOP.
   
2. Keith just drafts an "elaborate object model" but fails to note that it is a "Problem Domain Object Model" (PDOM) and a bad one at that (since it wasn't built with a domain expert involvement). The role of a PDOM is to form an initial working vocabulary with domain experts. It is usually a big mistake to use it as a foundation for development
3. Even as a PDOM and even considering it was built for the sake of argument, it still doesn't make sense to create a class for each type of dog. When you create a sub-class you need to have a new behavior not a different value in a property.
   
4. The example of the client side objects which do not match business side objects doesn't prove anything. client object (esp. search dialogs) tend to cross or aggregate domain boundaries. You can still have a meaningful OO domain model running in a backend.
5. The fact that the search screen aggregates data from multiple domain object doesn't mean that the other processes and screen don't use that but from the application type at hand I would bet that most of the screens only deal with CRUD which brings me to another point:
   
6. Another consideration Keith fails to mention is the type of application at hand. For the example given. It is probably a viable option to develop the whole thing as an Access application with embedded SQL in the controls. no OO and and no nothing.
7. In any event. It seems to me Keith doesn't provide enough requirements to justify either of the solutions

In summary, yes it is completely wasteful to draw an off-hand solution that doesn't support any concrete requirements but that's an absolute truth whether you are using OO or no.

Edit 5/5: After looking at Evan Hoff's post on this. I understand that the point is  better delivered if you can also see the object model in question. So here it is:

Figure 1: Poor OO model - don't use

 

Among the reactions I got for my previous post on the Singleton pattern in .NET were a couple that talked about the design rationale behind the solution I posted:

Adi Avnit posted on the risk of using a "generic singleton":

"However, there is one possible pitfall to this approach, as it makes this code possible:
Singleton obj = Singleton.Instance;
MyClass obj2 = new MyClass();

While I personally like the idea of having the freedom to use the same class in two different ways throughout the application, I know some people like their Singletons - well, single.
On the other hand, if you write a class from the start as a Singleton this is not an issue.

There is an inherit risk in decoupling a class from it's expected behavior, so take this into consideration before using this pattern."

And Cade Roux wrote in a comment:

"don't believe discoverability was a motivation for Singleton. The purpose was to ensure only one instance existed (a print spooler, a file system). You can get discoverability through the technique you explain but the primary purpose of singleton was a pattern of prohibition - to stop a programmer doing something they shouldn't do by accident by enforcing the primary goals of single point of access.

It is more than a global, and the OP, while comparing them to globals, does not acknowledge that this solves many of the problems which made "global" a 4-letter word. Knee-jerk reaction against globals is not healthy once you've mitigated their drawbacks.

It is true that usage of singleton can be misguided and cause coupling - which is why you need to ensure that the usage of the pattern matches the motivation in the first place.

This is a key point of design patterns which Alexander should have made clear in http://en.wikipedia.org/wiki/N...is_of_Form - the pattern is a result of a resolution of system of forces. It only satisfies that system - moving it to another different system will result in poor fitness. "

I guess that's a good opportunity  to talk about design issues regarding Design Patterns in general and the GoF ones in particular.

Aristotle Pagaltzis noted (in a comment on Cedric's blog) that

"design patterns are a sign of a deficiency of a language for the purpose that the design pattern addresses. In other words, the Visitor pattern used in Java points to the fact that Java is deficient in terms of list processing: the `map` and `filter` constructs need to be emulated with lengthy OO incantations."

In other words Concrete patterns* (like GoF's) are tied to implementation which means that the implementation language is  part of their "context". Thus when you come to apply a pattern in a different language you need to consider the language in use and make sure that

1. The pattern is needed
2. The solution in the pattern is the best one for the language.
   

Let's look at a few examples.
If we take the Visitor pattern mentioned above - The intent of the visitor pattern is to "represent an operation to be performed on the elements of an object structure without changing the classes of the elements". In .NET 3.5 you can do that with the use of LINQ (to find relevant items) and extension methods (to add external functionality) - same intent, completely different implementation.

In the case of the Singleton pattern (mentioned in the previous post). The intent is to "Ensure a class only has one instance, and provide a global point of access to it". An implementation using templates (or generics) is superior since it provides the same intent but also adds better compliance with the Single Responsibility Principle. While it is possible to create the class as a non-singleton on the one hand it solves the multi-threading issue in one place preventing both duplication of code and mistakes (again this can be especially important in non-forgiving environments like C++). It also lets you (not in the implementation I provided) add flexibility and e.g. let singletons die and (if needed) resurrect themselves etc. Note that even if you are not convinced that using generics is better, there's no doubt that using .NET's thread-safe static readonly variable (public static readonly MySingleton = new MySingleton();)  is a much simpler solution than the GoF one and again, answers the original intent with a different solution.

Another example would be the Template method pattern. The idea behind the template method is to support the Open closed principle (classes should be open for extension but closed for modification) or as the GoF defines the intent:

"Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template method lets subclasses redefine certain steps of an algorithm without changing the algorithm strucuture"

.NET 3.5 supports the notion of Generic Delegates so you can define functions as parameters and pass them along. This is a better implementation than the template method as now you don't have to subclass when you want to extend an algorithm you can just pass a function that matches the signature
e.g.

    class Program
    {
        static void Main(string[] args)
        {
            var alg = new Algorithm();
            alg.DoSomething(Funky);
        }

        static bool Funky(string value)
        {
            return value == "somevalue";
        }
    }


    class Algorithm
    {
        public void DoSomething(Func<string,bool>CanGoForward)
        {
            string someVariable = "somevalue";

            if (null!=CanGoForward && CanGoForward(someVariable))
            {
                //do one step
            }
            // whatever
        }
    }

You should note that this "refactoring to fit the language" phenomena isn't limited to GoF patterns (or .NET :)) , for instance you can consider the use (or lack thereof) of Dependency Injection in languages such as Ruby which I wrote about a few months ago.

To sum up - when you talk about design patterns you need to consider the implementation language, design pattern deal with deficiencies in the language and different languages have different deficiencies and may have better solutions to the problems solved by the patterns

---

* As opposed to architectural patterns which are more abstract and thus more reusable (e.g. Hohpe & Wolf Enterprise integration patterns, or Martin fowler Enterprise Architecture patterns)

 

Awhile ago I asked "who tests the test":

" One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?"

I would add that this is especially true considering that the same person that writes the code also writes its unit tests

One answer I had was making sure to keep the triad of tests->code->acceptance tests. so that each two test the third. This is especially good since, usually, the acceptance/integration tests are not written by the developer who wrote the code.

One answer I got was that tests are usually simple enough to be self evident and when tests contain complex constructs you should probably test them as well.

Basically, tests can have two types of problems

• Not testing the right thing
  
• Not testing the thing right

Code+unit tests +acceptance seem to take care of the first type of problems. The claim that tests are simple should take care of the second type of problems.
However I think that it isn't always true, especially when it comes to using the tests for regression. consider for example, the following two incidents I had  where the the problems were actually with the tests:

1. We have a component that interacts with resources in different URIs, so when resources wake up they register themselves with that component. Resources register two URIs -> one for control and one for interactions. We've added a business rule that both URIs should be on the same host (naturally a new test for that was also added but that not the point) suddenly a bunch of the old tests started failing. Sure enough, I took a look at the new code, found a bug, fixed it, but the tests were still in the  red. In the end it turned out one of the Uris that is registered during the tests had an extra "L" (locallhost instead of localhost) so the tests failed with the new rule - as they should..

2. After writing a piece of code to "just work" it had to be made multi-threaded. I needed to write a test that would make sure the component can handle multiple concurrent requests. Sure, I thought, I'll just run few or even all of the other tests in parallel and be done with it. The only problem was that the other tests were written to be isolated (i.e. new instance per test) running them in parallel had no effect what-so-ever.

What can we do to help us locate problems and bugs with tests?

• remeber that Test code is code - so as you write more tests you should also refine the tests design and refactor the code accordingly. For instance in the first example above, it was easier to diagnose the problem since the faulty setup was done by a single method used by alll the failing  tests (Single Responsibility Principle @ the method level).
• Test-First - One of the reasons test-first is beneficial is because you can actually see the test fail before you implement the code to make it work. This helps you make sure that the test actually tests the behavior you want. When the code is already in place it is harder to make sure if the test works because of the change or not. In the second example running all the tests in parallel everything was still green, which triggered me to write specific tests for checking for multi-threading issues.
  

What do you do?

 

While I personally don't like the Singleton pattern too much (it is essentially an OO mask for Global Variables, it makes it harder to unit test etc.), I still need to use them now and then. Anyway, I saw a post by Jack Altiere about "Using the Singleton Pattern" in .NET and since it presents an implementation that leaves a lot to be desired I decided to comment on that

For one, he presents a solution which locks every time the singleton is accessed (see below). The reason this is flawed is that singletons are instantiated once, but they are accessed a lot of times and with this implementation you pay every time you try to access the singleton

//bad implementation - don't use
//also note some initialization etc. is omitted for brevity
 public static Foo Instance
{
      get
       {
        // This lock allows thread safety.
             lock (_mutex)
             {
                   if (_instance == null)
                        _instance = new Foo();
                  return _instance;
            }
      }
   }

There's a better "standard" way to do that which basically means you check if the the instance is null, lock, check again (in case more than one thread got past the first check ) and only then instantiate (you'd also need to mark the instance volatile in this case)
.NET has a better way to create thread safe singletons by merely using a static readonly declaration as in:

public  sealed class singleton
{
        public static readonly MySigleton Instance = new MySingleton()
}

This implementation is not as lazy as the implementations above - but it is thread safe an efficient (you can get more laziness if you use a nested class as described here)

But that's not all- see, one of the problems of the Singleton pattern (besides the ones mentioned above) is that it is also a violation of the Single Responsibility Principle (SRP see OO primer). You can solve this problem if you use Generics and create something like C++'s template based Singleton class:

namespace ConsoleApplication5
{
    class Singleton<T> where T : new()
    {
        private static readonly T inst = new T();

        public static T Instance
        {
            get { return inst; }
        }
    }
}


and then use that simply by doing something like Singleton<MyClass>.Instance

[update]
Eran points out that Jack's code is not only inefficient but also not thread safe (see Eran's post on the subject) - basically you need to make the instance variable volatile just like in the double check scenario

Reshef points to the "Static Gateway Pattern" as an alternate  way to get the singleton effect. While we are talking about other possibilities, I should have probably mentioned that the usual way I handle situation where I need a single instance is instantiating one instance on the main/loader thread and then just using dependency injection... :)

 

If you recall what I currently work on is a type of a visual search engine. In a nutshell when we get a request (image) we allocate a bunch of algorithmic engines in a grid like manner to process the image  (e.g. try to perform OCR or whatever). As it happens, we are developing the different components using several different environments(*) - e.g. the control bits run on windows (.NET) and most algorithms run on Linux (mostly C++).
The need for easy cross-platform communications and extensibility, the resource nature of the solution and a few other tidbits led us to design our solution in a RESTful manner.

If you are a .NET developer/architect and wanted you may know that to implement a RESTful application in Windows Communication Foundation (WCF) you really have to jump through hoops.For instance  you have to go back to basics and use the HttpRequest and HttpResponse, handle the breakdown and parsing of URI hierarchies yourself not to mention  fight  with the  bindings .

Fortunetly this all changed with WCF 3.5. True, .Net doesn't have (to my knowledge anyway) something like RESTlets, but at least building REST on http is pretty straightforward.

Consider for example the following excerpt:

    [ServiceContract(Namespace = "http://paperlnx.Contracts/2007/12", Name = "ISessions")]
    public interface ISessions
    {
        [OperationContract]
        [WebGet(UriTemplate = "/Sessions/{sessionId}")]
        [ServiceKnownType(typeof(Atom10FeedFormatter))]
        SyndicationFeedFormatter ListSessionStatus(string sessionId);
	.
	.
	.

With these 6 lines of code you see the essence of  the .NET 3.5 REST goodies

1. Integrated support for HTTP verbs  - The sample above shows the support for GET. You can get the other verbs almost as easy with the WebInvoke Attribute. To do that simply specify the verb you want e.g.   [WebInvoke(Method = "PUT")] , [WebInvoke(Method="DELETE")] etc.


2. Support for URI templates -  In a way not too far from Joe Gregorio's IETF draft , WCF supports the notion of providing a way to describe families of URIs. This is done using the UriTemplate class. The WebGet and WebInvoke attributes also accept URI templates as variables and map the variable values (the curly brackets ones {}) to parameters of methods.
   
3. Support for standard  formats - you can use plain XML or you can choose to use RSS and ATOM syndication formats. In its most basic form you just create a syndicationfeed and format it to atom feed. Which is what we do for error messages:
   

   
           public static SyndicationFeedFormatter GenerateAtomError(string errormessage, string description,Uri location)
           {
               SyndicationFeed feed = new SyndicationFeed(errormessage, description, location);
               return new Atom10FeedFormatter(feed);
           }
   

   Naturally you can also add items and element extensions to all elements (e.g. the feed or items)
   

All in all, I am a happy camper :) After all, when you make an architectural decision, you always need to review it once you opted for an underlying technology. Even when a decision is right. The friction caused by a  technology which doesn't accommodate it well can both make your life miserable and make a good decision bad. .NET 3.5 with its newly added support for REST increases the architectural freedom and that's always a good thing

---

* Among other things, it helps us avoid the "Network is homogeneous" fallacy - but that's another story :)

 

Following the third part of my "Defining SOA" posts Udi Dahan left the following two questions:

How does this [layers - ARGO] play with two services talking with each other? One pubs
to the other's subs?The other requests to the first's response?
How valuable is the layered abstraction?

Considering Udi and me usually see things eye to eye. I guess that if I managed to get him confused, more clarification is warrant :) I'll do that in two posts, this one which will explain the concept of layers and the next which will explain why it is paramount  to SOA (and answer the two questions)

Usually when I review an architecture one of the first (sometimes the only) architectural artifact I am shown is a "layered diagram" of the architecture e.g. something like the following:

These sort of half layers/half block diagrams with or without the common 3 layers (which also appear in the diagram above) of "UI" "Business" and "Data" give the whole idea of layers a bad name

The key differentiator between layers and just a bunch of blocks is the limitations on the allowed communication paths between the components (layers vs. blocks). In the previous post I quoted an old (2005) definition I had for layers where I said the following

"Typically a layered is allowed to call only the layer below it and be called only by the layer above it (but there are variants e.g. a layer can call to any layer below it; vertical layers that can call multiple layers; etc. -- all is fine as long as the layers communication paths are limited by some rules)."

Alas, I was too quick on the Copy/past and everything in the brackets (bold) is wrong - it should actually say - "but there's another variant where layers are allowed to call the layer above it or below it". The other variants (like  a layer that can all any below it) just muddy the water, makes it hard to distinguish between layers and regular components and thus make layers seem unimportant. consider the following diagram:


So, in the above diagram the relations are the Component D and Component C know each other. Component D is made of two layers (A and B). Note that a more proper representation should also explain the relations allowed between the layers i.e. is it unidirectional or bidirectional (unless there's a common convention in the project)

Why is the distinction between layers and other type of components important? because Layering gives you some benefits which "just components" don't:
1. Layers allow information hiding. Since we don't know the inner working of what's beyond the layer
2. Layers allow separation  - Things beyond the immediate layer  are hidden from each other. This means that things which are beyond the layers are loosely coupled in a way that allows for  flexibility and the addition of capabilities. for instance adding a firewall between your computer and the internet.
3. Layers allows changing the abstraction level - since layers are hierarchical in nature, moving through the layer "stack" you can increase or decrease the level of abstraction you use. This allows expressing complex ideas with simple building blocks. The best known example for this is the TCP/IP stack moving from an abstraction level close to the hardware of the network interface layer to the application level protocols such as HTTP


On the downsides - layers hurt performance by adding latency. Also too many layers introduce added complexity to the overall solution (e.g. it is harder to debug).

It it interesting to note that Interfaces are in fact leaky layer abstractions (vs. for example SOA contracts which are not leaky) - since when you use an interface you still need to instantiate the object which is otherwise hidden behind the "layer" (interface). This is basically the reason we want something like dependency injection (DI) - to help make the abstraction complete and why languages like Ruby where the contract abstraction is complete - you don't need DI (I discussed Ruby and DI in another post)

Another issue which I mentioned here is the difference between logical layers and physical (or potentially  physical) layers. I usually call the first kind layers and the latter tiers. logical layers are local and can assume a lot about their neighboring layers. Tiers or physical layers can be distributed, which carries a lot of implications (something I discussed here recently in relation to MS Volta)





 

Wes Dyer, one of the principal people behind the Volta tier-splitting was kind enough to leave a comment on my previous post. Here is one quote from that comment

"I do want to clear up a few things about Volta that we apparently
didn't make clear enough. We do not believe that you can develop an
application as if it will run on a single tier and then just sprinkle a
few custom attributes here and there and be done with it. More than
anything else, programmers need brains. Volta does not claim that
programmer brains can be checked at the door. When the programmer wants
to divide the application across a particular boundary then things like
network latency, new failure modes, concurrency, etc. need to be
considered at that boundary. What Volta does is make expressing the
transition between boundaries easier. It reduces the accidental
complexity of writing all of the boilerplate code to express the
programmer's intention. This allows the programmer to focus on the
essential complexity of his problem domain -- figuring out how to write
effective code for that particular tier boundary."

For one, it is good to hear that the architects behind Volta have a deeper understanding of distributed computing challenges - even if the first version doesn't seem to show it. I didn't use MS Volta enough to say that indeed the problem is not with the inherent capabilities and design  (let's just take Wes words for that). I am also not against saving the boilerplate code (though I would personally favor libraries rather than code generation and try to keep the "generation" gap to a minimum (i.e. the amount of generated code or the distance between the abstraction and the next concrete level)).
Lastly I am also in favor of trusting developers have brains and that it is ok to provide developer "sharp tools". So if all is good, where's the problem?

The problem is that you have to make it "easy to do the right thing" and provide the means to do the more complicated, less safe things. When I teach my young kids (and I can objectively say they are very smart :) ) to use a knife, I don't hand them the razor sharp, butcher knife first. They start with the plastic ones. When they've mastered that they can try something more dangerous. When you allow distributing something at a flick of an attribute and put marketing blurb on the site that makes it compelling to use it you create the wrong impression to the less experienced folks.

In one project which architecture I reviewed, the (very talented) architect/developer designed his own distributed transactions system (he shouldn't have been doing that in the first place - but that's for another post). When designing this he built in a lot ways to control the transaction behavior including the option to allow transaction participants to prevent rollback without failing the transaction.  Circumventing the transaction was as easy as making it work properly. Are there edge cases where you may need to have one participant violate the ACIDness of the transaction ? I guess  so - but that is not the general rule. Most of the time when you commit a transaction you expect it to be ACID. if for some reason it didn't behave that way - you want to know about it, even if it didn't actually failed. When you don't make it easy to do the right you get unexpected behaviors, you get hard to explain bugs, you get slow performance etc..
Developers using tools, smart as they may be, don't usually go and read all the source code of the tool/framework they are using (maybe they should). If two options are just as easy to use, it seems safe to assume they are just equally right. Things which are unsafe should be clearly marked as such to prevent mis-use by unexperienced users. This is especially true for tools that are targeted for common use and to ease the life of inexperienced developers

 

I got a couple of emails with questions reagarding  my previous post on Volta. So here's another go at explaining why dynamic-tiering is not a good move - this time in technicolor.

Let's start with a simple illustration. The diagram below represents a typical local component(A) in its environment. As a component that works locally, it has access to other local components which it interacts with. These can be objects it created by itself or objects that where injected to it. The likely design for local components is to have a chatty interaction - After all objects can talk to instances of other objects quite easily.

 

Jeff Atwood (of Coding Horror) writes about Brian Foote and Joseph Yoder's "Big Ball of Mud" paper but completely misses the ball (pardon the pun). Jeff says that the paper describes

"classic architectural mistakes in software development."

while the paper describes the exact opposite. with the exception of the "Big Ball of Mud" pattern itself which can be seen as an anti-pattern (as,by the way, the authors explain) the rest contain exceptionally good advice on how to prevent problems. Let's look at them one by one

• Throwaway Code -  when you just want to make sure something works (prototype, spike etc.) or even a quick fix - you don't write elaborate and heavy code - instead you write something simple to solve the problem. Throwaway code only becomes a problem if you don't actually throw it away...
• Piecemeal Growth - The waterfallish "big plan" in advance has failed numerous times - so instead we should build incrementally, don't over plan or over design, build things into libraries only when it is proven to be recurrent problem, refactor etc.
• Keep it working - The pattern is basically about building often and keeping iterations short, having tests that always prove your code still works as you make changes to it
• Shearing Layers - This pattern is about identifying related components. We all know change is inevitable - however things don't change at the same rate. For instance we can probably get interfaces to be a little more stable than the implementations behind them. Frameworks evolve at a different rate than the business they support etc. The pattern is about dividing the system so that things that change at similar rates are together i.e. in the same package,  CSCI  or whatever you use to partition your system.
• Sweeping It Under the Rug - When you find that you do have badly designed or badly implemented code - the pattern suggest you localize and isolate it to a fixed area (e.g. behind a facade) to prevent it from propagating into the rest of the system.
• Reconstruction - This pattern is about understanding when the code base is so bad that it is better to start-over and rebuild from scratch rather then try to patch it. In many  way reconstruction is not a good or easy thing to do however the point here is to identify when it  is the lesser evil. 

Oh, and what about "Big Ball of Mud" itself - essentially from the architectural perspective it is indeed an anti-pattern - you have something that is not very maintainable, hard to understand and what not. However we should keep in mind that the idea behind designing an architecture is not to get the best, cleanest architecture. The idea is to make the right tradeoffs so that you'd be able to deliver the best overall solution under the constraints you face (budget, time, the team's skill and what not). If you're biggest constraint is time-to-market and your architect spends all eternity planning the 8th wonder of the world, fire his ass. I'd rather live with a Big Ball of Mud for the first release than not ever make a release...
Big Ball of Mud can be considered a pattern for pragmatic approach to building working software. This is probably not acceptable in the long term  - but it can be a good option for short term if you are aware that that's what you are doing and willing to treat what you get as "Throwaway code".

So again, except maybe big ball of mud, these patterns are not "project pathologies" as Jeff calls them - these are very good ways to keep delivering business value and working softwares

 

Well it seems setting up a new company can keep you busy at times - which is my official excuse :) for the quiet last week. Hopefully I'll have a little more free time this week

Note: I am talking in this post about roles and capabilities. i.e. when I say architect I mean someone who has the capacity to be an architect (e.g. as demonstrated in the architect training suggestion I made) and takes that role. On a specific project you may have a person that performs multiple roles such as an architect and project manger or an architect and developer while  other project warrant one or more full time architects.

Not all projects need architects. There, I've said it. Not all projects need architects and I am not talking here just about trivial projects. There are cases (maybe even many cases) where you can get by with what I call "off-the-shelf" architecture - maybe with a few adjustments that any master developer (i.e. seasoned and experienced developer) can handle. For instance a lot of web-sites can do pretty well by using Model-View-Controller (or a derivative of that) along with a simple O/R mapper such as active-record. In fact a lot of them do when they use a framework like Rails that made these architectural choices for them*. Another example is the vanilla 3-tier architecture provided by vendors (such as this one by Microsoft). Yes, when you take something off-the-shelf the result might not be optimal but that doesn't mean it isn't sufficient. You just have to be aware for the tradeoffs...

Another point is that "design" is not an exclusive architect thing. A developer is not a good developer unless she also known about  proper design. mastery of technical details of the language without understanding the wider context of design will just help you code a lot of crap faster.

Having said that we need to consider a few issues - When do you need architects? What do they Do? What's their relation and interaction with the developers ?

When do you need architects?

It is sometime a fine line between a project that can get by with an "off-the-shelf" architecture and one that needs an architect. It would be nice if we could have something like a litmus test that would tell us if architects are needed or not. I don't have one. The closest thing I could get to this is something I call the SCLR test (pronounced scaler). SCLR stands for Size, Complexity & Limited  Resources.

• Size - well if you are going to have something  estimated at 1000 man-years or dozens of teams. I think it is pretty obvious that you can't just use something  which isn't made to fit. If anything there's a need to divide the work between the teams in a way that would make sense so that you wouldn't get a big-ball of mud. There's also a lot of need to coordinate the efforts and keep the big-picture inline. Personally I think that it doesn't  have to be a huge project to warrent some architects involvement. Since as Fred Brooks notes the number of interactions grows exponentially as we add more people.  In my experience trouble starts even with more modest numbers - more than 4 or even 3 different teams working concurrently is probably a good number to start thinking  about architects
• Complexity - There are many signs for complexity in a project. The vision statement can provide a hint. "Let's design the software to support the next mars mission", "best CRM platform ever"  - an ambitions project will not make-do with "average" architecture. Size (which  I already mentioned) is also a sign of complexity, while previously I talked about size of the project, the size of data, users etc. is also relevant when we're thinking about complexity . A lot of external interfaces is another sign. Integration doesn't seem very complicated, until you actually try to pull it off. When you have to do a lot of that in a project that's complex. And there are many other signs
• Limited Resources - Naturally every project has limited resources, but limited resources should be considered as a sign for architect involvement if the resources are extreme. When resources are extremely limited the tradeoffs that have to be done are more meaningful, which is why wed want people who can help with that (i.e. architects). For instance in a projects I worked on in the past we had a lot of availability and performance requirements on one hand but only so many "U"s in the rack and even limited electricity to make all this magic happen. This turned something which otherwise was a relatively standard IT project into something a lot more challenging.
  

Assuming I manage to convince you that some projects can't just choose one of the available blue-prints and need some more work - the next step would be to convince you that architects are better suited to solve this than developers. I'll try to do that in the next post on this series where I'll explain what (I think) architects do and their place in the development team

---

* Rails has more than just MVC and Active-Record but that isn't an important point for this discussion

 

A side effect of my decision not to become an independent consultant at this time means that I have to shelve some of the projects I was considering. One of these projects was to create a training program for software architects which I was discussing with a couple of training centers here in Israel.
Since It seems I am not going to promote it, I thought I'd share what I think a training program for .NET/Java architects should look like in the hope that someone would find it useful and do promote it (or parts of it)

Soft Skills
The way I see it architects needs a bunch of soft skills to be able to perform their roles.
Here is the list I identified in the past (by the way, I began a series of posts on each of these skills and never got to finish it - maybe it is time that I will :) )

• Leadership. Influencing others to accomplish tasks and following your guidance
  
• System thinking. Understand decisions and constrains in the
  wide scope pertaining to whole of the solution at hand. This includes
  the ability to abstract problems.
  
• Strategic thinking. Understanding decisions and constrains and their alighments to the overall business of the company.
  
• Organizational politics. Understand the environment you operate in and how it influences you.
  
• Communications. Making sure you get your point across.
  
• Human relations. Understand the "people" aspects or human
  factors and dynamics. This includes things like negotiation, pragmatism
  etc
  

I am not sure if you can teach all of them, but few courses that can help (in my opinion) include:

• Negotiation/Persuation - Anything based on "Getting to yes","Getting past no" and "The power of a positive no" should be fine.

• Presentation Skills - While getting the architecture and technology right is what matters, if you can't explain it to the different stakeholders you're toast.

• Strategic Planning - This has to do with the vision thing I expect architects to manifest. Note that having a vision should not be confused with future-proofing a solution. future-proof means excess work not needed. Having a vision is knowing where you want to end - it can still be perfectly valid to completely re-write your applications along the way

Project Management
While the architect is not the project manager (mostly anyway), I think understanding the constraints coming from the project management point of view is very important. Since most environments call for a mix of agile and formal disciplines (hey, you've got to be pragmatic). I would train architect both in SCRUM and RUP (or some other formal methodology)

Also while not all environments needs this I would give an 2 days overview of important standards. The first would be IEEE 1471, which defines a standard for documenting software architectures. I would also teach ISO 90003 and CMMI.

It should be noted that the ISO 90003 is much better than the previous incarnation (ISO 9003) as it basically lets you define what you want to do to cover the different areas. The standard just helps you make sure you think about the various parts of project management (requirements, environment etc.). For instance I demonstrated how key areas of 90003 can be mapped to SCRUM to get it approved on my last project.


Languages, Design  and Patterns

I would want the .Net/Java architect training program to include at least 2 of the following languages:
Ruby, Scala, Erlang, F#, Python , Groovy ,OCaml
The reason for this is that these languages have different design goals than .NET and Java so learning them gives you additional perspectives and broaden your horizons for other ways of thinking (even if you don't use them in your project directly). You might have noticed that there's no .NET or Java training here. The reason for that is that's a prerequisite as far as I am concerned. You should master at least one

Object Oriented principles - hopefully aspiring architects already know this. However, I often see people who discovered some of the principles by themselves but haven't heard about all of them.
I am talking about principles such as Liskov Substitution  Principle , Open Closed Principle, Single Responsibility Principle , IoC containers, Don't Repeat Yourself  and YAGNI (I summarized my opinion on most of them in this paper)

The next step is to cover some design issues like Domain Driven Design, UI Design, Database modeling, Database alternatives l(after all the database is dead, right? )

Advanced design patterns - When most people hear the term design patterns they think about the GoF patterns. There are however literally hundreds of design patterns. Some of them are even worth learning :) . For instance there are patterns for concurrent and parallel systems like Proactor, Reactor Half-sync/Half-Async etc; Workflow patterns like Cancelling partial Join, Recovery Action etc; SOA patterns (ok, so I am still working on that :) )



Architecture Workshops
Another important part of the training, in my opinion, is to do some workshops and actually try to apply some of the material covered.

• Architecture Evaluation - workshop 2-3 days - It is probably worthwhile to delve a little on scenario based evaluation techniques such as LAAAM and ATAM. While I prefer evaluating architecture in code, the scenario based thinking is very valuable for eliciting architectural requirements
• Distributed Systems Architectures workshop - I'll expand on this in the next post

Lastly, there are also a few miscellaneous subjects like architect 101, the SPAMMED architecture framework , Agile architecture, Behavior Driven Design , common frameworks (though hopefully this would  not be needed ) like Spring/Spring.Net, Hibernate/NHibernate, iBatis  etc.


PS

Note that there are a few architect training programs available out there
One is offered by the Software Engineering Institute (SEI) and includes a 6 courses. SEI program seems to be focuses on formal sides of architecture as it includes courses on documenting software architecture and ATAM (You can see an old presentation I have on ATAM here)
Dana Bredemeyer  also offer architect training. Dana offers several workshops that cover the software architecture profession.
TOGAF (which is more of an enterprise architecture framework) offers both a certification and courses
Lastly, IASA is considering creating a software architect program and has a few courses in development
If you know any others I'd be happy to hear about them

 

You may have read about the guy who after spending 2 years on a Ruby on Rails project switched back to PHP. Without getting into the debate of whether Ruby on Rails is better than PHP or wether Ruby is overhyped (probably - but that doesn't mean it isn't any good either.By the way  it is the same with SOA, but I digress)

Reading his post I saw a few quotes that raised a red flag for me such as:

"But at every step, it seemed our needs clashed with Rails’ preferences. (Like trying to turn a train into a boat. It’s do-able with a lot of glue. But it’s damn hard. And certainly makes you ask why you’re really doing this.)"

and

#2 - OUR ENTIRE COMPANY’S STUFF WAS IN PHP: DON’T UNDERESTIMATE INTEGRATION
By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change..

and

Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables. I was always fighting against Rails and its migrations hiding my beloved SQL from me.

What these quotes say really is that this guy doesn't know about "Technology Mapping 101". Here is what I wrote about  technology mapping*  about 2 years ago (incidentally that's about the same time this guy started his Ruby adventure :) )

"Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible."

Every technology or framework has its own architecture. This architecture poses constrains and makes certain things easy (like using the ActiveRecord pattern in Rails) and certain things harder (like not using O/R Mapping ) so, for instance, on the case mentioned above a better technology mapping might have been RBatis (iBatis for Ruby/Rails) which lets you map SQL statements to objects. It is important to note (in Rails case) that one of the core tenets for Rails is preferring convention of configuration when you don't do that you have to work hard(er) - as you are working against the framework

Another problem with the technology mapping here was his point #2. It is a pity he only saw it in after the fact. It can be justifiable to switch everything but you've got to allow this change to occur iteratively. While I generally dislike the software architecture = building architecture metaphor, using it here does make sense: building software for an existing business (vs. greenfield development) is like building a new intersection. You have got to think about building all the detours that would allow the traffic (business) to continue to operate, sure it might run slower in the intermediate phase but it can't stop altogether.

So again, Once you have an architecture that fits your business, take a look at the technology options you have. try to choose the best fit. Whatever you choose - take a look at the implications of that technology and think about the tradeoffs you may find that you either have to adjust your architecture or change the technology altogether - if you don't you might find you waited 2 years of development effort or even more..

---

* Technology Mapping is one of the steps of a set of activities I identified as needed to make sure your have a solid architecture. The activities goes by the acronym SPAMMED and you can read about them more in this article and/or this presentation

 

Another great presentation at Architecture & Design world was Neal Ford's presentation on Domain Specific Languages (DSLs) . As the title suggests Neal gave examples both in static languages (Java) and Dynamic ones (Groovy, Ruby).

One interesting observation Neal made was that humans tend to create DSLs in real life whenever they (ok, we :)) have any non-trivial interaction or behavior. Neal gave sevaral examples such as the Starbuck's order taking ("Venti Iced Decaf with whip...") , musicians and a few others.

The next important point was contrasting ("classic") APIs and DSLs. The main difference is that the context is implicit and not repeated

A key  technique for building DSLs Neal mentioned was Fluent Interfaces. Fluent Interfaces means modeling the API so that lines of code are readable English-like sentences. The fluency comes from the easier readability by the interface user.

Fluent Interfaces, now that's a novel idea - what would a fluent interface look like, hmm, wait, I have an idea. Here are 3 samples that come to mind

DIVIDE x BY z GIVING y ROUNDED

INSPECT data REPLACING ALL "foo" BY "bar

READ someFile AT END SET eof TO TRUE

If you haven't guessed  the statements above  are in ...Cobol (by the way pardon the caps that's Cobol conventions..)
So ok, it isn't a new idea, but it is interesting to see it is making a comeback
Anyway one area where we see a lot of fluent interfaces emerging is configuration (mainly as an alternative to those lengthy XML files). For instance the following is an excerpt of configuring  Restlet components  (taken from my Edge Component pattern paper):

Builders.buildContainer()
            .addServer(Protocol.HTTP, portNumeber)
            .attachLog("Log Entry")
            .attachStatus(true, "webmaster@mysite.org", "http://www.mysite.org")
            .attachHost(portNumber)
            .attachRouter("/orders/[+")
            .attach("/getAll$", getAllRestlet).owner().start();
            .attach("/getLast$", getLastOrderRestlet).owner().start();


Note that this example also uses another fluent interface/DSL technique which is method chaining.

Dynamic languages make it even easier to write DSLs since they provide a lot of extension capabilities (see my previous post on OCP in Ruby), are less strict about types, allow reopening classes etc. 
Oneexample for a Ruby DSL is RSpec which is a framework to support Behavior Driven Development (BDD) in Ruby - The example below shows an excerpt for defining specifications for an eight-ball game

require 'eight_ball'

describe Eight_ball do
    before(:each) do
        @eight_ball=Eight_ball.new
    end
    .
    .
    .
    it "should lose if 8-ball sinked in pocket other than called" do
        [1,2,3,4,5,6,7].each ( | val | @eight_ball.sink(:player =>"Player1", :Ball=> val)
        @eight_ball.call(:player => "Player1", :pocket => :upper_left)
        @eight_ball.sink(:player => "Player1", :Ball => 8, :pocket =>:middle_left)
        @eight_ball.game_status.should == :ended
        @eight_ball.player_status("Player1").should == :lost
    end
end
 
By the way Joe Ocampo built a very nice port for rbehave (another Ruby BDD framework) to .NET 3.5 by extending NUnit which has a very Ruby-like syntax
      

Anyway, the DSLs demonstrated by Neal provide a very good example of the difference between dreaming big and actually doing stuff in the small. The counter example for that are "Software Factories". As I wrote here about a year and half ago

Software Factories is not a new idea  - see for example "Software reuse: From Library to Factory" by M. L. Griss  (published in 1993(!)) which talks about "Software Factories" and "Domain Specific kits": components, frameworks, glue languages etc.  The current Microsoft  incarnation of Software Factories takes a similar approach focusing on Domain Specific Languages, Frameworks but also adding important aspects like multiple viewpoints, patterns and designers. The idea is that  building on modern technologies, as well as learning from the mistakes from sister approaches to code generation (OMG's MDA, in case you are wondering) will enable us to build something that is useable.

Microsoft seems to be taking some steps in the right direction (GAT is probably the best example). Nevertheless there is still a long way to go before we can realize the dream of "factories" for vertical applications

Unlike small code based DSLs - the modeling based approaches of software factories, MDA etc. aim too high and thus provide much less value or suffer too much from the generation gap (the code generated is too generalized or far off from the actual need of the solution). Another problem with software factories/ MDA DSLs is the modeling (i.e. diagrams) - they say a picture is worth a thousand words. This is true if you treat models as sketches you can raise the level of abstraction by as much as you want and convey ideas with less clutter. However when you need to make the model very specific so it would allow code generation - you get to a stage where it is more convenient to do it in code and rely on generated or pre-built DSL or framework

Lastly you can  download Neal's presentation (in PDF form) from his site

 

A few weeks ago I wrote about Dependency Injection and how it doesn't really matter in Ruby. On one of the comments for this post Yoni said (in regard to extending classed - both in .Net 3.5 and Ruby):

"Regarding the issue of extending classes (and partial classes) I am very sorry it was ever invented. Developers should be encouraged to divide large classes by decomposing them into decoupled sub-components using design patterns and not by simply splitting them between files. Even though a feature is called "extending" a class it is in violation of the Open Closed Principle..."

that sounds like a serious charge :) So here's another round of  "Ruby does it better" the time featuring Bertrand Meyer's "Open Closed Principle" or OCP for short.
OCP was defined in 1988 in Bertrand's book "Object Oriented Software Construction"  as follows:

Modules should be both open (for extension and adaptation) and closed (to avoid modification that affect clients)

When we work with a language like C# or Java we have several ways to do that - here's what I had to say about it :

It is easy to see the benefit of having a class that answers this principle: When you need to add a requirement, instead of breaking dependent code (and tests) you just extend it somehow and everything is nice and dandy. Furthermore, violating OCP can result in Rigidity,Fragility, and Immobility.

But how do you do that? The obvious (and naïve) answer is inheritance. Every time something needs to change just add a sub-class. The parent class is not changes and voilà. However, if you add sub-classes all the time you'd get "lazy classes" or freeloaders--sub-classes without a real reason for existence not to mention a maintenance nightmare.

Thus, sub-classing is an option but we need to consider carefully where to apply it. Other (more practical ) OCP preserving steps include:

• Marking class fields as private


• Not adding getters/setters automatically. You may want to read "Why getter and setter methods are evil" by Allen Holub, and Martin Fowler's rebuttal/refinement in "Getter Eradicator".
  
• Inversion of Control strategies
  
  
  
  • TemplateMethod pattern
  
  
  • Strategy Pattern
  
  
  •  Anonymous delegates (in C# 2.0) and Closures in dynamic languages (also in C# 3.5)


• And don't forget to hide implementation details behind (stable) interfaces

The way I see it Ruby (and other dynamic languages for that matter) just allows us to extend this repertoire by adding a few other options such as:

Singleton methods (that's not a very good name - but the more appropriate name "instance methods" was already taken by another poor naming choice...) which are methods added to a specific instance of a class:

class Foo
  def bar
    puts "foo.bar"
  end
end
   
obj = Foo.new
obj2 = Foo.new
def obj.bar  # redefining bar just for obj
  puts "foo.newbar"
end
obj.bar   # prints foo.newbar
obj2.bar # prints foo.bar

Closures - I already mentioned closures for C# but it existed in dynamic lanaguages for a long time now. closures are sort of like injecting a procedure. Here's a quick sample:

class Foo
  def bar(myProc)   # accepts a callable object
    foobar = "foo.bar"
    myProc.call(foobar) # call the object with a parameter
  end
end
   
obj=Foo.new
printer=Proc.new {|msg| puts msg}

obj.bar(printer)

Meta programming - Ruby has a lot of ways to add and change classes. methods and whatnot. Again. here is a simple example:

class Foo
end
   
obj=Foo.new

# obj.bar - error
Foo.class_eval do           # this simplistic example can also use regular def
    define_method :bar do   # but class_eval can also evaluate strings to create
      puts "foo.bar"        # dynamic methods like setters etc.
    end
end

obj.bar

And there are a few other similar ways.

All of them  are not violating OCP!

OCP is kept since we do not alter the original class. Anyone using the original class not in our context (i.e. not in the modified context) will not be affected by the change. The interface of the class, in the sense that was originally defined is not changed either, and remains stable. Any client that uses the modified or the original class will not have to change it syntax because of the changes we've maid. This is also in-line with the "protected variation" way of looking at OCP :

"Identify points of predicted variation and create a stable interface around them."

While in C# your would maybe want to add a template method or a specialized interface in Ruby you can apply YAGNI and only change the behavior if the need arise

Lastly, we can also extend classes - but OCP lets us do that so again, we are OK.

Nevertheless, we should be careful not to violate Liskov Substitution Principle when we do all that, which I guess is relatively tempting to do if you using Ruby - but that's for another post ...

 

Ok now, that I got your attention, that it isn't dead yet - but we can see a whole class of applications (maybe a couple of classes) where the importance of the RDBMS as we know it today is greatly diminished.
In an article I posted recently on InfoQ, (which I also mentioned in the post on eBay architecture last week ) I discussed the notion of database denormalization on internet-scale sites (such as Amazon, eBay, Flickr etc.). One point of denormalization is immutable data where there isn't a lot of gain in normalization to begin with.
The other thing is entity representation vs. speed. The problem is that joins are slow and sometimes you get to corners where if we want any type of scent speed we need to denormalize. Todd Hoff notes that as well:

The problem is joins are relatively slow, especially over very large data sets, and if they are slow your website is slow. It takes a long time to get all those separate bits of information off disk and put them all together again. Flickr decided to denormalize because it took 13 Selects to each Insert, Delete or Update.

This point is, however, that these "corner cases" get more and more prevalent even in smaller scale application - especially when you have complex entities (as is the case with defense systems for example). Mats Helander, recently wrote a post about saving to Blob, and only adding fields as needed for indexing and identity purposes. Mats also suggest the semi-transparent way of using XML columns where the database can do something with the otherwise opaque data.
This point in fact, demonstrate that the relational data future is indeed not totally secures as we  do see that that leading databases  begin to treat XML data (which is hierarchical and not relational)  as a native citizen - to the point we can even index XML data.

So far we've seen a trend to denormalize more, handle non-relational data, what else? ah transactions
Ive worked on several systems where the data was constantly updated and actually gave the system's representation of the world out-side (of the system) the focus was on availability and latency. Which is again also aligned with the approach taken by the large internet sites which emphasis eventual consistency over immediate consistency.
In distributed systems crashes happen. The RDBMS is show-stopper when it comes to crashes - if we can't commit, we need to stop,roll back. now maybe we can start-over. Is this acceptable? there are many scenarios where it is not. I've seen it in defense systems, in communications systems and even in e-commerce systems (if you are not responsive, I'll just go to the competition).
What do you do in the presence of error? Joe Armstrong suggest the following as the basis for Erlang in his thesis:

To make a fault-tolerant software system which behaves reasonably in the presence of software errors we proceed as follows:

1. We organize the software into a hierarchy of tasks that the system has to perform. Each task corresponds to the achievement of a number of goals. The software for a given task has to try and achieve the goals associated with the task. Tasks are ordered by complexity. The top level task is the most complex, when all the goals in the top level task can be achieved then the system should function perfectly. Lower level tasks should still allow the system to function in an acceptable manner, though it may offer a reduced level of service.The goals of a lower level task should be easier to achieve than the goals of a higher level task in the system.

2. We try to perform the top level task.

3. If an error is detected when trying to achieve a goal, we make an attempt to correct the error. If we cannot correct the error we immediately abort the current task and start performing a simpler task.

On top of that we try to keep any update local i.e. within a task boundary on the hardware where the task occurred - distributing the transactions is not a good option. I outlined why when I talked about SOA and cross-services transactions but the reasoning holds.

Well, truth be said the RDBMS is not dead, its demise probably not even around the corner. Also this does not mean that there aren't any uses for a database. But that's true for other architectural choices. Who ever said that a single tier solution is not the right one for very specific types of system...
RDBMS succeeded to to become the de-facto standard to building system because they offer some very compelling attributes - ACID brings a lot of piece of mind. Large scale systems,low-latency system and fault tolerant systems opt for another set of compelling attributes  (BASE). The point is that  when you design your next solution maybe the conventional database thinking is something that you should at least give another thought to and instead of just following dogma

 

If the previous post made it look like I think everything is just perfect with Ruby - then the answer, in my opinion, is no, it isn't perfect.

For instance, Let's take a look at a basic class like the Array class, which in an effort to be "Humane" violates several OO principles.
For one it violates Don't Repeat Yourself (DRY) by including several methods that do the same (#length and #size for instance) more importantly it violates the Single Responsibility Principle by exposing Stack (#push and #pop methods) and Queue (e.g. #shift).  Making an Array double as a queue is not just violating SRP is also implementation revealing rather than intention revealing.

Another, even more annoying problem is the sometime confusing scoping :
if you do something like
class Foo
    attr_accessor :bar # create accessor methods for a bar attribute
    def barTender(value)
       bar=value
       # do something with bar
    end
end

What happens here is that the bar in bar_tender is a private variable of the bar_tender class so setting it doesn't affect the bar attrribute. If you want to  set the bar property you need to call that with self.bar. Fortunately, the IDE I am using (Netbeans) has recently added a warning for this issue so I won't be doing these again.

I also find it annoying that a dynamic language like Ruby is also case sensitive so if you call something like barTender and bartender you'd get two different methods/variables etc. I guess that's why the Ruby convention is to use all lowercase i.e. bar_tender ( maybe I'll switch to that as well, you know, when in Rome..)

oh yeah and why the hell does  elsif misses the "e" ? :)

And there are of course other things. I do have to say though, that on average I find Ruby code to be more readable than Java or C# - I guess it is easier to define small code DSLs in Ruby (or other dynamic languages) than it is in Java or C# but I'll write about that in a different post

 

I've been taking a look at ruby for a few month, I am finally getting to a stage where (I think/hope) I can actually say something intelligent about it. Last time I had an "aha moment" about a language was the fist time I saw Java.  C++  was oh-so-powerful, but Java was (is)  much more elegant and nice. Now Ruby changes the rules of the game again.
When I first heard about ruby I thought it was just a fad, something that the cool kids are using but its just another language. I getting more and more convinced that it isn't so. I am trying not to get too "silvery-bulletey"  here but working in ruby seems to actually increase productivity.
Why doesn't ruby need DI?

Well, I would say that it all starts at the basics. I remember when I learned OO, I was told objects communicate using messages. I never really understood why they call "method invocation" messages - it doesn't make any sense. The point is that in ruby you really don't "call a method" you "send a message"*.  When you make a call like someVariable.SomeMessage - the ruby interpreter  doesn't really care about the type of someVairable just that the object it holds (and everything is an object) has some entry which can handle SomeMessage.
Let's start with a simple example
consider the following code:

class Foo
  def bar
    puts "Foo-bar"
  end
end

class Foosa < Foo
end

class Baruser
  def baruse(b)
    b.bar  #dependency
  end
end

bu= Baruser.new

bu.baruse(Foo.new)
bu.baruse(Foosa.new)   # sub-class

Well, nothing particularly exciting here. if you run this  you get foo-bar printed twice. That's very much like the dependency injection you see in .NET or Java
It gets a little more interesting when we consider that the following classes would all work as well

class Foz
  def bar
    puts "Foz-bar"
  end
end

class NoBar
  def method_missing(methodname, *args)
    puts "NoBar" if "bar" == methodname.to_s
  end
end

class MakeBar
   define_method(:bar) {puts "madebar"}
end

bu= Baruser.new

bu.baruse(Foz.new)     # another type altogether
bu.baruse(NoBar.new)   # a class that doesn't have bar method
bu.baruse(MakeBar.new) # a class where the bar method is created programatically
 
We can see from the examples that what ruby does is searching for a handler for the bar message. The handler can be a method (symbol) called bar or a generic handle like Missing_Method - ruby doesn't care as long as the message get handled

I think that's pretty nice, we have a lot of flexibility on the dependency side but the depending class still  essentially gets the dependency by injection (the call to baruse)
Well, ruby can help us flex the dependent side as well. The answer is in the last example which uses (an overly simple an uninteresting example of) meta-programming Consider the following code example using Mocha which is a Mock object library for ruby
 
Lets say we modify our Baruser class to the following.

class Baruser
  def initialize
    buildfoo()
  end
  def baruse
    foo = buildfoo()
    foo.bar  #dependency
  end
 
 private 
  def buildfoo
    Foo.new #dependency
  end
end

Note that buildfoo is private - if we wanted to test this in.NET we need to have Foo around. i.e. we can no longer test Baruser by itself
if we use Mocha in ruby we can do the following:
foo = mock('foo')
foo.expects(:bar).at_least_once.

bu = Baruser.new
bu.stubs(:buildfoo).returns(foo) # basically what happens here is that the instance we have is changed
bu.baruse

I can't believe they invented it  - You can get  ruby for just the price of download and if you call within the next 15 minutes we'll throw in a copy of gems free of charge :)

In any event what we see here that using meta-programming and other ruby constructs we can forgo using DI altogether - no wonder Neil Ford defined DI as :

"Dependency Injection enables a vitally important but nevertheless weak, limited, syntactically confounding, and dauntingly complex form of one of the kinds of meta-programming that should exist in the language."

 Not to mention that the resulting code is much more elegant. which is actually what I like best about ruby, the code is much cleaner (but that's for another post)

Some closing thoughts

• .NET 3.5 bring some of the ruby goodness to C# - but as the previous post demonstrated it is just a move in the right direction but not the whole thing
• While I am on the subject of the previous post the whole interface vs. class thing is, of course, a moot point in Ruby since there are no interfaces. Interfaces, like DI, are another thing that is very important in C# and Java and not needed in Ruby
• I've read some complaints on Ruby's performance. performance is important but there are two things to remember. First, the fact that a solution is not the fastest doesn't mean that it isn't fast enough. Second, I can still vividly recall the performance benchmarks for our Java code before we got the first hot spot compiler installed. The point is, that if it is important to enough people it will get better.

---

* I know, I know, Smalltalk had it since the beginning of time. However in Smalltalk everything is an object, in ruby you can also write plain scripts and (more importantly for me) - I never really took more than a cursory look at smalltalk  so I never  saw that.

 

 

One of the interesting presentations I attended in Architecture & Design world was "User Interface Principles in API design" by Elliotte Rusty Harold.
Elliotte started off by mentioning that developers are people too (I guess...) and that designing an API we need to think about the people (developers) who are going to use it.  I totally agree, in fact, when it comes to API design is one place I insist developers use TDD even if they don't do it on their day-to-day development. Dogfooding their own API is probably the only way to really ensure you get a usable API. Elliotte thinks you should go further than TDD and write some sample programs first.

The challenges of API or library design come from the fact that you cannot be sure about Elliotte mentioned some of the fundamental principles of UI design that apply to API design.

• The importance of consistency
• Simple is better (and for god sake please keep complexity internal) which translated to YAGNI and "when in doubt, leave it out")
  
• Smaller surface (API) is easier to use.

To demonstrate the point of proper API design Elliotte compared the complexity of JMidi vs. the elegance of JFugue. While in JFugue you just use a couple of lines to play a few notes:

Player player = new Player();
player.play("C D E F G A B")

JMidi required more than 20 lines of setting all sorts of things related to its internal structure.

Elliotte also had a lot of practical advice on varios aspects like maintenance, specific advice for Java, specific advice for .NET etc. - Instead of me reiterating it I suggest you just take  a look at the presentation which is available on his site.

One point Elliotte made which I don't completely agree with was that you should prefer classes over interfaces. we discussed this issue after the presentation and I said that it is true for libraries but I am not sure that it is always true for frameworks (see Frameworks vs. Libraries for more details on the difference). in IoC scenarios interfaces allows extending the framewok for use with user defined types. One real-life example I offered was a plug-in framework we developed internally a few years ago. This framework incorporated a fixed set of components based on certain 3-rd party components (grid, tree etc.) - This worked very nicely, until in one project we wanted to use other 3rd party components (let's say a ribbon or another grid) which left us with either waiting for the infrastructure guy to find the time to make the changes, fork the infrastructure code, or build something new (we went with building something new since working with end-users we found we actually  didn't need most of the capabilities the framework offers like security/roles, role tayloring capabilities etc.)

On the other hand one argument for using classes over interfaces that Elliotte is right about is that when you use interfaces and then make a change in that interface. The chage  breaks all clients. If however all the clientes used a subclass they would automatically inherit the new functionality.

I thought I could work around this limitation in .NET 3.5 (or whatever microsoft will end-up calling it) by using extension methods so I did something like the following (tests etc. are excluded for brevity)

  class Program
    {
        static void Main(string[] args)
        {
            testInterface tc = new testClass();
            tc.DoSomthing();
            tc.DoSomethingElse();
            
        }
    }

    interface testInterface
    {
        void DoSomthing();
        void DoSomthingElse(string test); // compilation error....
    }

    static class testInterfaceDefaultBehaviour
    {
        public static void DoSomethingElse(this testInterface  ti)
       {
           System.Console.WriteLine("Doing something else");
       }
 
    }

    class testClass : testInterface
    {
        public void DoSomthing() 
        {
            System.Console.WriteLine("Doing something here...");
        }
    }

I was wrong....

Main does  recognized the DoSomethingElse pretty well - however adding it to the Interface definition caused a compilation error.
This is a shame since the MSDN documentation says that

 " In effect, extension methods make it possible to extend existing types and constructed types with additional methods."

If it had worked that would have let us have something like method_missing in ruby in C#. Well, .NET 3.5 is still in beta so maybe it would change (not counting on that)