One of the most interesting presentations in Architecture & Design world was the eBay Architecture presentation by Randy Shoup and Dan Pritchett. The presentation was only one hour long, so Randy and Dan didn't cover all the topics in the slide. Here are some of the insight I took from this presentation.

Architecture evolution  -  eBay actually went through several architecture revolutions. Their initial architecture cannot even begin to scale to their current loads. It was, however, a very good fit for their initial quality attributes - specifically, the emphasis on time to market and costs.  This shows the importance of balancing quality attributes. Sure an architectural change is painful but if they'd future proofed too much I doubt they would ever get something working.

V2 demonstrated that traditional 3-tier architecture would only scale so far. It was nice to see how it evolved though. Also with the move from version 2.4 to 2.5 and later to 3 we see eBay learning about  CAP - the hard way. In its final (current) incarnation eBay's data architecture prefers partitioning and availability over consistency. This doesn't mean they forgo consistency altogether - just that they trade the comfort zone of ACID transactions with the BASE approach. Where BASE - stands for Basically Available, Scalable/Soft state & Eventually Consistent. .
eBay partitions thier data in two levels one is a SOA like division by business areas (users, items etc.) and the second level is an horizental partitioning based on access paths.This BASE approach to data was dubbed by Dathan Pattishall (from Flicker and Friendster) as sharding (via HighScalability). This approach means things like high partitioning, no distributed transactions (also see below), denormalization etc. (you might also want to read the item I wrote on denormalizaiton in InfoQ yesterday).
The more major implication here is that when it comes to internet scale, the database looses its importnace - or as Bill de Hora nicely puts it:

The use of RDBMSes as data backbones have to be rethought under these volumes; as a result system designs and programming toolchains will be altered. When the likes of Adam Bosworth, Mike Stonebraker, Pat Helland and Werner Vogels are saying as much, it behooves us to listen.

As I said the data architecture of eBay is SOAish -  partitioned their components and data along business lines, and they apply many of SOA principles. They don't however unite data and components to create a service and  they don't (seem) to have the same contract boundaries that SOA promotes (Randy told me that they are currently contemplating SOA).

Returning to the  eBay do not use transactions. "no transactions" which seems very controversial  - but if we just consider some of the points I made on transactions between services in previous posts - it is the only logical way to ensure scaling. By the way, as can be expected  they do use transactions - when they are local e.g. if the users table is spread over a couple of table both will be updated together).

The application layers also follow the segmentation by business areas. eBay cacse metadata/immutable data as much as possible. keep the application stateless (i.e. state comes from client/db) e.g. they don't use sessions. The DAL virtualized the horizontal partitioning mentioned above for the rest of the code.

It was also interesting to  that eBay developed its own messaging infrastructure - though Randy and Dan did not provide alot of details on that

Development process - It seems that eBay is using some hybrid of feature driven development with waterfall (i.e. the development is feature by feature - but the development of a feature is waterfallish). The do have a constant delivery rate which they synchronize using the concept of a train. if you have a features that is it will be added to the train which is scheduled to arrive around the time your feature will be ready. Several features are delivered as a package which gives a predictable (weekly). I guess it also gives them some nice metaphors to use such as a feature that doesn't make it - misses the train or the train leaves on time etc.

The slides of the presentation can be downloaded from Dan Pritchett's site (They not from the same event but they are pretty much the same slides. Also you can read Elliotte Rusty Harold's account of the presentation.

If the previous post made it look like I think everything is just perfect with Ruby - then the answer, in my opinion, is no, it isn't perfect.

For instance, Let's take a look at a basic class like the Array class, which in an effort to be "Humane" violates several OO principles.
For one it violates Don't Repeat Yourself (DRY) by including several methods that do the same (#length and #size for instance) more importantly it violates the Single Responsibility Principle by exposing Stack (#push and #pop methods) and Queue (e.g. #shift).  Making an Array double as a queue is not just violating SRP is also implementation revealing rather than intention revealing.

Another, even more annoying problem is the sometime confusing scoping :
if you do something like
class Foo
    attr_accessor :bar # create accessor methods for a bar attribute
    def barTender(value)
       bar=value
       # do something with bar
    end
end

What happens here is that the bar in bar_tender is a private variable of the bar_tender class so setting it doesn't affect the bar attrribute. If you want to  set the bar property you need to call that with self.bar. Fortunately, the IDE I am using (Netbeans) has recently added a warning for this issue so I won't be doing these again.

I also find it annoying that a dynamic language like Ruby is also case sensitive so if you call something like barTender and bartender you'd get two different methods/variables etc. I guess that's why the Ruby convention is to use all lowercase i.e. bar_tender ( maybe I'll switch to that as well, you know, when in Rome..)

oh yeah and why the hell does  elsif misses the "e" ? :)

And there are of course other things. I do have to say though, that on average I find Ruby code to be more readable than Java or C# - I guess it is easier to define small code DSLs in Ruby (or other dynamic languages) than it is in Java or C# but I'll write about that in a different post

I've been taking a look at ruby for a few month, I am finally getting to a stage where (I think/hope) I can actually say something intelligent about it. Last time I had an "aha moment" about a language was the fist time I saw Java.  C++  was oh-so-powerful, but Java was (is)  much more elegant and nice. Now Ruby changes the rules of the game again.
When I first heard about ruby I thought it was just a fad, something that the cool kids are using but its just another language. I getting more and more convinced that it isn't so. I am trying not to get too "silvery-bulletey"  here but working in ruby seems to actually increase productivity.
Why doesn't ruby need DI?

Well, I would say that it all starts at the basics. I remember when I learned OO, I was told objects communicate using messages. I never really understood why they call "method invocation" messages - it doesn't make any sense. The point is that in ruby you really don't "call a method" you "send a message"*.  When you make a call like someVariable.SomeMessage - the ruby interpreter  doesn't really care about the type of someVairable just that the object it holds (and everything is an object) has some entry which can handle SomeMessage.
Let's start with a simple example
consider the following code:

class Foo
  def bar
    puts "Foo-bar"
  end
end

class Foosa < Foo
end

class Baruser
  def baruse(b)
    b.bar  #dependency
  end
end

bu= Baruser.new

bu.baruse(Foo.new)
bu.baruse(Foosa.new)   # sub-class

Well, nothing particularly exciting here. if you run this  you get foo-bar printed twice. That's very much like the dependency injection you see in .NET or Java
It gets a little more interesting when we consider that the following classes would all work as well

class Foz
  def bar
    puts "Foz-bar"
  end
end

class NoBar
  def method_missing(methodname, *args)
    puts "NoBar" if "bar" == methodname.to_s
  end
end

class MakeBar
   define_method(:bar) {puts "madebar"}
end

bu= Baruser.new

bu.baruse(Foz.new)     # another type altogether
bu.baruse(NoBar.new)   # a class that doesn't have bar method
bu.baruse(MakeBar.new) # a class where the bar method is created programatically
 
We can see from the examples that what ruby does is searching for a handler for the bar message. The handler can be a method (symbol) called bar or a generic handle like Missing_Method - ruby doesn't care as long as the message get handled

I think that's pretty nice, we have a lot of flexibility on the dependency side but the depending class still  essentially gets the dependency by injection (the call to baruse)
Well, ruby can help us flex the dependent side as well. The answer is in the last example which uses (an overly simple an uninteresting example of) meta-programming Consider the following code example using Mocha which is a Mock object library for ruby
 
Lets say we modify our Baruser class to the following.

class Baruser
  def initialize
    buildfoo()
  end
  def baruse
    foo = buildfoo()
    foo.bar  #dependency
  end
 
 private 
  def buildfoo
    Foo.new #dependency
  end
end

Note that buildfoo is private - if we wanted to test this in.NET we need to have Foo around. i.e. we can no longer test Baruser by itself
if we use Mocha in ruby we can do the following:
foo = mock('foo')
foo.expects(:bar).at_least_once.

bu = Baruser.new
bu.stubs(:buildfoo).returns(foo) # basically what happens here is that the instance we have is changed
bu.baruse

I can't believe they invented it  - You can get  ruby for just the price of download and if you call within the next 15 minutes we'll throw in a copy of gems free of charge :)

In any event what we see here that using meta-programming and other ruby constructs we can forgo using DI altogether - no wonder Neil Ford defined DI as :

"Dependency Injection enables a vitally important but nevertheless weak, limited, syntactically confounding, and dauntingly complex form of one of the kinds of meta-programming that should exist in the language."

 Not to mention that the resulting code is much more elegant. which is actually what I like best about ruby, the code is much cleaner (but that's for another post)

Some closing thoughts

• .NET 3.5 bring some of the ruby goodness to C# - but as the previous post demonstrated it is just a move in the right direction but not the whole thing
• While I am on the subject of the previous post the whole interface vs. class thing is, of course, a moot point in Ruby since there are no interfaces. Interfaces, like DI, are another thing that is very important in C# and Java and not needed in Ruby
• I've read some complaints on Ruby's performance. performance is important but there are two things to remember. First, the fact that a solution is not the fastest doesn't mean that it isn't fast enough. Second, I can still vividly recall the performance benchmarks for our Java code before we got the first hot spot compiler installed. The point is, that if it is important to enough people it will get better.

---

* I know, I know, Smalltalk had it since the beginning of time. However in Smalltalk everything is an object, in ruby you can also write plain scripts and (more importantly for me) - I never really took more than a cursory look at smalltalk  so I never  saw that.

 

One of the interesting presentations I attended in Architecture & Design world was "User Interface Principles in API design" by Elliotte Rusty Harold.
Elliotte started off by mentioning that developers are people too (I guess...) and that designing an API we need to think about the people (developers) who are going to use it.  I totally agree, in fact, when it comes to API design is one place I insist developers use TDD even if they don't do it on their day-to-day development. Dogfooding their own API is probably the only way to really ensure you get a usable API. Elliotte thinks you should go further than TDD and write some sample programs first.

The challenges of API or library design come from the fact that you cannot be sure about Elliotte mentioned some of the fundamental principles of UI design that apply to API design.

• The importance of consistency
• Simple is better (and for god sake please keep complexity internal) which translated to YAGNI and "when in doubt, leave it out")
  
• Smaller surface (API) is easier to use.

To demonstrate the point of proper API design Elliotte compared the complexity of JMidi vs. the elegance of JFugue. While in JFugue you just use a couple of lines to play a few notes:

Player player = new Player();
player.play("C D E F G A B")

JMidi required more than 20 lines of setting all sorts of things related to its internal structure.

Elliotte also had a lot of practical advice on varios aspects like maintenance, specific advice for Java, specific advice for .NET etc. - Instead of me reiterating it I suggest you just take  a look at the presentation which is available on his site.

One point Elliotte made which I don't completely agree with was that you should prefer classes over interfaces. we discussed this issue after the presentation and I said that it is true for libraries but I am not sure that it is always true for frameworks (see Frameworks vs. Libraries for more details on the difference). in IoC scenarios interfaces allows extending the framewok for use with user defined types. One real-life example I offered was a plug-in framework we developed internally a few years ago. This framework incorporated a fixed set of components based on certain 3-rd party components (grid, tree etc.) - This worked very nicely, until in one project we wanted to use other 3rd party components (let's say a ribbon or another grid) which left us with either waiting for the infrastructure guy to find the time to make the changes, fork the infrastructure code, or build something new (we went with building something new since working with end-users we found we actually  didn't need most of the capabilities the framework offers like security/roles, role tayloring capabilities etc.)

On the other hand one argument for using classes over interfaces that Elliotte is right about is that when you use interfaces and then make a change in that interface. The chage  breaks all clients. If however all the clientes used a subclass they would automatically inherit the new functionality.

I thought I could work around this limitation in .NET 3.5 (or whatever microsoft will end-up calling it) by using extension methods so I did something like the following (tests etc. are excluded for brevity)

  class Program
    {
        static void Main(string[] args)
        {
            testInterface tc = new testClass();
            tc.DoSomthing();
            tc.DoSomethingElse();
            
        }
    }

    interface testInterface
    {
        void DoSomthing();
        void DoSomthingElse(string test); // compilation error....
    }

    static class testInterfaceDefaultBehaviour
    {
        public static void DoSomethingElse(this testInterface  ti)
       {
           System.Console.WriteLine("Doing something else");
       }
 
    }

    class testClass : testInterface
    {
        public void DoSomthing() 
        {
            System.Console.WriteLine("Doing something here...");
        }
    }

I was wrong....

Main does  recognized the DoSomethingElse pretty well - however adding it to the Interface definition caused a compilation error.
This is a shame since the MSDN documentation says that

 " In effect, extension methods make it possible to extend existing types and constructed types with additional methods."

If it had worked that would have let us have something like method_missing in ruby in C#. Well, .NET 3.5 is still in beta so maybe it would change (not counting on that)

I won't say anything about my presentations (that's for others to say :) ). The point of this post is just to let you download them. So here they are:

• SOA Patterns (2.14mb) - Takes a look at different strategies (patterns) to solve common SOA pitfalls
  
• Getting SPAMMED for architecture (4.56mb) - Takes a look at the activities architects can/should do when they think about software architectures. The presentation also covers architecture in agile projects.
  

Just got back from Chicago, where I attended Architecture & Design World. The event was great. it had many interesting talks and sessions  which I'll blog about in  few of the following posts (Also I'll upload the slides from my sessions later this week). One session however, was very disappointing for me, Ivar Jacobson's Keynote on "Next Generation Process with Essential Modeling".

I was really looking forward for this keynote; After all Ivar is one of the "Three Amigos", father of use cases and all. Also I had the chance to attend a couple of his presentations several years ago and he had a very interesting and convincing appearances at those times. Unfortunetly this session was nothing like those previous events.
Ivar talked about his new methodology. The basic idea behind the essential software process is (in my opinion) correct. Instead of big bang prescribed processes - it is better to tailor a process from a set of practices to get something that fits the organization/project at hand. It might be a little unnerving to hear this from one of the people who brought (upon) us the RUP, but, I guess it is good to see people who are constantly learning.

The problem was that while the main idea can be summed up in one line (see above) Ivar went on fro the better part of the hour to explain how cool it was that his base set of practices comes on a set of que cards that can be sorted and arranged. He also went on to explain the game board (or something to that effect) where you can lay different cards to both describe your current process as well as the bunch of practices you want to use for your future process.
If that wasn't enough, when he (briefly) shown us the cards that talk about the architecture and architecture description practices the advice/guidance there was very mediocre and general
I expected much more from someone of the stature of Ivar Jacobson.
On the up-side this was only one keynote and the other sessions I listened to where much better - as I already said, I'll blog about few of them in upcoming posts.

While I am getting ready to fly to A&D world 2007 where I'll present both SOA patterns and the SPAMMED architecture framework, I thought I'd throw in a little update on the book as well.

I've made a small change to the way chapters 5-7 are organized. They are now grouped under a separate part called "Service Interaction Patterns" (and chapters 2-4 are grouped under "Structural Patterns").

• Chapter 5 is focused on Message Exchange Patterns (MEP): synchronous, asynchronous, events and transactional  - The patterns there are not new for SOA, instead the focus is on the meaning of implementing the usual MEPs under SOA constraints. I sent it to manning early last week so hopefully it would be available on MEAP soon.
  
• Chapter 6 is called "Consumer Interaction patterns" and includesthe UI interaction patterns as well as interaction pattern with other types of consumers. This is the chapter I am currently working on.
  
• Chapter 7 is unchanged for now

Lastly,  as you may remember,  I publish online one pattern from each chapter so I'd be happy to get comments on which of the following three patterns (from chapter 6) you like  to see on-line: Reservation pattern (making partial commitments), Client/Server/Service (integrating Legacy or thin clients with SOA) , Client/Service (integration Rich clients with SOA) - if you want to vote just send me an email or leave a comment

I've got a few emails from people who asked about the commit and consensus protocols (Paxos, 3PC etc.) mentioned in the previous post.
I was going to write something up but then I find a very good overview by Mark MC Keown which provides "A brief history of Consensus, 2PC and Transaction Commit". This short writeup provides links to the most important papers like the Fischer, Lynch and Paterson (FLP) "Impossibility of distributed consensus with one faulty process", Jim Gray (who, unfortunately, was lost at see earlier this year)  and Leslie Lapmort's "Consensus on Transaction Commit". In addition to Mark's post and all the papers there you may want to check out the presentation Jim and Leslie made on paxos commits

By the way, if on the other hands, you already know all this (and a little more) than check out Werner Vogel's (Amazon CTO) blog as there's a interesting  job opening for you there

Following the previous post I had a chance to exchange a few email with Mark Little (the director of engineering in the JBoss division of Red Hat). Mark thinks the topic of transactions and SOA has been beaten to death already and wonder's why does it need to resurface (see his post "Is anyone out there?") - I don't see a problem with discussions resurfacing when new people are faced with situations others already solved (but that's a matter for another post)

Anyway, the reasons we're here is  that I think that during this conversation  mark made a few interesting observations and I think the end result is pretty interesting. I decided (with his permission) to post it here ( It is only minimally edited: no deletions, few additions (in []) and a few time shifting to make it more coherent as a single conversation)

Mark : From what I can see it's [the arguments on transaction and services - are] the same old arguments that have gone round and round, ignoring the important fundamental issues and not doing enough background research.
Sagas are transactional - it's just an Extended Transaction model and not an ACID transaction model. Don't get hung up on the word "transaction", which is way to overloaded in our industry to actually mean anything by itself. Plus, 2PC is a consensus protocol too; it does not impose any other aspect of ACID than the A. Even the D is optional until/unless you want to tolerate failures.

Arnon I know this is an old argument - but that doesn't mean it isn't worthwhile

Mark It isn't worthwhile if people aren't going to listen I've been involved in these debates so many times over the past 7 years (for Web Services transactions) and longer for extended transactions, that it gets a bit old after a while. Maybe we should create a wiki page and point people at that ?

Arnon I guess, but you should keep in mind that people who are solely in the .NET camp only got WS-AT recently with Windows Communications Foundation so you can expect the issues to resurface. By the way a wiki might not be a bad idea 

[regarding 2PC.] 2PC is a distributed consensus protocol and in principle doesn’t have to be related to ACID transaction. But I think the common view and use of it is for ensuring distributed ACID behavior. Looking back at my experience with XA and COM+ transactions it seems it does a good job at achieving this ACIDness


Mark This is an education issue. The literature is clear on this. People who know and understand transactional protocols don't make the mistake of equating 2PC to ACID properties.

Arnon Yes it is an educational issue . But I am not sure that it is that common knowledge. It is expected that middleware vendors who build the tools to support these protocols to understand it better - I don’t think it is that widely known outside these circles. Most of the architects I’ve don’t (maybe It time to look for new friends )

By the way as 2PC is not resilient to failures of the coordinator so in a highly distributed environment like SOA it might have been a better idea to go with paxos commits if at all you go down that path.

Mark The reason WS-AT and WS-ACID chose 2PC is: interoperability. All TP monitors support it. Try getting IBM, MSFT, Oracle, BEA etc. to change to Paxos, 3PC, flat-commit, or anything else and you'll be waiting for the heat death of the universe.

Arnon Can’t argue with that

Mark [also] 2PC is resilient to failures if the coordinator eventually recovers. Paxos has its own failure assumptions too: Jim never disputed this. Same as 3PC and other consensus protocols. As with *any* fault tolerance approach (transactions, recovery blocks, replication, etc) it's always probabilistic. All we're doing is making it highly unlikely that the system cannot complete, but we can never make it entirely safe. Even in the airline industry they can "only" go to a probability that failures happen .000000001

Arnon You are probably right that in an SOA situations the chances of not getting an ACID transaction are worse than in a controlled environment - which actually make the situation even worse since people using WS-AT perceive it as allowing them ACID interaction (e.g. Juvals podcast) .


Mark WS-AT is *all* about ACID, in the same way WS-ACID is about ACID transactions. It is *nothing* to do with SOA though. Web Services are not purely the domain of SOA implementations!

Arnon I totally agree that Web-services and SOA are not directly related and can each exist independently of each other. Again this is an educational issue but,  SOA==Web-services is a very common misconception (I guess the word “service” in web-service doesn’t help )

[in any event] I think distributed transactions in general should be used carefully period.


Mark Absolutely. They are not a global panacea and people who push them as such do more harm than good.

Arnon WS-AT is more problematic than regular distributed transactions as by definition in an SOA you do not know who and how many other services will participate in your transactions so you are much more likely to run into problems.

Sagas which embrace the temporal shift don't give an illusion of ACIDness and allow to focus on achieving distributed consensus while keeping all parties involved consistent. I think that it is a much better option if you need transaction-like behavior

Mark For SOA, yes. Although Sagas are only good for a certain type of use case. That's why we've always tried to develop "live documents" that allow people to add new models when/if needed. With a couple of exceptions during the BTP days, there has always been consensus that one size does not fit all (http://www.webservices.org/weblog/mark_little/blackadder_and_the_micro_kernel_approach_to_web_services_transactions).

Web Services *anything*, whether it's WS-AT, WS-Sec, or WS-Addressing all have their non-SOA aspects because Web Services aren't developed purely with SOA in mind. If that were to happen then Web Services as a technology would lose some of their important benefits immediately.

Arnon This whole discussion is in the context of SOA (at least from my side ) – naturally there’s a place for ACID transactions for other uses.

Regarding Sagas - calling them "Extended transactions that are not ACID" is just semantics - my point was that they are not ACID transaction. I think most people equate transactions with ACID transactions as well (but I may be wrong)

Mark Many people do and that again is an education problem. The term Extended Transactions (don't need to say "that are not ACID") has a well defined meaning in the R&D community. There have been many good models and implementations around Extended Transactions. They really took off in the vendor community through the Additional Structuring Mechanisms for the OTS, back in the 1990's. If you check that out you'll see that it formed the basis of WS-TX and WS-CAF. Even in Jim's original technical report he discussed relaxing all of the ACID properties in a controller manner to get more flexibility. That was the first extended transaction. In fact, ACID transactions are just one type of extended transaction. There are many many others, including nested transactions, coloured actions, epsilon transactions, sagas etc.

Unless I qualify it beforehand, I try to never use the term "transaction" in isolation because it has different meanings to different people. For example, when talking to developers working in trading infrastructures, a "transaction" isn't an ACID transaction at all. In telecos it's different again.

Evan H asked a question about distributed transactions and services in the MSDN architecture forum:

Are distributed transactions (ie.. WS-Transaction) a violation of the "Autonomous" tenant of service orientation?   Yes or No and Why?  Kudos if you can address concurrency and scalability (in an enterprise with multiple interacting services).

I answerd this questions back in april when I wrote a couple of posts that explained why cross-service transactions are a bad idea:cross service transactions and some more thoughts on cross service transactions.
Roger Sessions also agrees with this view (well, it seems actually, he wrote about it well before I did :) ):

When the WS-Transaction specification was first proposed, back in 2002, I wrote an article explaining why I thought the idea of allowing true transactions to span services was a bad idea. I published the article in The ObjectWatch Newsletter, #41: http://www.objectwatch.com/newsletters/issue_41.htm. Nothing since then has changed my mind. Atomic transactions require holding locks, and spanning transactions across services requires allowing a foreign, untrusted service to determine how long you will hold your very precious database locks. Bad idea. Just because IBM and Microsoft agreed on something doesn't make it good!

The reason I am bringing this issue back is that Juval Lowy (who wrote the article that triggered my first post on the subject) has recorded an Arcast with Ron Jabobs. Where he re-iterated the idea that "Transactions is categoricaly the only viable programming model" and you should strive to use it whenever you can. It seems Juval admits you sometimes need to use Sagas (which he called "long running transactions" - you can see in my link why I think that's a wrong name). He also agrees that you can also use a transactionable transport and then only do internal transactions from each service to the transport (a pattern I call "Transactional Service"). However, at the end of the day, he still thinks you should use WS-AtomicTransactions whenever you can.

I agree that transactional programming is important. I think it is the simplest programming model (from the developers side). I would probably never write an interaction with a database that is not transactional; I look very favorably at initiatives for in-memory ACI (no Durability) transactions such as the one Ralf talks about.  Until we get to Distributed Transactions...

First, we should note that transactions are not "the only viable" option.As Martin Fowler notes Ebay seems to be doing fine without distributed transactions. Not only that, they abandoned distributed transaction and went "transactionless"because they needed one simple thing... Scalable performance .

In most COM+ scenarios you have a single server or a few internal servers where the distributed transaction happen - and even there you should plan your transactions carefully if you want to get any kind of decent performance. In SOA scenarios the situation is more complicated as the distribution level is expected to be higher (even if you don't involve services from other companies). More distribution means longer times to complete transactions (especially if a participant can flow the transaction and extend it). It also means increasing the chances of failure (see Steve Jones series of posts on five nines for SOA). In my opinion, the more distributed components you have the more you want their interaction to be decoupled in time - i.e. the opposite of transactions.

Juval also said he doesn't buy the denial of service problem I mentioned (supporting a transaction means you allow locks - if an external party doesn't commit you retain the lock..). Juval said he assumes that a solutions has both authentication and authorization so this shouldn't be an issue. For one, I have seen too many projects where security was something that was neglected or quickly patched in at the latest moment - so I would hardly assume security. Even with security on - you increase your attack surface.
But that's just the half of it. Even if all your service consumers have good intentions - you still don't know anything about their code. SOA is not like the "good old days" where you owned the whole application  - this means you cannot trust their security to be ample. Also you don't know anything about their code quality. Services are likely (in the general case) to be deployed on different machines, even if they start co-located. I think that a Service boundary should be treated as a trust boundary just like a tier boundary. I strongly believe you should have reduced assumptions on what's on the other side of the service's boundary - transactions are not reduced assumptions

SOA and distributed transactions do not go hand in hand - it isn't just autonomy at stake here. It is a problem for performance and scalability and even security period.

To finish this post - I would also highly recommend looking at Pat Helland's paper "Life Beyond Distributed Transactions: an Apostate's Opinion" and a post he recently made called  "SOA and Newton's Universe", where he explains more eloquently than I ever could why SOA is not a good fit for distributed transactions.

Steve Jones has (yet another) great post called "Le Tour SOA - why support services are critical, but not important".
You should go read the article - but in a nutshell, Steve explains that important services are the ones that bring business values and critical services are the supporting ones that help keep the light on for the important services to function properly. 

While the post has SOA in the title. I think it is more general and is also applicable to applications or any other IT generated components. In fact it can also be applicable to IT itself as Nicholas Carr noted in 2003 when he published his paper "IT doesn't matter". Nicholas argues that IT will become akin to electricity and as such be critical for the business to continue operating but not important. As a side note I'd say I think this is might be true for traditional businesses but not for businesses where the IT is the business (such as banks, insurance companies, etc.)

Back to critical vs. important - I think this is an important for architects to make this distinction to be able to prioritize work and not confuse business value with semblance of business value due to criticality for operations. This doesn't mean you can neglect critical tasks (after all they are critical...). It is the important stuff that will bring your business the competitive edge.

Coming back from vacation I saw that Jeff Atwood (coding horror) wrote a post on "rethinking design patterns"
a few days ago. Jeff criticizes the GoF design patterns book
Jeff says his two main grips with the book are

1. Design patterns are a form of complexity. As with all complexity, I'd rather see developers focus on simpler solutions before going straight to a complex recipe of design patterns.
2. If you find yourself frequently writing a bunch of boilerplate design pattern code to deal with a "recurring design problem", that's not good engineering-- it's a sign that your language is fundamentally broken.

I agree that these can be a problem with using design patterns in general and the GoF ones in particular. I would add to this that I think that using design patterns just to be able to say you've used them is very wrond as well (this may sound obvious - but I have seen organizations where this was encouraged).

I don't think however, any of those problems are problems with the patterns themselves. This is not to say that the GoF book is perfect.Looking back at the GoF book I think that it isn't clear of problems. For instance, I don't think all the patterns stand the test of time. The most prominent example for that is the Singleton pattern, which I hardly recommend using these days. Singletons are problematic for testing and  create a tight coupling for a specific instance. There are better ways to create a mono-instance if that's what you need (I think others have posted about it in the past - but I can post about it separately if needed)
Also, newer languages sometimes provide better ways to tackle some of the design patterns solutions - see for example a recent post by Alex Miller where he talks about an alternative for Template Method.
I can even say, that not all the patterns are that useful (Flyweight comes to mind as an example).

Nevertheless, I still think that the GoF book is one of the most important books in computer science. First it is a seminal work which introduced the patterns thinking into software development. Today we have literally hundreds of patterns on all subjects and technologies. I think it is a very good think since the looking at a problem from a pattern perspective gives us more depth and understanding on both the problem and the solutions vs. other ways I've seen.
Even in itself, the GoF book is great, since many of the patterns are very valuable and can help us solve real problems. we just need to keep in mind that the sample implementation in the book is just that - sample. There can be more than one way to code a pattern and still gain the benefits (these are "design" patterns not "coding" patterns). A few weeks ago, someone asked a question on the visitor pattern in one of the forums I monitor. The guy needed to add an additional parameter to the Visit method and asked if it wasn't a violation of the pattern. I told him that there isn't such a thing as "violating a design pattern". The patterns are a means to an end not some coding codex we should keep. I think, that if we treat the design patterns as pieces of knowledge
rather then a holly script, they can really help avoid some stupid mistakes.

Thus, at the end of the day, I still think the Gof book is required reading for any new developer. But hey, that's just my .2 cents :)

A couple of quick  observations following the Events and temporal coupling post

Events, Current data and aggregated data all have Time-to-live aspects.

• Events value usually diminishes over time until the TTL reaches
• Current data usually have a constant value while their TTL lasts (until a new value is the current data) - unless we are talking about version data which is a  component of or a step  in the direction of aggregated data.
• Aggregated data  has the longest TTL, it is interesting to note that its value increases over time

Also while the Current data TTL is determined by the producer both Events and Aggregated data TTLs are determined by consumers

Yeah, I know these are not not earth shattering observations but I still think they are  interesting
 

It was more than 10 years ago that I got my first MVP award from Microsoft (I was probably the first Israeli MVP, because few years later Microsoft Israel awarded what they said werethe first Israeli MVP awards... :) )
Anyway, now, for the forth time, I got awarded again - this time it is in the Solution Architect category -
Thanks

You raise an event when something interesting happens to you, you think it is important, but you don't care enough to know who is interested. you are even less interested in to personally going and to each an every interested party and letting them know. So - instead, you raise and event, and let the poor buggers take care of any implications by themselves. We raise the event "now" when the change happened - it is only important now anyway...


Looking from the "poor buggers" -the event consumer point of view things are more complicated. There are events which are cyclic in nature like stock price updates, the blips from a sonar etc. if you missed one, then it isn't really important you'd get the right information in the next update (actually, that isn't entirely true - see later in this post). Then, there are the events which only occur once. sometime it isn't important for you to listen to them if you are not up and running in the same time. Other times you can't afford to lose an event for instance if your ordering service (or component for that matter) communicates with the invoicing one using events you don't want to miss the event of a new order else you would loose money.

This basically means that the event producer and the event consumer are coupled in time - one way to solve it is to make sure both of these services are available at the same time i.e. if the invoices crashed, then processing orders should be suspended (note that this doesn't mean that you don't accept orders just that you don't process them).
Ok - maybe we can just raise the event "transactionally" - this would probably work, but we need to remember that the event producer doesn't really care about the event consumers, why would it want to fail because of them?!
Maybe a better way would be to "raise" the event over some reliable transport  - this has a few problems. one is that we've passed the problem to the connection between the event producer and the transport. It might be acceptable to have a transaction between the event producer and the transport. However, as I've already said the producer doesn't care much about the consumers..
We can have persistent subscriptions for existing consumers to prevent events from getting lost which make both creates a er minor problem that new consumers can't see past events but also has the risk of existing subscribers disappearing and their queue can then grow endlessly (or until an administrator would remove the subscription)

Ok, let's try to look at the problem from a different angle. looking at the events, what we can really see is that an event has a time-to-live (TTL) as far as the event consumer is concerned. For instance in the case of the cyclic events the TTL is the interval until the next  event. Actually, even with cyclic events the TTL might be larger - if we are also interested in analyzing trends or  ab normal occurrences (which is why I said it isn't entirely true we don't care about old events). In case of one-time events the TTL might be indefinite or maybe even then it might be some definite value (one day, week, year etc.). Since we can't know about the TTL of consumers it can be a good idea to make past events available somehow.

Thus, when you design an event centric architecture like  EDA (whether on top of SOA or not)  it is important to think about event consumers - we don't want to think about  specific  consumers since it negates the benefits of thinking in events, but I would say that you want to think about event consumers in general, after all your component is also an event consumer (do unto others as you would have them do unto you)

One option, which I already talked about, is to make past events available as a feed. Event consumers can then come at their own leisure and consume past event  (this can be in addition  to  to raising the events in real-time). This provides a partial solution as the maximal TTL is determined by the event producer (after which the event is deleted from the feed). This may be acceptable but you must be aware of that.
The other option is to to log all the events and provide an API to retrieve past events. In a sense the max TTL is still at the hands of the event producer only if you use a database it would probably be a large time compared with a feed. Alternatively the events can be logged on by a central "always present" event aggregator (in a manner similar to the aggregated reporting pattern I described for SOA).

To sum all this - events they seem only to matter in the instance in time they are created, we are used to that thinking from building OO systems where all the components are co-located in the same address-space and time (even there I can think of scenarios where we would want past events) - in a distributed world events need to have a TTL, the TTLs can vary and are determined by the events consumers. Lastly, as I demonstrated in the paragraph above, there are several strategies we can use to help solve the event TTL dilemma (and there are probably a few others).

Few months ago I wrote here about solving the mismatch between Service Oriented Architecture (SOA)  and Business Intelligence (BI) (see papers and articles section). Recently I got the following question from Ben:

One major question I have is around large data sets. As an experienced BI/DW architect and developer I have worked on a number of large scale data warehouses. Retrieving large data sets (i.e. millions of records) doesn't seem to fit well into SOA. As you state in your article, we could have another point-to-point interface, where the service which houses data we need gets a request and writes out a batch file (xml or plain ascii text). Then using typical ETL, we grab the file and load it. The underlying source system (service) can use optimization in generating a large data set (vs. record by record) and
the data warehouse can correspondingly load in bulk.

Like most architectural questions - the answer is "it depends"
For instance, if you do a run-of-the-mill ETL as a on-time setup then it is just that- a one time setup and I, personally, don't see any contradiction between SOA goals or tenets and that.

I do think that iit is better to enhance SOA with EDA interactions to provide a long term solution to the BI problem. You can also have a dedicated component that aggregated the information that flows in in these events and builds batch files that are suited for the ETL you've used during the setup phase (mentioned above).
It is true though that moving an SOA which is already in-place to EDA is not a small feat, but adding EDA layers does not have to mean that the old interfaces go away - especially not immediately (remember to treat services as products)

If you have a business that generated millions of records on a daily basis - then the situation is more complicated. Now you have to think about the trade-offs between "compromising" SOA and adding a dedicated interface (or a backdoor to the database) for the ETL vs. the implications of performance, bandwidth, transition costs, ROI  etc. of pushing that information with EDA.
 I, personally believe in pragmatism and the "no-silver-bullet" approach so I can't say that EDA is always the best solution (As an aside, this is part of the reason I write my book as patterns not as "best-practices guidance"). You may find that ETL is the best trade off in your situation. Yes I know that it isn't a definitive answer - but real life is (usually) a little more complicated than black and white solutions. As architects we need to find the best trade off for the situation at hand.

I've been working in Rafael for a total of 5 years now and I guess my tenure here is nearing its end.  The latest company-wide reorg has brought with it the end of the biometric product line (or maybe just its evolution into something I don't really like - it isn't completely clear yet, but since I don't like both option it doesn't rally matters).

I am checking several possibilities of joining one of the consulting companies here in Israel - none of the options has fully matured as of yet but I believe that in the next month or so something will be finalized - meanwhile I am also open for other options (my CV is on the about page - hint, hint :) ).

In any event, between the book, DDJ, InfoQ and looking for a new job (not to mention my family) I sure have have my hands full - I guess now I understand why they say "may you live in interesting times" is an old Chinese curse :)

I thought I has this  RESTful web services thing figured out, but following one of the threads on the Yahoo group on Service-Oriented-Architecture I came to the conclusion that maybe I don't.

Steve Jones tried to see if he understands REST by giving an example and that example was corrected by Anne Thomas Manes (who is a research director with the Burton Group which recently stated that the future of SOA is REST).
Here are the examples from the above mentioned thread:

POST http://example.org/customer
HTTP message body contains a representation of "anne"
server creates a subordinate resource called http://example.org/customer/anne

GET http://example.org/customer/anne
returns a representation of "anne"

GET http://example.org/customer/personByName?name=anne
returns a representation of "anne"
or perhaps returns the URI of the "anne" resource
or perhaps returns a list of URIs of all people named "anne"
might also be specified more simply as
GET http://example.org/customer?name=anne

GET http://example.org/customer/personByAge?age=27
returns a list of URIs of people whose age is 27
or perhaps returns a collection of representations of all people aged 27
might also be specified more simply as
GET http://example.org/customer?age=27

PUT http://example.org/customer/anne
HTTP message body contains a representation of "anne"
either creates a new resource called "anne" (if none exists)
or replaces the existing "anne" resource

PUT http://example.org/company/newco
HTTP message body contains a representation of "newco"
either creates a new resource called "newco" (if none exists)
or replaces the existing "newco" resource

If you prefer the server to assign the URI you would instead say

POST http://example.org/company
HTTP message body contains a representation of "newco"
server creates a subordinate resource called http://example.org/company/newco

POST http://example.org/customer/anne?addCompany=http://example.org/company/newco
this would append the newco company reference to the "anne" resource

You can see another example for what I am talking about here on Jon Udell's blog giving an example from RESTful Web Services, by Leonard Richardson and Sam Rubycovering  of doing a transaction in RESTful style

If all these are indeed "legal" or "correct" RESTful interactions I have 2 observations to make
First, I guess Pat Helland is right when he said "Every noun can be verbed" since I don't see the real difference between having a contract with a PersonsByAge request which returns a document* of Persons and a REST request like " GET http://example.org/customer/personByAge?age=27" or even " GET http://example.org/customer?age=27".

The second observation has to do with the so called "uniform interface". I would argue that the resources and their attributes (age=27, name="anne") are the interface. the POST, GET etc. uniform interface does not mean much more than the "uniform" SEND, BROADCAST  interface of messaging.
Further more if resources and their attributes are indeed "the interface" - than not only does REST not have a uniform contract - it actually has a dynamic one which changes in run-time as new resources are created - such as the "POST http://example.org/company"  which creates a new resource "http://example.org/company/newco" in the example above

---

* I think it is very important for SOA to have document oriented messages and not RPC one I\ll blog in a separate post about the differences. for now it is suffice to say that the REST hypermedia notion of returning the URIs of all the relevant persons should also be present (one way or another) in a good document oriented message even if you are using WS-* or plain messaging as transport

In addition to the drafts of selected patterns I publish on my site, you can now purchase my book via the Manning Early Access Program (MEAP).
MEAP means you can get chapter drafts as I write them and the complete book when its done (ebook or printed). Here is Manning's explanation:

"Buy now through MEAP (Manning Early Access Program) and get early access to the book, chapter by chapter, as soon as they become available. You choose the format - PDF or ThoutReader - or both. By subscribing to MEAP chapters, you get an opportunity to participate in the most sensitive, final piece of the publishing cycle by offering feedback to the author. Reader feedback to the author is welcome in the Author Online forum. As new chapters are released, announcements are made in the MEAP Announcement Forum. After all chapters are released, you will be able to download the complete edited ebook. If you order the print edition, we will ship it to you upon release, direct from the bindery, weeks before it is widely available elsewhere.

By the way, this is probably also a good time to mention that I'll be speaking about quite a few of the patterns in Architecture & Design World 2007 which will take place this July.

There is still a lot of work, but I already like to thank all the people in manning that helped me get this far. especially to Cynthia Kane my editor (hey, maybe now she'll give me more slack :) )
Ok, 'nuff blubbering, back to completing chapter 5...

Agile and documentation? what gives? First things first documentation is not something that is prohibited by Agile Manifesto. Working software is definitely preferred over "comprehensive documentation" but there can also be some value in documentation

The first question is why would we want to document anything if we have a working software. I think there are several stakeholders like project newcomers, maintainers etc. who will be interested in something that will let them get up to speed and provide them an overview of what's going on before they delve into code. You can read more on that in  a post I wrote almost a year ago called "Who needs a software architecture document" but in essence the main motivation for documentation is that assuming that the software is successful it will outlive the team - i.e. the people that built the software will not be the ones that will have to develop, maintain and support it for most of the software's life.

If we agree that we need a software architecture document the question is what to document and when.

There are two main "things" you can document in regard to architecture, the first is the obvious one: the architecture itself. In my experience the most value can be derived from documenting recipes i.e. how  to do stuff that is common in the architecture. These recipes can be a short description of the context and then a
pointer to a test (or tests) and an implementation that exercises this. You can think of the recipes as a type of a tutorial to the architecture.

Other documentation worthy elements related to the architecture are an overview and technology mapping (including what a developer needs to install to start working). The overview allows a newcomer to understand where to find what, the technology mapping allows for understanding what she needs to learn and install to be productive. Note that to be useful the overview should be at a higher level of abstraction than the code - otherwise you run the risk of missing the forest for the trees or at least not saving any time.

It is obvious that documenting any of this before your architecture is stable more or less is useless, as a rule of thumb I would say this can be around the 5th-6th iteration - assuming the team has to grow during the project. If the team stays stable for the duration of the project, this documentation can take place towards the end of the project (though I would probably add recipes to a wiki or something similar during the project as development patterns emerge).

The second "thing" you can document in regard to architecture are the decisions you decided against, in my opinion this is more important than all of the other items mentioned above together.  The reason for that, while it might take a while to understand a well written software and infer its architecture it can be done, but it is virtually impossible to understand the options that were disqualified from looking at the chosen solution.
Understanding the options that weren't used can save time for the person reading that description. both in understanding why things are the way they are. Furthermore it can save time trying things that didn't work or provide clues to options when the circumstances change (since, as we all know, requirements change..)
The best time to document decisions you decided not to take is when you opt not to use them - this is when you remember best "why". for instance, in my current project we use x.509 certificates to authenticate clients and we use decided to use Kerberos tickets to authenticate components within the service. There's a reason for making that translation*,   there's also a reason for making the transition by replacing the client certificate with the edge component's credentials instead of mapping the client's certificate to a Kerberos ticket using an Identity provider*. we had two developers spike different options for two weeks until we came to the current solution instead of the more obvious choice of passing the x.509 certificate from the edge into the service and using the client's credentials. This question is likely to come up when/if someone else would take over the project, when the technology will be updated etc. Again, if we know why we didn't make that choice we can better decide what to do when the circumstances change.

To sum up, there are few architecture related issues that are worth documenting even in agile projects. some of them can be postponed some of them are worth documenting a little earlier. In any event it is better to document after the fact and to keep the documentation light.

---

* It all has to do to to limitations of WCF in regard to  the transports we use (HTTP, MSMQ and TCP) and the request/reaction pattern (asynchronous communication) we use.

While I am on the topic of REST, it is probably a good time to comment on my (first) post on InfoQ "Debate: Does REST need a Description Language"

Personally, I think there's merit in Services publishing their message structures in a machine readable format. When a Service has a machine readable contact. generated stubs allows you to make the interaction with less bugs vs. hand crafted interactions. It also makes it easier to test the service itself.

I do agree with Stefan's views on runtime interface dependency where he said that if a service consumer needs just 20% of the information in a service it shouldn't be forced to deserialize (i.e. know or care about) the whole message.However, I think this is a weakness of tooling not the concept. What if you had a tool that reads the machine readable contract, allow you to pick the 20% you need and generate for you a stub that ignores all the other 80% and "hand pick" the 20% you need. This is what you would personally do yourself anyway, and since the code is generated from the Service's definition it would be more resilient and error-free This is effectively designing a personalized mini-contract from the published general one. It does mean that when that 20% changes you will be affected, but this is something you'd have anyway.

I also agree that that the WS-* standards and resulting contract are (and getting more) complicated. Much of this can probably be attributed to the "design by committee" effect. However, there are also some real challenged that the SOA and ROA architectural styles do not address and we still need to solve those. Trying to solve these challenges is, by the way, what prompted me to write my SOA patterns book...

DevHawk (Harry Pierson) raised today a question I was toying with myself for a while now - if REST is an architectural style can it exist without the specific technologies that define it today. Or as Harry put it :

1. REST is a an "architectural style for distributed hypermedia systems".
2. REST "has been used to guide the design and development" of HTTP and URI.
3. Therefore REST as an architectural style is independent of HTTP and URI.
4. Yet, I get the feeling that the REST community would consider a solution that uses the REST architectural style but not HTTP and/or URI as "not RESTful".

What I had in mind for example is to use messaging where the equivalent of the URI would be a topic hierarchy.
Topic hierarchy allows you to have a unique "URI" for each resource.
The next thing we need to take care of are the PUT, GET, POST and DELETE verbs - we can do that by making the verbs part of the message headers.
As an aside I'll also say that  if we try to think about it as an architectural constraint then we don't necessarily have to use these verbs, a more general rule would say that the verbs are uniform and well known rather than specific ones
The rest (no pun intended) of the concerns, like specifying related states etc. can be dealt with making conventions on the message formats
Is that still REST?! I wonder...

In any event, what worries me the most in regard to REST is the religious manner that some people seem to treat it. By the way that is the same phenomena we see with some of the Agile folks. As for me? - Well, I don't really care if I fit that label or the other. I am just payed to deliver working  and viable software :), but hey, that's another discussion.

Yesterday I attended an SOA governance presentation by Brent Carlson. The presentation was basically an updated version of an article he authored in 2006 "SOA Governance Best Practices - Architectural, Organizational and SDLC implications" As a  tool vendor Brent has a lot of focus on the governance processes which I don't completely agree with (I prefer Jim Coplien's organizational patterns approach - see my post from last week). I also think the reuse figures he cites (registration required) are a little optimistic common place for what I consider the right granularity for services.
He also made a few points I  strongly agree with

• Brent talked about difference between the needs of run-time service repository (e.g. UDDI or an ESB) and a development time one. You need to address the services and their interactions during the development and you need to do that in a way that would be easy for the development teams. For example, one thing you want to log is usage, who is using the services since that will let you perform impact analysis when you have to make a change
  
• Building an SOA for an organization is an iterative process not a "big-bang" effort. This means you can't do just top-down design. you need to be pragmatic and also roll out working services.

The reason for this post however is the insight Brent gave regarding treating services as products rather than applications

Treating Services as products is  important because even if you don't believe that the SOA initiative should be  an iterative process once the move is  finished you would have quite a few services deployed in your organization. These services would integrate and interact with other services - some of which outside of your organization. You would also want to capitalize on flexibility claim that SOA makes and adapt your services to the changing business needs.
The challenges you face regarding updating and upgrading  functionality , anticipating consumer's needs, allowing consumers to get used to changes etc. are exactly the challenges product management techniques and principles come to answer

Treating services as products means a lot of things. let's look at a few examples: For one, it means predictable release cycles services like products get updated over time you want service users to be able to cope with this changes. Predictable release cycles means they can get organized in advance. Another aspect is the emphasis on backward compatibility e.g. orderly deprecation of features and version management,.One other thing  is introducing a "product manager". someone whose responsibility is to interact with customers, and potential customers, understand their needs and build a release road map for the services.

You might be used to doing some of that with applications   but thinking about services an products makes all this more explicit and that in itself is also important.

Last month we saw Ajax was making a move to the desktop with the aid of Adobe Apollo. Earlier this month I talked about it again when Sun announced JavaFX and after  Microsoft officially announced Silverlight.
I guess it was inevitable that Google would also join the fun of blurring the lines between desktop applications and web applications. And indeed I just read that Google released a beta of Google Gears.

What does Google Gears do? well, in essence it lets you develop occasionally disconnected web applications by providing an API to store application resources and data locally as well as Workerpools  that allow you to run tasks in the background (sort of like a collection of processes that host javascripts and  can communicate by messages).

What all these means is that it wouldn't be too long before the the architectural choice between web clients or desktop clients will not be that important it will be more about making a technology choice on the  preferred development environment and tooling as well as designing robust occasionally disconnected applications regardless of that technology. Just look at  the architectural guidance Google provides for utilizing Google gear.  You'd see that the components that they identify are very similar to the high-level components you are likely to design with any occasionally disconnected desktop application...

Yesterday I attended Jim Coplien's presentation on "Organizational Patterns - a Key for Agile Systems Development". Overall I think It was a very good presentation. Jim makes a few interesting claims, some of which are controversial within both in the traditional and the agile spaces
Few examples

• Process guidance (ISOs etc.) doesn't work - roles are stabler than processes, processes always change.
• Jim says that in order to make a change you need to make it at the organizational structure level. The processes will then support these changes
  
• TDD is evil - it is just an re-incarnation of bottom-up procedural design. It is better to follow "Design by contract"
• He says XP is not a good methodology (He thinks SCRUM is good)
• etc.

Additionally he talked about some of the organizational patterns he and Neil Harrison discovered studying organizations for more than a decade. You can  read the Top ten patterns on his site.

Jim covered 2 patterns that are related to software architecture: Architect controls product and Architect also Implements
Architect controls Product basically says that you should have an architect and that she should oversee that the direction of the project is flowing in the right direction.

Architect also implements - this pattern says that in order for the architect to broaden her leadership without sacrificing depth and pragmatics she must also participate in the implementation (beyond advising and communicating). Jim gave the example of the development of Borland's Quatro pro for windows in 1993 where the team's architect had a daily meeting (akin to scrum stand-ups) for synchronization and would then go and code with the developers. The Quatro pro team had 4 architects out of 12 persons that made the team.If a third of the development team is architect I'd say he is right -  My experience, however with most organizations I see is that you hardly have one architect per  project (sometimes you only have one for several projects). In these cases I hardly see the architect writing production code as part of the team since she would not have time to fulfill her architectural responsibilities. She must know how to code though and she must be able to prove her designs in code or be able to offer a candidate implementation if needed (I also wrote about that in the past see "Should architect's code" part 1, part 2, part 3

By the way if you are located in Israel, Jim will be here for a couple of weeks and he is giving a few courses like Agile Architecture, Patterns of Agile Project Management etc. You can find more information on pacificsoft's site

Udi Dahan writes that ".NET/Java Interop is not a reason for SOA". Udi writes that companies that  need to integrate two technologies turn to web-services and that

"The only problem is that in order for things to work right, they really must have a chatty interface, and flow transaction context between these “services”, and all the other things I describe as anti-patterns"

Udi is right that if you don't rethink and remodel your systems you will (probably) not  have an SOA as you are likely to find your self implementing  anti-patterns such as the ones he mentions.

However, using Web-services does not automatically mean that you are doing an SOA. If you don't think about moving to SOA you can still opt to use web-services as a remoting  or RPC technology to connect two systems. The advantage over the other proprietary products Udi mentions is that web-services are a standard technology. This will work well or fail is orthogonal to the technology choice. It depends on the architectures of the systems you integrate. If you need to flow transaction between the systems you'd also need that even if you cross-compile one of the applications in the other environment.

Another thing I don't agree with is the word must Udi uses. First, while it is likely that older systems has chatty interfaces it is not a must. The designers of the legacy system may have thought about the consequences of distribution without regard to SOA. Also you can still wrap an existing system with a service contract (using web-services or any other technology) and not get to chatty interfaces etc. However that means that the wrapper should have some substance or business logic inside it to mask the old system's behavior this is especially important  if you are thinking about moving to SOA and you take into consideration that the business will not just halt and wait there until you are done. You have to think about interim solutions, such interim solutions can include wrapping a legacy system with an Edge Component and a SOA facade (a pattern I call Legacy Bridge) while you move in the grader direction of a full blown SOA.

It has been awhile since I last published a pattern draft - but I guess it is better late than never.
The saga pattern deals with manginc complex interactions between services without the use of atomic transactions, which as I mentioned in the past are not a good idea (see "Transactions between services? No, No, No!" and "Some more thoughts on Cross-service transactions" )

You can download the draft for the Saga pattern from here.
I'll also add a link to it from the SOA Patterns book section (where you can also download the other pattern drafts I published)

By the way, I am not happy with the current  sketch (the pattern illustration) in this pattern, so it will probably change in later drafts. I would be happy to hear any suggestions you have for improving it.

I just read Shy Cohen's (via Nick Malik) article in Microsoft's Architecture Journal entitled "Ontology and Taxonomy of Services in a Service-Oriented Architecture" Shy provides a list of what he calls service types. He identifies two major types  bus services and application. He then continues to sub-divide them, the Bus Services are divided into communication and utility services and the Application services  are divided into entity, capability, activity and process services. I have to say it was quite alarming to see this coming from someone who had deep involvement in defining Windows Communication Foundation Indigo.

Where do I start?

well, for one, it seems completely fails to make the distinction between Services as in Service Oriented Architecture and Services as in capabilities or features an infrastructure provide. The "Communication services" are for the most part capabilities that a service infrastructure (such as an ESB) provides. Not Services you would define in an SOA initiative And then there's the matter of service granularity and the difference between Remote objects and SOA for instance, the example  Shy gives for  a "method" (his word) on a Customer service (entity service):

"An example of a domain-specific operation is a customers service that exposes a method called FindCustomerByLocation that can locate a customer's ID given the customer's address"

why would a service return a customer ID? This is the kind of call you would make on an object you hold a reference to not some remote "Something" that also want to authorize your call and may reside in a different company. This kind of thinking is what made remote objects fail. Gregor Hohpe explained that nicely in a paper  called Developing software in a Service Oriented World

The Transparency Illusion.  Distributed components promised to hide remote communication from the developer by making the remoteness "transparent". While the basic syntactic interaction between remote components can be wrapped inside a proxy object, it turned out that dealing with partial failures, latency, and remote exceptions could not be hidden from the developer. It turned out that 90% transparency was actually worse than no transparency because it gave developers a false sense of comfort.

As a side note, Gregor recently gave a presentation that covers this paper at JavaZone which you can watch online at InfoQ

Returning to Shy's article let's take a look at another quote:

Capability Service may flow an atomic transaction in which it is included to the Entity Services that it uses. Capability Services are also used to implement a Reservation Pattern over Entity Services that do not support that pattern, and to a much lesser extent over other Capability Services that do not support that pattern.

I already explained why cross-service transactions and especially flowing transactions is not a good idea in SOA so I won't do it again here - but you can read about it both here ("Transactions between Services? No, No, No!") and here ("Cross-Service Transactions"). Also I truly hope Shy didn't mean .NET data sets when he said "In some cases, typically for convenience reasons, Entity Service implementers choose to expose the underlying data as data sets rather than strongly-schematized XML data. Even though data sets are not entities in the strict sense, those services are still considered Entity Services for classification purposes." In any event the whole decomposition of Services into fine grained "capability", "Activity" and "process" takes no account of the fact the SOA is a distributed architecture...maybe Microsoft is not affected by the  fallacies of distributed computing ?

---

*ad nauseam (latin)- to the point of disgust

Yesterday, we had a meeting of a few architects and Memi Lavi gave a nice presentation on Architecture usability. The notions of architecture and frameworks/software infrastructure  are a little mixed in this presentation, but in essence Memi is right to say that among the quality attributes the architecture has to address you should have modifiability and architecture usage scenarios like "adding a complex form will take less than 2 weeks" or "wiring a new component into the workflow will take less than one day" etc. Reducing  friction* of frameworks is very important. I don't think it is the most important though. since if you could only have one of  the quality attributes you need to support (performance, correctness, architecture usability, framework friction, system usability, availability, budget etc.) I am not sure you would always pick architecture usability.
Driving home, I thought that essentially software architecture is just

The set of compromises, trying to maximize the effect of a limited set of  prioritized quality attributes

we can't have it all - so we need to prioritize our list of quality attributes and focus on the most important ones.

---

*Friction (from Ryan via Udi): "friction" is a (subjective) measure of how much the tooling gets in your way when trying to solve a specific-case problem. I've come to evaluate frameworks based on two rough metrics: how far the framework goes in solving the general case problem out of the box and how little friction the framework creates when you have to solve the specific-case problem yourself. When a framework finds a balance between these two areas, we call it "well designed."

Pat Helland is back in Microsoft (after a two years vacation in Amazon)  and more importantly he also restarted blogging. I only met him in person a few times - but he is definitely one of the few persons really worth listening to - especially when it comes to distributed computing. Not only does he make interesting observations he is also capable of explaining them in a crisp and interesting manner.  Indeed, it didn't take too long (his second post) before he blogged some valuable content. The post is called Memories, guesses and apologies (go read it).

Pat talks about how the notion of time in a distributed environment is subjective and you can really know what happened before what and what we can do about it (I really think you should  just go read it :) ).
Another related aspect of the phenomena Pat mentioned is that taking a snapshot in time, the chances of having a single unified truth in a distributed system degrade in a proportional manner to the system's load. I  had a chance to work on a few systems where some of the sites had either occasionally connected or connected over  low bandwidth networks. This situation makes the whole notion of guessing the state and compensating and/or apologizing for wrong conclusions much more explicit than in always connected high bandwidth system. Nevertheless, latency still exists even in connected systems and and you should really be weary of assuming a universal truth - unless you can stop the businesses  long enough to allow complete synchronization.
As I mentioned a few days ago, we can't afford to have cross-service transactions (I also think we can't afford too many distributed transaction in non-SOA architectures, but this is a especially true for SOA) which makes things even worse in this sense. One thing we can do in an SOA to achieve distributed consensus is to run a Saga. Saga, which is a long running conversation between services, is probably one of the most important interaction patterns for SOA.
You know what? instead of trying to explain it here in a haste i'll just publish the pattern draft - I'll try to do that before the end of the week.

Johanna Rothman writed about "letting go of BDUF" (Big Design Up Front). One statement she makes is that you can't predict what the architecture needs to be. I can't say I agree with that, since many times you do know something about the project and you do have prior expereice that can give you enough confidence to lay some ground rules. I called this Just Enough Design Up Front or JEDUF for short.

Another statement Johanna made, which I wholeheartedly agree with, is that you should let the architecture evolve. Evolving an architecture sounds very compelling but it is not a simple feat. Architectural decisions tend to have system wide implications which means that changing one too late in the game you'd get a lot of rewrite and/or refactoring to do
.
My strategy to solve that conflict is to:

1. Set the first one or two iterations as architectural ones. Some of the work in these iterations is to spike technological and architectural risk. Nevertheless most of architectural iterations are still about delivering business value and user stories. The difference is that the prioritization of the requirements is also done based on technical risks and not just business ones. By the way, when you write quality attribute requirements as scenarios makes them usable as user stories helps customers understand their business value.
2. Try to think about prior experience to produce the baseline architecture
3. One of the quality attributes that you should bring into the table is flexibility - but be weary of putting too much effort into building this flexibility in
4. Don't try to implement architectural components thoroughly - it is enough to run a thin thread through them and expand then when the need arise. Sometimes it is even enough just to identify them as possible future extensions.
5. Try to postpone architectural decisions to the last responsible moment. However, when that moment comes -make the decision. try to validate the architectural decisions by spiking them out before you introduce them into the project

These steps don't promise that the initial architecture sticks, but in my experience it makes it possible to minimize the number of architectural decisions but still have a relatively solid foundation to base your project on

In the article mentioned in the previous post, I talk about adding EDA to SOA and how you can use Complex Event Processing (CEP) to process the event streams and infer the trends and enhance the understanding of what happen inside your business. All the tools I knew about were Java tools I knew about were Java tools but now I've found out (via Nauman Leghari's blog) that there's also a .NET CEP engine and it is even open source - It is called NESper and like many other tools it is a port of a Java tool.
I am not sure how good it is - but at least I'll have an interesting evening today :)

An article I wrote on Business Intelligence (BI) and Service Oriented Architecture (SOA) has just been published on MSDN.
You can find it here http://msdn2.microsoft.com/en-us/library/bb419307.aspx.

The article explains the SOA & BI mismatch and how to bridge it by adding EDA to SOA. (I bloged about it here before, but the article is more ordered and complete)

Back in January, I took part in an architect panel that Microsoft Israel organized. The panel was led by Ron Jacobs and it featured Udi Dahan, Assaf Jacoby, Coby Cohen, Dudu Benabou and myself. A few days ago Ron edited this recoding and turned it into a podcast in his Arcast series.

The panel's focus was on lessons learned from mistakes made in past project. Ego maniac as I may be :) -- even though you don't get to hear me much in the final edited version -- I think the podcast is worth listening to, as the panel raised some interesting points. You can download the podcast here (don't worry it is in English even though it was recorded in Israel)

PS

I am the first speaker after the introduction, in case you are wondering.

I'll be presenting a  90 minute class on SOA Patterns on the upcoming Architecture & Design world 2007 - which will take place in Chicago on July 24-27th.
If any of you happen to be there, I'd be very happy if you drop by and say hello :)

 

I have seen the following question on one of the forums I follow

"I have studied up on the SOA approach and it all sounds good.  But most articles stop at the theory.

Lets say I sell things.   I have a CustomerProfileService.   The application does CRUD through this service to a back end database.  Its autonomous and isolated.
I have anther service, InventoryItemProfileService.  Again, the application does CRUD through this service to a back end database. It is autonomous from the CustomerProfileService.  Not only may it live on a different DB from the CustomerProfileService, it might exist on a different platform.
Now lets get to the InvoiceService.  Lets say from the client side, I would guess that i would have a CreateInvoice(custID,itemID[] ) method.  The InvoiceService would then call out to the CustomerProfileService for profile that meets the needs of the invoice, then another call out to the InventoryItemProfileService for the item descriptions and such.

Here is the question.  It would seem like in the back end (the db) of the InvoiceService there would be tables to support the customer info and the item info from the invoice.  Where prior to SOA, when everything was in the same db, these requirements would be largely satisfied by joins.  Now a logical join across services just seems radically expensive (everytime you touch the invoice).  hence the need for the customer and item tables local to the invoice service.

Does this sound right?  Just how often does the InvoiceService have to go back to these other supporting services?"

I also got a comment with a similar theme on my Cross Service Transactions post.

I see a few problems with the way the services in the question are modeled (like CRUDy interface) but in the end it all boils down to the root cause -and the real problem: granularity of the services.

Sure when "a service" is too small it doesn't make sense to separate its tables from those of other services. it doesn't make sense to have transactions that span only what's internal to the service. It doesn't make sense to pay the price to make a service autonomous (like caching reference data from other services). When the granularity is too small you will often find that you need to make a loot of interactions with other so called services. you are more likely to have CRUDy interfaces.
You are also more likely to have slow performing solution and suffer from  low-availability.
Using services in a granularity mentioned above is, in my opinion, a nightmare that would probably make you work very hard to maintain  the SOA principles in place  - or the more likely option, that you would circumvent the principles so that you can get something maintainable, usable and performing (and flip the bozo bit on this all SOA thing)

So what is the right granularity. Well, it is not a one-size-fits-all kind of thing, but as a rule of thumb I would say anything just shy of a sub-system and up. A service has to have enough meat so that it would make sense having it autonomous; that the transactions would fit nicely inside its boundaries; that it would be worthwile making it highly-available; that you can pass a complete task/document to it and it won't have to talk to a gazillion other services to complete processing it; etc.

If your application's idea of invoices is a 2 tables one with a header and one with invoice details - then don't make that a service. if invoicing is a sub-system with complex business rules a lot of options and what-not - then it can be a good candidate

Think about it next time you design a service :)



 

Back in January I opined that  moving to web applications was not the optimal solution to the real problem we have/had with desktop applications which was  installation woes. What we got was a poor UI without installation problems so we (software industry) started to resolve problems like we had when we moved from  terminals to graphical UIs etc. - 
So now we have Rich Internet Applications (RIA) - using technologies like AJAX - but they suffer from other problems  which again we've already been through

Well that was the topic of the post in January. Now I've stumbled upon an interesting/amusing  twist - called Adobe Apollo
Apollo let's you, yes you've guessed it, take your RIA applications and deploy them as desktop applications.  you can now take your HTML, CSSs , AJAX scripts pack them up as a single file (AIR) and lo and behold deploy them on the desktop. You even get these nifty start menu and desktop shortcuts :)

The reason not to dismiss this a complete waste of time - is that what we actually see here is another example of a trend to convergence  web and desktop UI architectures  and programming models. I say "another" because coming from the desktop direction Microsoft is doing pretty much the same thing. WPF brings the web-programming model  with its markup (XAML) and "code-behind" concepts to the desktop as well as pushing the same model to the browser with WPF/E .

The difference between Microsoft's and Adobe's solutions is that, Adobe is coming from the web-side and, as I said, Microsoft is coming from the Desktop side - both companies are striding toward the same goal -  and what we are left with,  is yet another technology war

I've updated the draft for the Edge Component Pattern to a more legible version (thanks to Cynthia Cane my editor @ manning).

The Edge component pattern solves the following dilemma:

How do we allow the business aspects of the service, technological concerns and other cross-
cutting concerns like security, logging etc. to evolve in their own pace and independently of
each other?

 

Follwoing several months of hard work, IASA - The International Association of Software Architect, has finally published on-line the Architect Skill set library. What is the skill set library you ask ? Well, it is what you get when a few dozens of architects (your humble servant included) author papers relating their experience on the different aspects of the architect’s role- everything from design, infrastructure, quality attributes to human dynamics (soft skills)

I should probably mention mention both Nicole Tedesco and Dana Gerow who orchestrated all this effort, and whiteout whom, I personally doubt if the project was now online

By the way, what is even more interesting is that IASA is organizing training material based on this body of collective knowledge - and if this works well we’ll finally have a comprehensive course to train new architects. Meanwhile, you can find the all the PDFs online at http://www.iasahome.org/web/home/skillset

P.S.

As I mentioned earlier I also wrote a paper for this project – on Views and Viewpoints which you can download directly from here

I was going to try to explain why it took me so long since I've posted the last pattern draft on-line when I saw that a couple of my fellow Manning authors already did that. See Roy Osherov's "Writing a book is like developing Software" and Fabrice Marguerie's "My Writing Experience". I have similar experience here -there are a few commonalities for software writing and it seems that the counter measures of shorter iterations, refactorings (which I guess writers know as rephrasing) and increased inspections seem to work here as well.

Finally, I am back to writing new stuff and I am completing Chapter 4 now. Chapter 4  deals with SOA security pattern, and I've decided to release the "Service Firewall" pattern as free draft. Note that it is a draft and it can change by the time it gets to publication for example the Edge Component, which I published a few months ago already went through some extensive rewrite (maybe I'll post the updated draft..)

The Service Firewall helps deal with malicious "service consumers" and protect the services from several types of attack including for example XDoS (XML Denial of Service), malicious content, preventing leaking private information from the service etc.

You can download the draft for  Service Firewall  pattern  from here .

Following my post on SOA definition, Alex left the following comment

“One question - how can an organization achieve "agility" through an SOA, if not through "re-use"? Isn't re-use really the ROI for implementing a Service?”

The way I see it, Agility means the ability to change rapidly and it doesn’t have to mean reuse  –for instance, it can come from the ability to replace a component without disturbing other dependent components – though you can say that this is reuse as you are reusing the interface (contract).

When you replace or update a service  you may reuse some or maybe even all of the previous version of a service – as long as the context for that service didn’t change significantly – if it did the granularity of the reusable components will be much smaller than a  “service”. 

I would also note that I think there’s a difference between reuse and use. If you take the same ordering capabilities and you include it in two business processes that just using it. I’ve seen reuse of services in product companies where services were reused with few modifications between two or more solutions but this isn’t very common.

Regarding the ROI of SOA –That doesn’t have to be reuse or just reuse it is also things like easier connectivity so that you can integrate faster with partners or new components that are developed . Another way to measure ROI is measure the gains in easier replacability and adaptability so you can faster respond to changing business requirements (e.g., changing what counts as a VIP customer without letting any of the service’s consumers that something changed).  

Udi has some comments on my SOA definition. Udi says that the definition I provided does not support  the notion of publish/subscribe using topics for services. My answer to this is yes and no :)

First thing first, I never said (or at least I never meant to say) that contracts are limited to only incoming messages. Contracts contain incoming and outgoing messages.   I probably should have stated it more clearly though.
Udi says “Contract: Who owns the message type being published? The publisher or the subscriber? Common SOA knowledge would say that the message belongs to the contract of the service that receives it”

I don’t know who is “Common SOA knowledge”. In my opinion, this thinking is a wrong “even” for request/reply. The reply message belongs to the service the sends the reply

Regarding Endpoints – if the subscribers go to a topic as in “ServiceName\TopicName “ then yes I would call that an Endpoint since this is a well known address consumers (subscribers) go to find messages published by a service

Regarding consumers Udi says “ Is the publishing service “using” the subscriber when it publishes a message? I don’t think so, and the subscriber definitely isn’t using the publisher at that point either. So, we’ve got some inter-service message-based communication going on and it isn’t clear if we even have a service consumer. In fact, if all a service ever did was subscribe to some topics, and publish messages on other topics, it looks like we’d have very loose-coupling but be straying from the common SOA wisdom.”

Maybe that’s just semantics but I don’t see why the subscriber isn’t using the publisher- The publisher publishes a message on a topic this is part of its offering. The subscriber chooses to consume that information and maybe do some stuff with that – possibly publishing some other messages. That’s a “using” relationship to me.

Nevertheless - SOA is not a synonym for "Distributed system" so there are cases when distributed components that communicates through messages aren’t SOA. For example publish/Subscribe using topics where the topics are common and shared between components so that multiple services can publish on the same topic does not, in my opinion, fall under the definition of SOA . This doesn’t say that this is a bad architecture in any way – but it isn’t SOA either.
As I said in the “What is SOA posts” for an architecture to be SOA you need autonomous components , that publish and accepts messages defined in contracts, delivered at an endpoint and governed by policies to service consumers – no more, but no less either.

I've been talking about SOA for a while now it's finally time to (try to) properly define it

I've publised this as a 5 posts on my DDJ blog and I thought it was good enough to be publised as a single whitepaper:

"Service Oriented Architecture or SOA for short has been with us for quite a  while.  Yefim V. Natiz, a Gartner’s analyst, first talked about SOA back  in 1996. However it seems that only in the recent year or so SOA has matured enough for real systems based on the SOA concepts to start to appear – or has it?  There is so much hype and misconceptions surrounding SOA that we first have to clear them all up before we can explain what SOA is – let alone identify who really uses it...." (Download full PDF (670K))

You can see additional presentations and papers I wrote here

[originally published in my DDJ blog]

You may have read my BI and SOA post where I suggested using EDA within SOA to solve the BI/SOA impedance mismatch. Jack Van Hoof made the following comment on that post:

Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0. Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other….

I think that "Synchronous RPC" is not a very good (or useful) definition for SOA (see my series on what is SOA anyway). Nevertheless, I also think that even if all you have is synchronous request/reply you can still implement both asynchronous messaging and EDA How can we implement Asynchronous Messaging?

Option 1 Duplex Channel
Let’s say you are a service consumer. You send me your request. Instead of a reply I just acknowledge you that I got the message. I put the message into a queue and process it on my "spare" time. I then call you with the answer.

Option 2 One way Channel
Again you send the request. Instead of a reply, I give you a token or a ticket for the answer. When you think it is time, for example when the time promised in the SLA elapse (or whenever), you call me again, give me the ticket, and I look up the answer and give you your reply. If we hide all this protocol inside some software infrastructure the applications can see asynchronous messaging even though we have synchronous request/reply on the lower levels.

Okay, so what about Events? How can we publish events just using request/reply. The previous example would not work since we can miss out on important events?

If you are reading this blog -- chances are you already have the answer working on your computer -- yes, it is RSS. Think about it using an RSS Reader that pulls the server that publishes this blog you reach out using synchronous request/reply and get all the posts (events) that were added since the last time you asked.

There are a few additional architectural benefits for working this way. For one the service does not have to manage subscribers. Secondly, the consumer doesn’t have to be there the moment the event occurred to be able to consume it -- and the management and set up is easier and simpler than using queuing engines

Jack Van Hoof left the following comment on my post on BI & SOA:

"Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0. 

Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other. In my article 'How EDA extends SOA and why it is important' I explained the differences between the two patterns and when to use the one or the other.

 

Because of the completely different nature and use of the two patterns, it is necessary to be able to distinguish between the both and to name them. You might say making such a distinction is a universal architectural principle. Combining both of the patterns into an increment of the version number of one of them is - IMHO - not a very clever act. I believe it is appropriate and desirable to use the acronyms SOA and EDA to make this distinction, because SOA and EDA are both positioned in the same architectural domain; SOA focusing on (the decomposition of) business functions and EDA focusing on business events."

I agree with some of the things Jack says but not all of them. The way I see it EDA and SOA are two different architectural styles- but I guess that I see it a little different than Jack does

EDA is an evolution of the publish-subscribe style - and can exist independent of SOA i.e. you can implement it with other architectural styles SOA is an evolution of the component based development style which puts an emphasis on interoperability and adaptability.

However I don't agree that SOA is "Synchronous RPC". That's just the initial "wave" of SOA implementations since synchronous interactions are easier to grasp and implement. I think that adhering to SOA principles you can also implement additional interaction patterns including, asynchronous messages, publish/subscribe and EDA (and combining SOA with EDA is what I suggested for solving BI in an SOA)

 

I don't like the SOA 2.0 term as well - but that's just because I don’t see a need for defining a new term :)

 

I'll post more about this once I finish the "What is SOA anway" series on DDJ where I explain the way I see SOA

 

All I can say is I want one

http://cellphones.engadget.com/2007/01/09/the-apple-iphone/

[based on a few posts from my DDJ blog]

Implementing Business Intelligence (BI) solution on top of Service Oriented Architecture (SOA) is not a simple feat. A recent survey by Ventana Research shows that "...only one-third of respondents reported they believe their internal IT personnel have the knowledge and skills to implement BI services.". There's a good reason for that since there an inherent impedance mismatch between BI and SOA which takes some effort to overcome. The purpose of this paper is to look to explain the problem as well as look at the possible solutions.

Service-Oriented Architecture is about autonomous loosely coupled components. These traits gives you lots of benefits such as greater flexibility and agility but it also means that services have private data. Data that you don't want to expose to the outside as exposing it will decrease autonomy and increase coupling. This is why services only expose data and processes via contracts rather then exposing their internal structure.

That is all fine until you start to think about business intelligence. The cornerstone of any business intelligence initiative is gathering, collecting and consolidating data from all over the place. Once you have the data, you can use tools to analyze it, data mine it, slice, splice, aggregate, and whatnot. Traditionally BI builds on ETL (Extract, Transfer, Load) which goes directly to the database of the involved sources.

And here lies the problem: On the one hand we have services that want to keep their data private, and on the other we have a datamart or warehouse that wants that data badly.

What are our options?

• If you go with traditional ETL, you introduce coupling into your service.
  
• If you only rely on contracts that were constructed for business processes you may be missing out on important data.
  
• If you build a specific contract that exposes "all" the data you are back at the point-to-point integration -- solving point-to-point integration is one of the reason we want SOA in the first place.
  

The second option seems to be the most reasonable choice of the three -- but it also has several problems. One problem is that the BI needs to know about all the contracts. The second was already mentioned -- important data might be missing. The third problem is that the BI system need to fetch data from the services which means it may miss out on data in the intervals between request. On the other hand, too frequent requests and you can congest your network easily as well as cause DOS on your own services.

Clearly we need a fourth option

In my opinion, the best way to tackle BI in SOA is to add publication messages into the contract. By "publication messages", I mean that the service will publish its state either in a periodic manner or per event to anyone who listening. This is a service communication pattern which I call "Inversion of Communications" since it reverse the request/reply communication style which is common for SOA.

To make the solution complete, you can add additional requests/reply or request/reaction messages to allow consumers to retrieve initial snapshots. Following this approach, you get an event stream of the changes within the service in a manner that is not specific for the BI. In fact, having other services react on the event stream can increase the overall loose coupling in the system - for instance by caching results of other services

Why is this better than the other three approaches? For one , you can get a good picture of what happens within the service. However the contract is not specific for the BI and can be used by other services to cache the service state (thus increasing their own autonomy), for reporting (you can see an early draft of the aggregated reporting pattern), and for BI purposes. By working against a steady stream of events, the BI platforms can Analise treands, keep history and get the complete picture they need.

The approach above is sometimes referred to as "Event Driven Architecture" (EDA) and while I (and others) see EDA as another facet of SOA, not everyone agrees. Gartner, for instance, sees EDA as another paradigm and SOA just for request/reply, or client/server. Recently, however, they published a paper that calls the approach described here as "Advanced SOA". I tend to agree more with the "advanced SOA" definition and don't see a contradiction with EDA and the SOA definitions. We are still using the same components and the same relations only adding an additional message exchange pattern into our toolbox.

A note on implementation: If you are implementing SOA over an ESB that is rather easy to implement as most ESBs support publishing events out of the box. Using the WS* stack of protocols, you have WS-BaseNotification, WS-BrokeredNotification and WS-Topic set of standards. If you are on the REST camp, then I guess you will need to implement publish/subscribe by yourself.

Once you have event streams on the network, The BI components grab that data scrub it as much as they like and push it to their datamarts and data warehouses. However, event steams can also enable much more complex and interesting analysis of real time events and real time trend data using complex event processing (CEP) tools to get real-time business activity monitoring (BAM)

You can also get post as as a presentation down loadable from the papers section on my site or directly from here. (The download is about 3MB.)

Welcome to chain-letters blogsphere style. There's this on-line tag game going around, I've been watching it spiraling around on many of the blogs I read and now Udi dragged me into this as well :)

So here goes - here are 5 things you don't know about me:

1. It only took me 14 years to get my BA degree in Computer Science. I began studying in 1990 in the Technion ,quit after 2 years and only bothered to graduate when I wanted to get a Masters degree

2. I was a Microsoft Foxpro MVP for 3 years in the 90s - about half of that time I was working on completely different platforms and tools (J++ and C++ on Windows and then J2EE on Solaris).

3. first met English in fourth grade (like most other Israeli kids at that time) - by the eighth grade I read my first real book - Shogun. I took me 3-4 month to get through the 1200 pages of the book, but I've been reading English since. In fact I hardly read Hebrew anymore.

4. I learned to program on the ZX81 - I remember the joy the first time I fully used the 1K memory it had, as well as the disappointment the followed thereafter when the instructor tried to add the 16K expansion which caused the machine to reboot.

5. I used to be a hobbyist bar tender. I still have more than 70 different bottles at home, with everything from Grenadine to a 25 years old Glenmorangie. I don't mix too many drinks these days, I mostly drink the Macallan.

I don't want to stay "it" for too long :), so on to tagging some other folks: Ohad Israeli, Andrew Johnston, Tad Anderson, Nancy Folsom and Ruth Malan. You are all it

I've stumbled upon a presentation by Ron Jacobs on the Software architect's role (via Shahid Sah's blog) called Architects and the Architecture of Software. In this presentation Ron compares the architect's role for that of an explorer, an advocate and a designer.

While I can go for the designer bit - although I don't like the heavy analogy to building architects (I know, I know I have that as well in my software architecture presentation - but at least it is no longer there in the next version)

However I would personally replace advocate with a mentor and explorer with a polymath or renaissance man and add a leader and visionary as well (although Ron mentiones that as part of the discussion on explorer)

Advocate is someone who observes, listens and gives advice - but a mentor is someone who helps others reach the right decisions and help them learn and evolve. I think that has much more value. I want a Socrates not an Alan Dershowitz on my team

An explorer looks for new grounds and is a bit of a visionary - but a renaissance man is both knowledgable and inventive. As a development manager I rather have someone who knows what he is doing, understand the wider perspective and can find solutions to my problems - and not someone who would take me on a road to uncharted territories. I'd take Leonardo Da Vinchi over Columbos ( who accidently gave the competitive edge to spain and didn't even know it) any day.

A visionary and leader is also important - you want someone who is able to look far and that can help your team get there- I guess that is somewhat akin to an explorer (in the sense Ron mentions) but again I'd rather have a Martin Luther King than a Columbos (though lucky wouldn't heart).

But hey, that's just my opinion

One unique aspect of SOA vs. other architecture styles like Object Orientation , Client/Server or even 3-Tier architecture is that it is built for highly distributed systems. Each and every service is a sub-system in itself it can run on its own machine and be located everywhere in the world . Many times, the service itself needs to be distributed in its own right. One reason to use distributed computing inside the service is computational intensive tasks.

 

 One of my recent projects was the development of a  biometric platform.  The platform can be used for many usage scenarios. A simple scenario is an access control systems - e.g. authorize entrance into a secure building or area. This is a relatively simple scenario as you usually only have to deal with few thousands of people and as a person requests entry she also declares who she is (e.g. using an RFID card with her ID). In these cases you can go to the database, lookup the appropriate record , run the biometric algorithm or algorithms and verify the person is who she says she is. However the same platform also has to work for other, much more demanding and computing intensive scenarios. For example consider a forensics scenario where you have a fingerprint collected at a crime scene, in this case you don’t know who the person you are looking for is, and you have to run your search on basically all the database which can contain millions of records. Keep in mind that when you match a biometric template[1] you calculate the probability of a match (based on the internal structure of the template) and  that each template weights about a one kilobyte you quickly realize that this can be quite a CPU intensive task.

Sometimes when you develop you SOAs you will have algorithmic tasks or other computational heavy tasks such as the one mentioned above and the question is

How can a Service handle  computational heavy tasks in a  scalable manner?

 

You can get the full pattern from here

[This is an early draft of one of the Performance, Scalability and availability Patterns from my SOA Patterns book]

---

---

 

I've added a section called SOA Patterns on the site while holds the current draft for the table of contents of the SOA Patterns book I am writing. The section lists the problem each pattern addresses as well as links to published patterns. Also, you can  use this to monitor my progress (patterns that already have their problem written down already have drafts; the others are in-progress or not started).

I am currently working on chapter 4: Security & Manageability patterns (not counting delays mentioned in the previous post).

Also, as I think I've already mentioned, I'll make public at least one pattern per month, if you are interested in a specific pattern in particular (from those which are ready - now chapters 2&3) drop me a note and I'll publish the one that gets the most votes

 

My editors at manning think that my chapter 1 of the SOA patterns book is not good enough.

They basically say that the chapter talks about too much theory vs. the other chapters which contain much more down-to-earth stuff (e.g. Edge Pattern, Aggregated Reporting Pattern, Decoupled Invocation Pattern ). Also they’ve said that I spend too many pages explaining what architecture is or taking about distributed system before I get to SOA – which is the topic of the book.

The way I see it, understanding architecture and distributed systems is essential to understanding SOA (from the development side i.e. when you want to design and build services). For example the discussion on quality attributes explains how you can use scenarios to find architectural requirements (and each pattern then has a section on relevant scenarios to help you find if the pattern is applicable to your needs)

I would be very interested in hearing what you have to say (either as comments here or emails to me) about the Chapter’s structure and content (considering most of the books will be patterns like the Edge pattern)

Thanks in advance

The business rationale behind going on the SOA road is increasing the alignment of the business and IT, so we divide the business into a bunch of business services and everything is just fine. However the minute we start diving into the SOA implementation details we are swamped by a horde of technologies, cross-cutting concerns (auditing, security, etc.) and whatnot.

For example, in one project I was involved with, we implemented an SOA over a messaging middleware (Tibco's Rendezvous). Just when everything was fine and dandy - along came another project which could potentially use few of the services. Well, almost, it needed a slightly different contract and it also used completely different wire protocol - WSE 3.0 (Microsoft interim solution for the WS-* stack before Windows Communication Foundation). And that's just one simple example - cross cutting concerns and implementation details are everywhere. The question is then:

How can you handle cross cutting concerns like multiple technologies, protocols, changing policies etc. while keeping the service's focuses on its core concerns - i.e. the business logic.

You can get the full pattern from here

[This is an early draft of one of the Service Structural Patterns from my SOA Patterns book]

I've posted a new paper on the site. Designing a good architecture is not enough. The paper explains how to make sure the architecture is both relevant ands followed throughout the project. You can download the paper directly from here.

I am going to present SOA in one of our internal forums next week - so I thought it would be a good opportunity to dust-off my SOA presentation and give it a little face lift. You can download a copy from the papers and articles section (or get it directly from here).

As always, any comments are welcome

The draft for the first chapter of my SOA Patterns book is available on-line from Manning Publications Co.

The first chapter talks about  software architecture and the inputs the architect can/should use to design one (emphasizing Quality Attributes); Explains the challenges of distributed systems and takes a look at the SOA from an architectural perspective.

You can download the chapter from here

Any comments are welcome (you can also leave your comments at soa@rgoarchitects.com)

Roy Osherov recommended this site today - but he also urged me to write more frequently.

This is probably a good opportunity to explain how posts are divided between my 3 blogs

• First theres the blog on Dr. Dobb's Journal. This blog is published on the "Architecture & Design" section of DDJ portal. I blog there about 3 times a week. Jon (my Editor @ DDJ) prefers a steady stream of blogs over longer posts which means that I break down large subjects (like OO principles, fallacies of distributed computing and the currently running series on the Architect's soft skills) into many parts.

• The second blog is a new one on Microsoft's Israel blogs site. The aim of this site is to bring Architecture content in Hebrew to the Israeli architects (As can be imagined, most of the technical content available is in English, I thought it was important to generate some content in Hebrew as well)

• The last blog is this one. My current plan for this blog is as follows

• • Cross posting selected posts from the DDJ site
  • I am posting here complete articles made by editing and aggregating multi-part blogs posts (again such as the fallacies etc.)
  • Pointers to presentations and articles I publish
  • In the near future, I'll start posting bits of my upcoming SOA patterns book (I am currently writing chapter 3). I've already documented 8 patterns (of more than 50 patterns and about 30 anti-patters). I plan to publish here at least some of the patterns here for review (I am still crossing the t's and dotting the i's with my publisher but I expect this to be finalized soon)

so Roy, does seven (7) posts in seven days (including 3 on Ms Israel site, 3 on DDJ and this one) qualify as posting often enough? :)

[Will also be cross-posted on my DDJ blog]

Working on my SOA patterns book, I thought of this rule for contract versioning which my shameless ego wanted to dub Arnon's Contract Versioning Principle. I was happy playing with this thought, until I realized that there isn't some profound new understanding here, this is just an application of LSP for service contracts.

Liskov Substitution Principle (LSP) which I recently blogged about here as part of a series of blogs on Object Oriented Principles, basically states that a subclass should be usable instead of it parent class. To put this in other words you could say that a subclass should meet the expectations that users of the parent class have come to expect from the parent class's observable behavior.

So LSP applied to SOA would state that:

When changing the internal behavior of a service, you don't need to create a new version of the contract if for each operation defined in the contract the preconditions are the same or weaker and the postconditions (i.e. the outcome of the request) are the same or stronger or in other words the to retain the same contract version, the new version of the service should meet the expectations that consumers of the service have come to expect from the old version's observable behavior

For example, let's say you have a customer service and the contract lets you get a customer's VIP status. If you changed the way the VIP status is calculated (e.g. in the old version the customer had to have 1 million dollars in her account, but now she must have 10 million dollars) there's no need to create a new contract version. However if you introduced a new level of VIP status (e.g. 1 Million = Gold, 10 Million = Platinum) you do need a new version for the contract

I've added a new section on the site www.rgoarchitects.com/Papers to allow easy access to all the papers, presentations and articles I published (and will be publishing e.g. I'll add a paper on architect soft skills in a month or so etc.)

 

The October issue of Dr. Dobb's Journal is available and with it my article on the SPAMMED Architecture Framework .

 

Dr. Dobb's has begun a special (6 issues) E-Zine  called  "Dr. Dobb's Requirements Development" and I am happy to say that Issue #1 includes the first part of an article by me on the subject of use case modeling.

The issue also includes articles by Joe Marasaco (author of "Software Development Edge"), Andrew Stellman & Jenifer Geene (authors of "Applied Software Project Management") and Karl E. Wiegers (Author of "Software Requirments" 2nd edition.)

You can get the EZine by registering on www.ravenflow.com to the bi-monthly Requirement Develompment magazine

 

The Microsoft Name Machine is at it again - this time's poor victim is Atlas:

[from Joshua Flanagan's blog] " The Microsoft AJAX (formerly known as DHTML) technology, codename Atlas, will be officially named... ASP.NET 7.0!"

nuff' said

 

Over the last few months I've posted a series of blogs on DDJ that cover the basic Object Oriented principles (e.g. Single Responsibility Principle, Don't Repeat Yourself, Inversion of Control etc.).

I've assembled all the posts into a single whitepaper which you can get here.

Also you can download the same (plus a little more) material as a powerpoint presentation.

 

[Crosspost from my DDJ blog]

When talking about multi-tiered architectures, we need to remember that the tier boundary is significant. The tier boundary is where distribution happens and if you remember the "fallacies of distributed computing", you know not to take that lightly.

A tier is a physical boundary (versus an Edge in an SOA which is a logical boundary, for example) and the implications are numerous. For instance, you need to consider:

• Trust--who do you let in?
• Security--what do you send out?
• Performance--you need to serialize to pass the boundary, and remote data is expensive to fetch.
• Availability--what happens if you crash?
• Manageability--can anyone see what's your state? Help you recover?
• Temporal coupling--can you afford to make synchronous calls?
• and many similar questions.

Yet many times people think passing a tier is as simple as passing a logical layer. I should know. I made this stupid mistake more than 15 years ago in one of the first distributed systems I designed. I planned this beautiful separation of the UI controls from the business logic (I didn't know it was called "MVC" and that someone else had figured it out ages ago, so I was pretty proud of myself). When you clicked on a button you just used metadata to say that BL should catch it. I had all this wonderful "infrastructure"that handled passing the call to its destination.

But then we wanted to take this n-layer application and put the BL in an "application server" which will handle multiple clients. Oh--now we need to move events over the wire , handle calls from multiple unrelated clients, pass a lot of data back and forth, and what about security... you can imagine the fiasco.

Thus, as Niels Bohr once said, "An expert is a person who has made all the mistakes which can be made in a very narrow field." But you don't have to make the same mistakes. Just remember that a tier is a natural boundary. You know what? You should probably even want to consider it the edge of a cliff at the end of your application--and be careful not to fall down.

[crosspost from my DDJ blog]

In a comment to my previous post on Architecture vs. Design, Yoni said:

It seems you are categorizing technical issues as architecture and logical issues as design. I think Martin Fowler's definition of "Making sure important things remain decoupled and easy to change" transverses both categories and is easier to follow.

I have a few things to say about this.

First, I don't categorize technical things as "architectural" and logical ones as "design." What I do say is both "architecture" and "design" are types of design where with one you focus on the wider aspects of the solution and quality attributes, while with the other you focus on local and functional aspects.

I don't see how the definition Yoni brings up is a better way to distinguish between the two? Who is to say what is important and what is not? Isn't decoupling an important trait at all level including so called "detailed design" level (e.g. utilizing dependency injection at the class level will give you better testability). Moreover, decoupling is important but sometimes you need to trade that to be able to satisfy a more prioritized quality attribute (if you want to meet a projec's quality, budget and schedule targets); see my definition of what software architecture is.

Another thing is that it doesn't matter that much where the line between architecture and design passes. The distinction between architecture and design is a semantic one that reminds us that the design of a system needs to be done in several levels of abstraction (provided the system is not too trivial). We need to abstract certain aspects of the system in order to be able to grasp the big picture. You cannot (well, I can't anyway) think at a 100 man-years project. You can only think about it at the class level and understand how everything will work together. Again, architecture is there to remind us to focus on a level of abstraction that lets you deal with non-local decisions and to make sure quality attributes are met if we cross the line to design -  No biggie.

[edited version of post I made on Dr. Dobbs Portal]

Back in April I  provided a definition for "architecture" as one of my first posts on DDJ. I also promised I'll talk about the distinction betwen Architecture and Design. Well this time is now.

When I try to think about it. I see two base criteria to distinguish between the architecture and design:

• Design deals with local decisions, where architecture is broader. For instance, you "design" the interfaces for your classes, but you "architect" the division into tiers.
• Design is mostly about the functional requirements, while architecture is mostly about quality attributes. You design how a specific workflow will fulfill a certain use case, but you architect the solution to the system's availability.

It is probably quite evident that this distinction only provides blurry borders between architecture and design; for example, when you have a multi-tier solution and you "architect" the UI and say it will implement MVP pattern. Can this be considered local decision and thus design or is this the overall decision (for the UI) and thus architecture?

The way I see it the exact cross-point from architecture to design is not that important. The point in talking about two distinct activities in the development process is to maintain separation of concern. you need to handle both to make sure a solution will actually work whether you do a little design while architecting or do a little architecture while designing really doesn't matter. Also architects should be involved in both activities anyway...

Udi Dahan, which is one of the best architects I know has recently created an excellent course on SOA - you can find the details of the syllabus on Udi's site .

I had a chance to work with Udi in the past and the solution we implemented utilized many of the patterns and techniques Udi covers in his course - so these are not just nice theories but rather real stuff that works

 

[crossposted from DDJ]

Yesterday I attended an interesting presentation on SOA by Dr. Donald F. Ferguson, chief architect for IBM's software group.

I was happy to hear him validate some of my thoughts on SOA (e.g., workflows are better kept inside services rather then outside, transaction boundaries should be inside a service, and so on), and introduced a couple of things I didn't know much about (for example, OSGi, a SOA platform for networked devices that's not based on web services. He also presented some nice insights (for instance, looking at the middleware as an infrastructure service and thus nicely unifying SOA and EDA)

On of the insights Donald presented was the use of heuristics as an aid to modeling and validating architectures. Some of the heuristics he mentioned include:

• Occam's Razor -- avoid needless repetition
• Don't create something new if you can compose existing stuff to get the same result
• externalize volatility -- don't put in the code things that are likely to change
• Focus on "name,value" programming not "offset programming" -- make things easy to understand
• Different is hard
  

If you look at heuristics as an abstraction of experience, they can provide as a good tool for keeping yourself on the right track. Some heuristics are universal (maybe the ones mentioned above and a few others like "simplify, simplify, simplify" or "the original statement of a problem is probably not the best one or it may even not be the right one"), but the problem is, as always, deciding (in advance) which heuristics to apply to a problem.

If you interested in using heuristics as an architect tool you may want to look at " The Art of Sysytem Architecting", by Mark Maier and Eberhardt Rechtin. The book discusses the architectures of different system types (collaborative, IT, Manufacacturing, etc.) and provides heuristics for each of these systems.

Heuristics are a good tool to use when you design an architecture and in a way the different design principles (e.g., the single responsibility principle) can also be considered heuristics. Nevertheless it is very important to verify designs by additional methods like code and formal evaluation and not rely on heurisitcs as the only tool.

Last week I published a 3 part article on O/R mapping on my blog @ Dr. Dobb's Portal. The paper describes the benefits and costs of using O/R mapping as well as recommend when O/R mapping should be used.

Here it is as a single whitepaper: Architecture Dilemmas - OR Mappin.pdf (228.78 KB)

[Crossposted from DDJ]

Following the series of posts that discussed whether architects should code (see Part I, Part II and Part III), I thought I'd expand on how I see the software architect's role.

To be a good architect, a person needs to have experience taking a project from inception to production and maintenance, they should have project management experience, and have spent time working directly with customers. On top of that, know-how and experience is needed in design and technology. Having a diverse experience is what enables the architect to have a wide view on a project, understand the need for pragmatism, better understand stakeholders concerns as well understand the implications of design and technology decisions (on scheduling and budgeting, for instance). Having a wide experience also puts architects in a unique position where they can really help make projects meet their goals.

Software Architecture deals with the system's quality attributes. This makes the architect the ultimate person responsible for making sure the solution meets these quality goals (and I don't mean quality as in low bug count, rather the soundness of the solution and it ability to address the various stakeholders concerns).

To make this quality claim a reality I tend to take a holistic view in regard to the architect's role, which, simply put, means that the architect needs to do whatever it is needed to make the project go forward and succeed. Taking this direction toward quality is only possible if the architect has the wide range of experiences mentioned above.

How does that translate to real life? Here are few examples for "non-architect tasks" from my experience:

• On one project the system engineering team was working in the wrong direction in regard to analyzing requirements (this cost us six months of delay). I helped introduce the concept of Use Cases and created the project's initial use case model. (You can read the insights I have from my experience in my paper Use Case Methodology for large systems (pdf) )

• On another occasion one of the project managers was trying to evaluate how a certain technology will help advance the project. I helped her construct the WBS of the activities needed to complete the needed functionality (assuming the technology is in place) and the helped her create the estimates.
  
  
• On another project we had performance bottlenecks related to the technology used (ESRI, displaying vector maps with high refresh rates for displayed objects). I got together with another architect to pin-point the problem, diagnose it, and came up with a solutions (mapping ESRI temp files to a RAM-disk).

• I worked on a project where time was running out and a milestone was looming fast. I helped introduce the project to some SCRUM-like techniques (why only "SCRUM-like"? Remember that Rome was not built in one day either) and by working closely with the PM helped the team reach the milestone successfully (indeed not all the feature were completed--but the ones that did work were the features important to the end-user which made the milestone a success).

And the list goes on and on. And yes, coding can also be a part of this list (though for me it usually only translates to writing short proof of concepts--and for others it may mean coding some of the tasks with the team).

To sum up, a good architect is not just a lead developer on steroids. An architect should have a much wider range of capabilities and experience. Architects are "enablers". They should use their capabilities to help advance the solution and ensure its overall quality.

Well, not right away - but I just read he will be leaving MS by 2008. I don't know whether it will be good or bad for Microsoft in the long run but whatever the outcome will be it will definitely be the end of an era.

 

I have amassed more than 30 patterns related to SOA (e.g. SOA Patterns - Decoupled Invocation and SOA Patterns - Aggregated Reporting which I previously published here). I have patterns around security, availability, scalability, composition, adding a UI etc. Some of the patterns are original (I think) and some are based on other peoples work.

 

I am trying to decide whether or not it would be worthwhile putting all these patterns into a book. Writing a book is a very time consuming task (or so I am told) - so I thought I'd run a quick poll between the readers of this blog to see how many of you would be interested in reading (and buying) this book if it will get published.

I know this is not a representative crowd - but it can give me a (very) rough idea on the interest in such a book.

 

Please  send any comments (comments like "forget it, no one would ever want to read anything you write" are also ok) to soa@rgoarchitects.com (or leave a comment here)

 

Thanks in advance - Arnon.

[Edited version of post in DDJ]

Since I have been blogging for about a year now on this blog and now also on the DDJ blog. I think  it is time to try making something with more two-way communications.

Consequently, I am going to run a little experiment for a few weeks and see how it goes.

The idea is as follows: If you have an interesting architectural or design dilemma, drop me an email at ask@rgoarchitects.com I'll pick one issue per week and post (on the DDJ blog) the dilemma (anonymously) plus voice my opinion (and/or suggested solution)--and then everyone else can chime in with their comments and insight which hopefully will shed some light on the subject.

I'd be interested to hear both your opinions on this initiative and, of course, interesting dilemmas you are facing. Again, send your dilemmas to ask@rgoarchitects.com)

I gave a peresetation on CMMI today.

The main points of the presentation are

• CMMI focuses on process
• It builds on the premise that improving the process imporves the quality of the product
• CMMI integrated several others CMMs including Software CMM
• CMMI is a framework that lets you define the process - you need to show you cover the CMMI process areas for certification
• Newer methdologies (Agile) focus on people rather than process.
• However considering CMMI is a framework you can map agile processes to CMMI
• You would want to do this (mapping) if you want to introduce agility into CMMI organizations
• Another reasons to mix both approaches is sometimes therese a need to use formal processes but you still want some agility for sub-projects.

I just finished blogging a series of posts on the 8 fallacies of distributed computing on my DDJ blog.

I think it turned out pretty well so I aggregated all the posts into a white paper .I  hope you'd find it useful.

Scott Ambler has a new article comparing most of the leading development methodologies. He also tries to  recommend which methodology fits which kind of project (e.g. for a commercial of the shelf products use EUP/RUP, ISO 12207, TSP/PSP or Data Driven Approach).

The article serves as a nice overview of available methods - however Scott doesn't explain the reasoning on why he think a particular methodology fits (or doesn't) a certain type of application which is a pity. Furthermore, I think Scott is missing the point a little by neglecting organizational, cultural other people related reasons for choosing a methodology. For example if all your teams are versed with RUP, you would most likely "force-fit" it to your new COBOL project rather than choose a better fit methodology.

Also, I am not sure I agree with the all his mapping - the most notable example is  mapping XP for  safety critical projects. To get a DO-178B (the certification required by the FAA for aviation software) you need to have the following documents (DO-178B has 5 levels and not all documents are needed for all levels):

DO-178B Documents:

DO-178B Records:

Plan for Software Aspects of Certification (PSAC) Software Development Plan (SDP) Software Verification Plan (SVP) Software Configuration Management Plan (SCMP) Software Quality Assurance Plan (SQAP) Software Requirements Standards (SRS) Software Design Standards (SDS) Software Code Standards (SCS) Software Requirements Data (SRD) Software Design Description (SDD) Software Verification Cases and Procedures (SVCP) Software Life Cycle Environment Configuration Index (SECI) Software Configuration Index (SCI) Software Accomplishment Summary (SAS)

Software Verification Results (SVR) Problem Reports Software Configuration Management Records         *list copied from LynxOS site

I don't think that this level of formality is a good fit for XP

 

[crosspost from DDJ]

Reading the comments on my previous two posts on whether architects should code (here and here) as well as the comments on Johanna Rothman's posts (here, here and here) leads me to a few observations:

The first apparent thing is that the issue is a very loaded. Some people believe it is essential for architects to code, while others (like me) believe that their time is better spent on other issues. (That said, it seems that a small majority of the commenters think architects should code as part of the development team--at least for feedback purposes if nothing else.)

There is a wide consensus (me included)that architects should know how to code and have extensive experience in coding. It is also agreed that architects should be involved in the project--that is, not just drop off the architecture, then disengage.

I still believe that when the project is big enough (that is, big enough to warrent more than one team working on it) the project is better served by the architect getting involved in all the teams, rather than participating as a developer in one of them. If you are an architect and develop as part of the development team you are (or should be anyway) committed--meaning you need to deliver the piece of code under your responsibility at an acceptable quality level as other developers. Which is exactly why you would be less likely to deliver on your responsibilities for the total quality of the project. I assume some of the differences in opinion can be attributed to disagreement on what software architecture is , at least when compared to design).

I also think those who think architects must code see the architect as some sort of a lead developer again. I don't buy that. The architect's role is much broader than that (see also this post by Kevin Seal, which also discusses this issue). I see a holistic view of the architect role, which is making sure the product is delivererable. This may translate to the architect coding a module or two, but it can also translate to a lot of other things. Examples from my experience as an architect include preparing initial cost estimates, iteration planning, helping debug and testing, solving installation problems, analyzing requirements, conducting design and code review, design, and prototyping (yes, that's coding but as I said in the previous posts, that's not writing the production code and this is not having to meet deadlines etc.).

I also liked a comment by Graham Oakses on one of Johanna's posts :

My experience is that an architect is pulled between three poles--the product, the team and the client. The product pole pulls you towards managing the "conceptual integrity" of the design. The team pole pulls you towards mentoring people, helping them build skills, etc (which may mean consciously letting someone write code that you could do much better yourself). The client pole pulls you towards translating between the technical and the client domains (which is often where you get pulled into powerpoint). You need to trade these poles off differently on every project...

To sum up, the answer to "should architects code? " is like so many things in life is--it depends.

While I still hold my view on the Current State of Software Factories - at least one company (EDS) is repoting  (in an MSDN article) that they've built a software factory for their domain entities [found via Steve Cook]. Nevertheless this is still a generic generator (i.e. not a factory for ordering system or something similar- but rather something like  a DSL for O/R Mapping).

Arnon

PS

I also wonder why do they generate unit tests fot the generated code - assuming they properly tested their templates the generated code should just work ...

Roy Osherove blogs about some of the questions that were discussed in the architect's panel in the Recent TechEd Israel

 

I've been thinking a lot about some of these questions lately  (and not just because I helped  draft the questions for the panel :)) specifically about when and how to introduce agile methods. One problem, which Roy points out, of fixed price/time projects (which unfortunately in Israel are pretty much the norm.). Another problem is with organizations such as the one I work for which have CMMI level 3 

(or more) certification (not to mention ISO 9002) which makes it really hard to introduce agility.

 

I stumbled upon this presentation  which analyzes the CMMI compliance of agile methodologies (I've started to try map SCRUM to CMMI 3 to get SEPG (Software Engineering & Practices Group) off of my back)

 

Another interesting approach I found is AgileTek's Agile+ methodologywhich is a mix and match approach that claims to be the best of both worlds (I am not sure I am 100% convinced but it is worth a look)

 

Lastly you can look at this interesting presentation by Barry Boehm and Richard Turner which talks about When to use which approach.

[Crossposted from my DDJ blog]

About the same time I wrote the post on whether architects should code, saying that architects should be able to prototype but shouldn't be part of the dev team (in the sense that the architect shouldn't get coding tasks that results in production code), Johanna Rothman wrote a blogpost that claimed architects must code .

Two days ago she posted a more detailed explanation of her view. I agree with most of the points she made:

1. Architects need to participate in the project; that is, not be some outsider who just drops her architecture on the team and leaves).
   
2. The best way to test a design is to code and run it.
   
3. It is beneficial for architects to know to code.
   
4. It is important that architects understand the implications of their decisions on the code and developers.

I don't see how architects taking coding tasks serves the greater good, versus their monitoring teams that code and making sure all aspects of the architecture actually fit the problem and work. Again, this may work on smaller projects, but probably not on larger ones.

You may also want to look at two related posts I made in the past
SAF Architecture Evaluation: Evaluation in Code talks about some of the ways architecture can be validated in code.
SAF Deployment: What to do when the architecture seems stable? talks about the architect's involvement in the project when they think the architecture is "finished".

A couple of points regarding the analogy Rothman uses--that is, architects who design bathrooms for hotels. Building architects are seldom a good analogy for software architects (I once used it as well). However, there are far too many differences (maybe I'll blog about that sometime in the future).

This brings me to the second point. This analogy doesn't serve Rothman's point well since building architects never actually participate in laying down brick or installing bathrooms. The fact that hotel bathrooms are not comfortable means that this quality was low on their priorities. In any event, verifying if a bathroom is usable--you don't have to install it just use it. (If you do take the analogy, you don't have to code it just stick around to see what's going on

Here is another SOA pattern from the list of patterns I am publishing.

---

One of the core goals of going with SOA is to enable loose coupling. The request-reply communication pattern, which is very prevalent, inhibits this decoupling. The problem is for the caller or consumer of the service - the consumer service is dependant on the timely response of the called service for its normal operations. To help elevate the consequences of this dependency the service that is consumed should maintain QOS (Quality of Service) as part of its contract (it doesn't have to be part of the machine-readable contract but it needs to be defined and adhered to). Consider for example an on-line music store. On a normal business day that can have few thousands of purchases nicely distributed around the clock. And then when a new <name your favorite band here> album debuts they can have much higher peaks than their usual requests load. They still need to be able to handle all coming requests or the (potential) buyers will take their business elsewhere.

 

How can I maintain a level of QOS, handle peaks and high-loads without my service failing?

 

One option is to estimate the peak loads and get enough computation power to ensure you can handle them – this causes problems. One is a problem of waste you can have machines just sitting there twiddling their thumbs so to speak. However the idle computers have purchase, maintenance and operational costs. The other problem is unexpected loads (e.g. Harry Potter craze for an Amazon-like site) – the estimated load might not be enough.

 

Ensuring QOS gets even more problematic when some of the actions performed in the service access resources/services that are not in the under the service control (- e.g. taking to a credit card clearing in the e-commerce example mentioned earlier).

 

Another issue that needs to be take care of is prioritizing requests – a Service most likely handles several types of requests – not all of them need the same level of QOS. You can set the QOS for according to the most demanding request type – but then you may need more resources.

 

Decouple the invocation- separate the reply from the request: Acknowledge receipt in the edge, pass incoming request to a queue, load-balance and  prioritize behind the queue.

 

 

Making the Edge acknowledge the receipt of the request (for our e-commerce example this can translate to "Your order has been received and is being processed, you would get confirmation email when the transaction completes") allows hiding operations that take long time from the service consumers (be that other services or end-users).

 

Writing requests to the Queue is a relatively low-cost operation that can be performed fast thus allowing handling request peaks. The actual handling of the incoming requests can be performed more slowly according to the available resources of the service. The load balancing can be done by setting different number of readers working against the queue.

 

Making the Queue a Priority Queue (or having several queues according to priority) allows for maintaining different levels of QOS for different message types.

Decoupled Invocation can be combined with the Gateway pattern to allow scaling out the service.

 

Decoupled Invocation is enhanced by the use of Correlated Messages pattern which helps relate the request and the reactions.

 

Acknowledge in the Service

Sometimes the initial response needs to involve some business logic and is not just an acknowledgment. In this case the Edge doesn't respond, it just passes the request to the service, the service sends both the initial reaction and the reaction.

 

 

Michael Platt talks about SOA vs. Web 2.0 He provides several links to blogs and article  that basically claim that SOA is dead and long live the new king Web 2.0.

 

One thing I have to say is that if indeed the hype around SOA is starting to calm - this is a very good sign. Finally we can go about adding SOA to our toolset and use it when it is appropriate (not just because management has got to have an SOA). Also it can be a good sign that SOA is maturing.

Another point I'd like to make is that SOA and Web 2.0 are not really related - there is no reason why one should compete with the other. Why using an AJAX front-end makes it impossible to have Services in the backend (it may be appropriate to have a Client/Server/Service Scenario - where the front-ends don't hit the services directly (the other option is Peer/Service) - I may talk about these 2 mini-patterns in my SOA pattern series). Another example where SOA and Web 2.0 can work together is RSS. A service can expose its list of recent changes as an RSS feed (as well as providing the more "traditional" web-services API). Exposing an RSS feed can be an option to implement the Inversion of Communication pattern I mentioned in a previous post).

 

To sum things up - Web 2.0 may be more hyped today than SOA. Web 2.0 and SOA can co-exist and actually complement each other.

In any event I think we (as an industry) should focus more on delivering great applications and solutions rather than fight about whose trend-du-jour is fancier or sexier.

 

[Edited]

After writing about the example of using RSS for Service communication I stumbled today on RSSBus which is an effort to create an ESB on top of RSS protocol ...

[crosspost from Dr. Dobb's Portal]

Test Driven Development (TDD) is, in a nutshell, writing a unit test up front--making it fail, making it work , refactor, and repeat until the product is finished. (If this is new to you, read more at testdriven.com )

So with TDD you get a bunch of unit tests that are also proven as regression tests. That's pretty cool.

TDD also lets you work in small increments while maintaining the working code. That's even cooler.

And lastly TDD has a very good influence on design:

1. It encourage loose-coupling. When you want make something testable you want to remove the dependencies it has so you can test it by itself.
2. It makes you think about the interface of the unit under test--how is the interface going to look?
3. It makes you think about how the unit under test would be used--for example, the behavior of what you are writing (or designing).

Sounds great to me. I think TDD is a great way to do the detailed design. You specify the results (interface + behavior), then implement that design. One thing I don't buy though is that TDD alone will produce an "emergent design" for the whole system. The way I see it is that you have to do some design up-front (assuming your system is not a trivial one) since TDD, being a coding technique, keeps you working at sea-level.

There's also a fundamental matter of scale--it might be possible in theory to start that 100 man-year project as a single object, then refactor it in baby-steps until you'd get the perfect system. I believe that if you don't work at a higher level of abstraction (vs. code), you will not be able to partition the system in a reasonable time. This was true when we moved from assembly code to higher level languages which enabled us to write much more complex software--and it is true today as we need to answer the ever-changing requirements of modern enterprises.

To sum up, TDD is good for testing and it is a good design methodology for the detailed design level. It can be used to drive the overall design on smaller project--but on larger systems we need additional methods and tools to cope with the overall design and architecture.

Scott Bellware attacks Microsoft's cluelessness of modern development methodologies and tools. By talking about the 3 (in?)famous typecasts (Mort, Elvis & Einstein personas) used by Microsoft to model developers.

 

While I agree with a lot of the points Scott makes I think that (unfortunately) there are individuals and organizations that are suitable for to the Mort-Elvis-Einstein approach -  Where people are not as smart or competent as Scott is and can't handle agility. Also there are situations  where agility cannot be practiced -e.g.  clients insist on fixed price projects and waterfall-ish milestones where, against our better judgment, we were forced to do a lot of up-front planning (that had to be reworked later…), safety-critical system etc.

 

I much prefer the direction Scott took in an earlier post where he talked about a missing persona - Hugo the Agilist. I think Microsoft will be making a grave mistake if they will not pay attention to the needs of the growing community of developers that prefer agile methods and practices.

Dr. Dobb's has recently launched a new portal site, where they want industry experts to post their views - well, I guess it is not limited to experts only, as they've also offered me to write there - I'll be writing the blog on software architecture and design.

 

I am going to post there on a daily basis (read: ~5 posts a week). Naturally they are going to be shorter posts that will try to highlight and comment (hopefully) interesting things related to architecture and design (my thoughts, other peoples posts, news etc.).

I am still trying to decide on the balance between this blog and the new one, but  I guess most longer posts will go here (though I may cross-post them) and that whitepapers and presentations will continue to be posted here. Also note that the new blog has a wider spectrum as it also talks about design.

 

You will find my new blog "If you build it…will they come" here.

There are 113 domain models over at eclipse.org site of all sorts of things.  I guess most models don't have any practical value (what will I do with a metamodel of COBOL) but there are several interesting ones of things like RSS, WSDL, and SDM

 

It is also interesting to note that the models are expressed in several forms including MS DSL models  , UML 

  and images . The transformation from format to format was done by code.

 

(Found via Steve Cook )

As promised, here is the first pattern. If you like this pattern but you think there is something missing to gain better understanding please drop me an email: arnon at rgoarchitects.com . Naturally any other comments are also welcome :)

---

Getting an SOA right is very hard, not so much because of the technical problems (we know how to deal with those, don't we?), but rather it is very hard to figure where to put the borders and keep the right business alignment.  Assuming you somehow managed that, the real fun begins - you now have to produce reports, dozens and dozens of reports. Many reports will fall within the boundaries of single services (if you have a good partition), however many reports will also require adding data from several services. For example, in a Telco scenario, you may have a Customer, Billing and Provisioning Service (a real-life example would have dozens of additional services) now a customer is calling customer care and you want the CRM to show everything about the customer what outstanding invoices does she have, what equipments and services (GPRS, UMTS, friends and family etc.) she got, what her status as a customer (loyal , VIP, senior citizen …)  open service requests etc. Things get much more complicated when you need to summarize or group data from multiple services 

 

How do you get a decent cross business entities report with the data scattered about in all those services?

 

One possible solution would be to create the report at the consuming end (e.g. UI) visit all of the services involved then do all the grouping, cross-cuts etc. This solution is not very good from the performance perspective (you need to get more data then needed and you have to post-process it). It is also problematic from the flexibility perspective each service involved has to expose interfaces to get the data for the specific query (otherwise you mobilize even more data).

 

Another option is to go straight to the data, you may still need to hit multiple database servers to get to the data but the performance will be better. The problem is this is throwing your service boundaries down the drain and introducing a lot of dependency.

 

A third is to create interim Services ("Entity Aggregation") - this works fine as long as you have real business reasons to do the aggregations (there is an overhead with adding business logic to handle the aggregated data) and as long as you only have few of those  (or you might end up with a single "service" with all the business).

 

Create an Aggregated Reporting Service by building  an Operational Data Store (ODS) to enable creating sophisticated reports on otherwise dispersed data 

 

 

 

The ODS is similar in concept to a data mart e.g. data is subject based, integrated, scrubbed etc. However,  the main differences are that the data is up-to-date and that there is little or no history.

 

For incoming data the Aggregated Reporting Edge performs the data transformations from contract data into reporting data. The service updates the ODS by scrubbing the data (can be limited unless the data has to go to a data mart / data warehouse) and then integrating it and De-normalize into subjects.  Incoming report request fill parameters for the pre-prepared reports.

 

One problem with Aggregated Reporting is that it is not a Business Service (i.e. it is a technical solution rather than a business oriented one) - however since unlike Entity Aggregation the data in Aggregated Reporting is Read-Only this doesn't affect the business.

 

Aggregated Reporting is easier to implement when combined with  Inversion of Communication

 

Aggregated reporting with Data Mart/Data Warehouse

 

Instead of just storing recent operational data, this version enhances the depth and complexity of queries that can be executed against the service. The downside is the increased complexity in setting up the data mart - both from the operational costs perspective (e.g. additional storage) and from the design and development perspective (you need to think about long term aspects, indexing etc.) as you also need to scrub data and consider the structure of your schemas much more carefully.

 

 

---

 

Sidebar: Operational Data Store (ODS)

The ODS is probably the best kept secret of data warehousing technology. It has been around almost as long but it isn't as famous.

The data in the ODS is operational - live data and not static data. The ODS can be thought of the as the cache memory of the data mart / data warehouse.

It is important to note that while it doesn't need the same amount of planning and set-up as a data mart, an ODS still requires careful planning in order to bring real business value.

 

The figure below shows the classical usage of an ODS in an OLTP/Data Mart environment.

 

Originally it was thought there would be 4 types of ODS

 

Class I - Near Real-Time synchronization of the ODS with operational data from the OLTP databases.  an implementation of Class I is the preferred type for the Aggregated Reporting pattern

Class II - Update the ODS every four hours or so

Class III - Overnight updates of the ODS

Class IV - the ODS is updated from  the data mart / data warehouse

 

In reality there are more variants - for example a powerful (and complex to build) option is to merge a Class IV ODS with one of the other Classes and get.

 

 

I decide to write a short series of blog post on SOA patterns. These are not patterns that are only usable for SOA, however, I have found them particularly useful in implementing SOAs.

 

This isn’t an exhaustive list of pattern - on the contrary I'll try not repeat patterns which are well known (like  Entity Aggregation  http://patternshare.org/default.aspx/Home.PP.EntityAggregation )

 

I am a little busy these days (e.g. I have to complete an architecture document for one of my projects) - so this post will only introduce the (first batch of) patterns . And the following posts (in the series) will expand on each one (i.e. explain  What to do, usage context, consequences etc.). Then, if I'll get good feedback maybe I'll publish some more.

 

So, what patterns are we talking about here?

Well:

 

• Gateway - How do you scale a service without exposing too many endpoints?

 

• Inversion of Communication - How do I get the data from other services without too much coupling?

 

• Biztalkize - How do I control volatile behavior inside the  service ?

 

• Aggregated Reporting - How do you get a decent cross business entities report with the data scattered about in all those services?

 

• Transparent Emergence - How do I know where to find a service?

 

• Decoupled invocation - How can I handle peaks and high-loads without my service failing?

 

• Orchestrated Choreography  - How do I expand the behavior of hard-to-change service (e.g. legacy systems exposed as services) ?

 

Well, I hope this sparkle enough interest to make you follow the rest of the posts on this subject :)

Uncle Bob (Apparently Robert C. Martin?)  writes about Architecture as a secondary effect .

The article postulate that : <Quote>

1. The main goal of architecture is flexibility, maintainability, and scalability.
2. But we have learned that the kind of unit tests and acceptance tests produced by the discipline of Test Driven Development are much more important to flexibility, maintainability, and scalability.
3. Therefore architecture is a second order effect and tests are the primary effect

 

I had not thought about this before this round table. Here we were, a bunch of architects and designers, strongly debating the role and procedure of architecture, and the conclusion we come up with is that all the effort and struggle we go through results in a secondary improvement in flexibility, maintainability, and scalability. Writing tests (writing them first) has the primary effect

 

</Quote>

I think Bob is missing/downplaying one very important aspect - and that is level of abstraction.

 

• Many agree (me included) that code is the final design artifact.
• Many agree (again me included) that TDD is a powerful design technique (you may want to check out TDD Misconceptions or Rocky Lhotka vs. the world as summed by Jeremy Miler)

 

However since code is (obviously) detailed design you just can't go straight to code (test code or otherwise). In order to cope with a large/complex problem you need to tackle the problem at higher levels of abstractions first. Even more so there may be a need to go through several abstractions levels before you start coding anything. Unfortunately there aren't any real options to test models*.

 

In my opinion you cannot escape designing architecture in other (non-code) models for any, but the most trivial, system.

 

The way I see it the correct approach is to

1. Work iteratively
2. Test early - i.e. make sure that the architecture designed really works as soon as possible (see, for example, my post on evaluating architecture in code )

 

So - Is Architecture a secondary effect?

No, sorry, but I really, really don't think so.

---

* There are some options to allow simulation and validation (i.e. tests ) of models in the embedded world (e.g. http://www.embeddedplus.com/EmbPlusSMST.php or http://www.ilogix.com/sublevel.aspx?id=286) - however I find that these approaches don't scale well to IT problems (which have a lot more variables and are usually much larger than embedded systems) or even to complex embedded system. You just have to specify too much before you get a usable simulation rendering the whole effort useless.

One of the roles of the software architect is to act as a mentor/coach. Reviewing some of the designs in one of my projects' teams it seems the time was ripe for doing just that. Thus, last week I gave them a presentation on the basics of good OO design  - which I thought might also be of interest for other people (you can download a copy here - 312KB).

 

The presentation starts with the  7 deadly sins of software design:

• Rigidity – make it hard to change
• Fragility – make it easy to break
• Immobility – make it hard to reuse
• Viscosity – make it hard to do the right thing
• Needless Complexity – over design
• Needless Repetition – error prone
• Not doing any

 

It is interesting to note that just yesterday I read an interesting piece on what makes good design (i.e. looking from the positive side) by James Shore (found via Sam Gentile) 

 

 

The main part of the presentation demonstrates the 5 basic design principles (drafted by people like Robert C. Martin , and Barbara Liskov ):

• OCP open-closed principle - a class should be open for extension but closed for modifications
• SRP single responsibility principle - a class should have a single responsibility
• ISP interface segregation principle - there should be separate interfaces for different consumer types
• LSP Liskov substitution principle - basically design by contract - a sub-class should fulfill the same expectations its suparclass set
• DIP dependency inversion principle - classes should depends on abstractions, class consumers should depend on abstractions and abstractions shouldn't depend on details.

 

These principles are the basis for  some of the techniques widely used today - few examples include:

Inversion Of Control - builds on OCP

Dependency Injection - a mechanism to allow DIP

Contract First - building on LSP,DIP

 

At the end of the day following these principles helps managing classes dependencies, increase overall loose coupling and cohesion thus increasing the overall quality of design. It sometimes amazes me how using just a  few simple rules can improve maintainability, flexibility and usefulness of designs so much.

For those of you who are following my writing on SAF (or the Milestone 1 post for more details) - here is an introductory presentation to SAF (1.2 Mb). I delivered a slightly modified version of this 2 days ago and someone commented he would like to see more information on strategies to deal with quality attributes. I'd be happy to hear any other comments you may have

Last week Beat Schwegler and Ingo Rammer visited Israel with the Software Factories Seminar  (The link is to the same seminar presented in  Finland -  videos are in English)

 

Software Factories is not a new idea  - see for example "Software reuse: From Library to Factory" by M. L. Griss  (published in 1993(!)) which talks about "Software Factories" and "Domain Specific kits": components, frameworks, glue languages etc.  The current Microsoft  incarnation of Software Factories takes a similar approach focusing on Domain Specific Languages, Frameworks but also adding important aspects like multiple viewpoints, patterns and designers. The idea is that  building on modern technologies, as well as learning from the mistakes from sister approaches to code generation (OMG's MDA, in case you are wondering) will enable us to build something that is useable.

 

Microsoft seems to be taking some steps in the right direction (GAT is probably the best example). Nevertheless there is still a long way to go before we can realize the dream of "factories" for vertical applications

 

This is evident if you take a look at the  current crop of DSL examples. These are either some Horizontal languages and tools  (i.e. not aimed at a specific business domain) like UIP designer or the GAT4WS beat talked about @ the event or even worse UML designers... ('nuff said).

 

The other thing is, that developing real factories, and getting them right is a really complicated  task, which requires a lot of domain knowledge and effort. In the recent Microsoft Architect's Journal  there's an article by Jack Greenfield and Mauro Regio regarding a software factory for the Health Level 7 ( a standard for Health organization collaboration) - As far as I know this project has been under way for more than a year now (first time I heard about it was Feb. 2005) and yet the article ends with :

 

"Our experience in developing a factory for HL7 collaboration ports

has shown that we need to define better frameworks, tools, and processes

to specify the factory schema, to manage factory configuration

in a flexible and extensible way, and to better understand how

and when domain-specific languages should be used."

My expericen with MDA shows similar results. Nevertheless, they continue to say that:

 

"At the same time, initial implementations of extension mechanisms like GAT and

DSL have proven their value, filling significant gaps in software factory

infrastructure, and pointing to future innovation in that area.

 

So, maybe there's hope after all :)

I recently found MSF 4.0 is out  - yep, it is still in two flavors…

 

MSF Agile (or officially "MSF for Agile Software Development Process Guidance") and MSF 4 CMMI (MSF for CMMI Process Improvement Process Guidance)

I consider MSF Agile as a lightweight process, however I prefer SCRUM as an Agile project management  process.

 

I find the CMMI version more interesting - Both for  cases where Agile is not a good fit (also see  Agile vs. Plan driven development  )

and/or for organizations that need to have CMMI certifications (such as the one I currently work for). MSF 4 CMMI covers level 3 pretty well and also has some guidance moving on to the next levels.

 

Definitely worth a look :)

I just read an excellent post by Gregor Hohpe talking about the motivation for Event Driven semantics for service communication.

Gregor gives an example of a shipping service listening on order events and address change events to produce shipments.

It is nice to see how architectural approaches transcends business domains so well -  The Naval C4I project Udi Dahan  and myself are working on, we basically try to take the same approach. For example: A Sensors service publishes its status every predefined time - The sensor knows if something is wrong with its state. A sensor, however, doesn't know if the problem is important or  not. We designed an Alerts service that listens in on status messages, based on (changing) business rules a certain status may trigger an alert event (which a UI can then choose to display); a severe alert may result in an SMS alerting a technician to come and have a look.

 

However, while this approach is very good for inter-service communication -things aren't as rosy when it  comes to interacting with UIs. The point is UIs  are based on interaction so the request-reply idiom (should actually be implemented as request-reaction) is much more prevalent. Users really want to know their request is being taken care of

 

Another lesson we learnt is that since services make go on-line and off-line independently of each other, it is not enough just to support event listening for event aggregators to be up-to-date. One option is to relay on reliable messaging to any event posted will eventually get to the listener - there are several problems with this approach for example:

•  For one, you need a reliable message transport which might be a problem e.g. you may not be able to use JMS/MSMQ between enterprises and/or the protocols you use don't support it (e.g. WS-RM is not durable see here  and here )
• Even if you have reliable communication,  if one service has been offline for a long period of time (where long is defined by the communication load) - it may be a waste of time (or plainly wrong)  to process old events that are no longer valid

 Another option to handle this situation is to supply in the contract request for current state (the current state can be published using the same message structure used by the matching event). The advantage here is that a server coming on-line can quickly and efficiently get up-to-speed on the current situation.

 

The event thinking is relatively on par with Take-it-or-leave-it approach for contracts construction, but as I said in the previous post on contracts, I think it is more beneficial to know about your consumer and take their input into account

 

Apropos EDA,  I also learnt today that the Micro-Services  strategy Udi and me came up with had already been "invented" several years ago. It is called SEDA (Staged Event Driven Architecture) there's a nice presentation explaining it here 

When I first learnt about WPF (it was still called Avalon back then) I thought it was strange that Microsoft is touting Smart Client development using WinForms when the WPF/XAML development model is much more closer to the code behind concept of ASP.NET.

 

Developers with ASP.NET experience will have life much  easier in the WPF era - They'll be able to utilize the same skill-set and thinking to build:

•  Regular, run of the mill, ASP.NET application
• AJAX application (utilizing Atlas)
• XBAP (XAML Browser Applications) - Smart clients running in the browser
• XAML stand-alone applications

While  the WinForm guys (and gals) will be left behind to try and learn the new  programming model.
Looking at the different CTPs of WPF it seemed all of us have a lot to learn in order put it to full use - so I guess no one will have it realy easy :).

 

Anyway, today I read Charles Petzold's "WinForms: More Vital Than Ever" blog post  - it seems that WinForms will stay with us for a while after all (ok, ok, so no real surprise there - I just wanted to make a point). By the way, it is especially interesting seeing this  coming from  Charles considering he is one of the authors currently working on WPF books 

 

ANother interesting point is that Microsoft is going into a lot of trouble unifying all the remoting and  distribution technologies into WCF on the one hand - but will leave us with 5 different technologies for UI development (ASP.NET, WinForms, DirectX, Atlas and WPF) on the other.

Yoni Commented  on my SAF - Deployment post :

 

"Somewhere in your blog you've mentioned Martin Fowler's "Who needs an Architect" article in a positive way. However, it seems that in contrast to Fowler's notions of an architect who's job is to "remove architecture by finding ways to eliminate irreversibility in software designs" - you are advocating making long-term binding decisions in the initial stages of the development cycle"

 

It is very hard to argue with Martin Fowler (as Gregor Hohpe put it in a recent post "Now who would want to argue with Martin Fowler? His opinions are facts " ) However, I think that devising solutions for flexibility (such as the database schema example in the article) are also architectural decisions. What makes things even "worse" is that these decisions are usually made at the expense of other quality attributes (e.g. performace) or other project constrains (e.g. time to market) - meaning that a decision on flexibility is a making  tradeoff  or in other words an architectural decision "par excelance".

 

Thus - architectural decisions, by definition,  are early  - whether we like that or not. Furthermore, Flexibility is an important quality attribute, one the architect brings into the table (in the same way that other stakeholders bring their concerns - e.g. the Project manager who want the project to end on time and on budget). It is the architect's role to balance quality attributes, understand the tradeoffs and make the decisions that will enable the project to achieve its goals. Many of these decisions have to be made early so that the project can move on, some of these decisions have to be made early to prevent ad-hoc architecture (un-balanced, spur of the moment decisions that will be hard to change). I believe very few decisions can be postponed without doing anything (i.e. without making some active decision to introduce flexibility).

 

Udi Dahan writes about "Contract First, Discussion Second?" saying that "a service's contract is a more "take-it, or leave-it" kind of deal." 

There are situations when this is true - for example when Amazon decided to expose some of its functionality as Services they probably didn't negotiate it with most (if not all) of us. Similarly, whenever you want to consume a deployed version of a service you can either use it as is or move on.

However, services are rarely developed in a void. This means that when you set up to design the next iteration of a service (first or otherwise) there are usually several potential consumers out there (other internal systems, partners etc.) - and like it or no, you will be negotiating the service contract with them, after all, the  whole idea of the service is to add some business value. If you disregard your consumers, it will make it harder on them to actually make use of the (hopefully) wonderful functionality you will be providing.

 

This, Also means that it is better to negotiate the contract first (i.e. as one of the first steps of  developing the next service version). Again, deciding on the contract upfront, allows the other parties to get organize to better take use of the functionality that will be exposed through the service once it is deployed.

 

I suggest you be pragmatic when you set up to develop a service, meet with the potential consumers and try to agree on something that will be useful for them - or as the Beatles once said "let it be, let it be, speaking words of WSDL, let it be"…

I've just found out (via  Gianpaolo's blog) that Roger Wolter (former PM of Service Broker) started blogging . He is going to focus on  Data in a Service Oriented world. I had a chance to work with Roger for a short time, which was enough to notice that if anyone knows about data, it is him.  I guess there is no surprise there, considering his past at Microsoft working on :SQL Server Service Broker, SQL Express, SQL XML Datatype, SOAP Toolkit, SQLXML and COM Plus.

His first post (after the obligatory "hello world" post) is about Service Broker positioning (vs. MSMQ, Biztalk and WCF) - Subscribed

It took awhile but I finally managed to complete describing all the activities of SAF at least at some level of detail.

What is SAF?

There is very little guidance on how one can go about designing/developing an architecture for a software project. The SPAMMED architecture framework (SAF) aims  to help fill this gap. 

 

SAF  is a set of activities that an architect can follow when she sets out to design an architecture. These activities helps the architect to keep abreast of the project's needs and the drivers that affect the architecture.

 

Overview posts:

Getting SPAMMED for architecture

SPAMMED State chart

 

 

SAF Activities

The Activities of SAF include

• Stakeholders -identify the stakeholders   - Anyone with vested interest in the project (end users, clients, project manager, developers etc.) These are the people you will have to explain you architecture to. These are the people that have concerns that the architecture will have to satisfy (or most likely balance). Thus, the fist step is to identify and rank them.

• Stakeholders Everywhere
• Stakeholder Template

 

• Principles – list Principles, Goals and Constrains. These are the properties you wish your architecture to have (or lack) based on your previous experience. Constrains can also come from your stakeholders (e.g. management decided that all project should be .NET, a deadline etc.)

• P is for Principles (and Guidelines and Constraints...) - Iteration I
• P is for Principles (and Guidelines and Constraints...) - Iteration II
• Principles Template
• Constraints Template

 

• Attributes –  discover quality attributes, the non-functional reqeirements, that (once prioritized) serve as the guide for the overall goodness of the system (Performace, Availability, scalability etc.)

• Quality Attributes - Introduction
• Utility Trees - Hatching quality attributes
• Architecture & Functional Requirements

 

 

• Model – model (and document if needed)  the architecture as seen from different viewpoints (list of viewpoints is stakeholder driven). Example for viewpoints include package diagrams, deployment diagrams, DB Schema etc. etc.

• Architectural Modeling - First Step
• Architectural Modeling - Choosing your Viewpoints
• Modeling - Architectural Styles
• Software Architecture Document Template

 

• Map – Technology mapping, buy vs. make decisions etc.

• SAF - Mapping -Don't forget the Technology

 

• Evaluate – Since architecture is the set of decision that are hardest to change it is worthwhile to spend some time trying to evaluate if they are indeed correct before commencing on

• SAF - Architecture Evaluation (Introduction)
• SAF - Architecture Evaluation - Evaluation in Code
• SAF - Evaluation part II - the "Formal Methods"

 

• Deploy – Software architectures are not a fire and forget thing. As an architect you still have to make sure that the guidelines set are indeed followed and even more importantly that the architecture chosen indeed match the project’s needs and doesn’t have to be reworked.

• SAF - Deployment - What to do when the architecture seems stable?

 

Feedback & Future

I would be very interested to hear any feedback on SAF - you can send  feedback to saf@rgoarchitects.com or just leave a comment

 

On the next SAF posts I am going to talk about SAF alignment with development methodologies (RUP, MSF 4, SCRUM etc.) as well as strategies for following or acting upon the activities.

 

 

 

 

 

The last activity of SAF is deployment of the architecture.  This step can make the difference between an ivory-tower architect and one whose designs are actually used in real software projects.

Deployment of the architecture actually  means two things

1. Verification and feedback loop. - making sure the architecture is actually the right one.
2. Governance - making sure that the architecture is followed by the designers and developers

Verification and Feedback loop

It is common practice to think of software development as an iterative process. Knowing that Software architecture  is early and that it encompass the important decision which are hard to change, it is probably wise to think of the first few iterations as architectural ones. You would try to work with the team and form the major abstractions, hopefully come up with a blueprint for processes etc. However, the fact that you've decided only the (let say) first two iterations are architectural ones, doesn't mean that new non-functional requirements won't emerge during later iterations. Since practice shows us it is (almost always) futile to try to analyze "all" the requirements before we do any design - this is almost bound to happen.

Be ready to monitor the project's progress after the "architectural iterations" to see and deal with any emergent requirements. As the stakeholders (e.g. product owner, project manager and the architect) try to prioritize tasks/requirements from iteration to iteration, hopefully most of the architectural issues would both surface and be taken care of the first few iterations (remember - architectural decisions are the ones that are hard to change…) - but unfortunately this isn't always so. I recently audited the architecture of a project (which has been running for more than a year) where they identifies the need for transactions, but following some stupid 80/20 rule decided that most of the system does not need that (deadly sin #1) and that they'd be able to easily patch it in later down the road (deadly sin #2) - so not enough just to identify a need, you also need to deal with it early.

The first few iterations also serve as a feedback loop on the suitability of the architecture, sure you've tried to evaluate  the architecture both in code  and on paper , but once it is deployed, or used for real - that is its real test. You will probably want to allocate some time to sit with developers and designers and actually see how the architecture is used and the implications of your decisions.

Governance

It isn't enough to design a software architecture, show it off at some architecture review and maybe at product demonstrations. Designers and developers are more than likely to bend your architecture to their convenience - this can happen for a multitude of reasons - for example:

• They flipped the bozo on you  - they think they know better
• They don't see the big picture and they try to optimize locally
• They don't know any better, they just do what they always did
• They believe they are following your design but they didn't really understand what you've meant (or alternatively - you didn't explain yourself very well)
• They cut corners e.g. to meet dead-lines (just to appease the project manager breathing over their shoulder)
• Etc.

When the architecture is compromised, it can definitely  have severe  effects on later iterations and the overall stability/usability of the system. It is important to note that  it may even result in harsh implications for the organization - imagine someone circumvents some auditing mechanisms you've put in place to comply with SOX or Basel II.

What this means for the architect is that she cannot disengage from the project completely even once the architecture is stabilized (i.e. after the first few iterations). Architect involvement is needed for design reviews, maybe key code reviews , on mile stones  etc.

I think it is one of  the architect rolls to act as a "comptroller" and oversee that the project stays on track as far as the architecture goes. Unfortunately there aren't too many automated tools to help with that. DSLs is one direction that may help (look for example at the Guidance and Automation Toolkit here  and here  ). I've also recently saw an application that claims it can enforce and monitor SOA policies on e.g. on contacts (for both Java and .NET). I hope we'll see more of these tools in the future - meanwhile it is up to us, architects, to do that as part of our overall responsibility.

 

Scott Bellware writes about Microsoft missing an Agilist Persona (in addition to Mort, Elvis and Einstein. I pretty much agree with Scott's views - MS's lack of understanding of the Agile crowd is evident in "fiascos" like the TDD article  on MSDN or even MSF Agile, which is  relatively a light process but still very far from processes such as  XP, SCRUM and the like.

 

Personas is a very interesting concept, usable as a communication aide, for development teams - as a means to help maintain user focus.

 

The CHAOS Chronicles 3.0 by Standish group* (www.standishgroup.com)  cite "User involvement" as second most important success factor for development project success  (after executive management support). Indeed many agile methodologies also encourage high customer involvement in development process

 

During my professional career, I had the chance to work for both product companies and solution companies.  - Why is that relevant you ask? Well, when you work for a solution company you usually have tangible, real-life, customers to work with. You can walk them through early usability prototypes, you can consult with then on problematic requirements, you can have them on site for instant feedback etc. etc.

 

Things are more problematic  when you work on "shrink wrapped" products - now your customers are much more abstract , and elusive, yes you can still hold focus groups etc. but your you can't have that day-to-day interaction with end-users and customers - enter Personas.

 

I first heard about Personas several years ago, when I read Alan Cooper's "The Inmates are running the Asylum". The book demonstrates some bad  designs for software-based products and then introduces an approach (Personas...) to help avoid these problems (He elaborates more on the approach in "About Face 2.0").

 

Personas are basically a way to define archetypes of users. Unlike Use Case Actors which represent a role in the system, Persons try to high-light the characteristics of real users. The idea is to come-up with representative model of a user and to give it a full-bio and characteristics  so as to help the development team understand motivations and relate to actual users. The model for absent users is the reason this technique is very  important for product companies. If you don't have real users come up with abstract ones representing them. Alan Cooper introduced Personas as a means to help designers in the initial phases of product design. Microsoft extended the use of Personas as a communication aide to the full range of the development team (designers, developers, testers, managers, marketers etc.) - which brings more benefits. I highly recommend reading the very  interesting paper "Personas: Practice and Theory" by John Pruitt and Jonathan Grudin which  relates Microsoft's experience on the subject.

 

While Personas are very important for product companies, it can also be important for solutions development especially on larger projects where only few members of the development team get a chance to interact with customers and user.

When you cannot have the customer on site (or the customer representative doesn't give you the full picture of all user types of the system) you can use Persona-Scenarios as a way to augment user stories (or in other circumstances as an alternative to actor-use cases)

 

---

  

*The Standish Group has been collecting metrics from thousands of projects since 1994. They analyze success and failure factors and publish them on yearly basis. For example, while our ratios as an industry are getting better over the years - as 2004 only about third of the projects achieved their goals both  on budget and on time...

Forget that stupid agile methods and all that iterative junk - Waterfall 2006 is here http://www.waterfall2006.com

or as the "report of the DEFENSE SCIENCE BOARD TASK FORCE ON DEFENSE SOFTWARE 2000"   sums nicely:

"About 90% of the time, the [waterfall] process results in a late, over-budget, fragile, and expensive-to maintain software system. A typical result of following the waterfall model is that integration and testing consume too much time and effort relative to the other software development activities. Most waterfall projects, expend over 40% of their effort and schedule in integration and testing."

Oh well, maybe we should stick with what we know after all :)

(Thanks Grady Booch and David Ing blogs for the conference link)

 

Being an Architect is a lonely job. You get to interact with all the stakeholder one can think of, and sure, everybody has an opinion, but in the end you get to make the important decisions by yourself. Even when an organization has  several architects, many times only one is assigned to a project at a given time. Well maybe it is time to move to pair architecting (pair programming redux for architects)

 

Over the past few months I've had the chance to work with another architect (Udi Dahan ) on an the architecture for a new product line.

 

This actually proved to be a very positive experience:

• You get informed feedback for ideas
• You get to look at a problem from more angles
• Working together helps refine the design (instant reviews and mutual feedback)
• You can play good cop/bad cop (or bad cop/worse cop :)) vs. The different stakeholders (PM, Devs, SMEs etc.)
• You can divide the work to get more things done (be less of a bottleneck)
  • It is also easier to work at the different levels required with less context switching ( presenting to non-technical customers, working with programmers, convincing upper management etc.)

 

Now, few iterations into the project, the architecture is pretty stabilized, but, we're still working together only now we mostly divide the work between us. I get to do the "fun" stuff - working with the marketing guys; working on the schedule and iterations with the project manager; etc. While Udi plays the "Architect as a coach" with the developers of the team as well as redesigning the clients (After we gave too much slack to the client team during the previous 2 iterations)

 

Now I wouldn't recommend bringing too many architects into the fray as this can easily degrade to a "design by committee" sort of effort  but it is definitely  beneficial to have more than one architect working on a project.

 

About a month ago Microsoft launched two Forums aimed at architects on the MSDN forums

   Architecture Forums: General Forum and Architecture Forums: Modeling and Tools

Here is the introduction as posted by Simon Guest :

"Welcome to the Architecture General Forum. This forum is for discussing general issues and experiences related to architecture and designing solutions on the Microsoft platform.  This forum is moderated by several members of Microsoft's Architecture Strategy Team.  We welcome all questions and comments related to architecture, although we recommend that product specific questions (for example, "I can't get x to install") are directed to more appropriate forums.  "

This is actually a bit of old news - last month the IASA (International Association of Software Architects) renewed web-site was launched with few interesting articles by Scott Ambler, Simon Guest and others.

You can find the site here

On the previous post on Architecture evaluation I talked about evaluating a candidate architecture in code. This post is dedicated to evaluation on paper.

 

I remember one system I was working on, I was keen on making the architecture asynchronous and message oriented (it was all circa 2001 by the way) However, I was new on the team and my role (as the project's architect) wasn't well defined so I had to do many compromises and get wide acceptance in order to get anything going forward. We set a team to try to come up with a suitable architecture, since each of the team members had his/her own experience we came out of these meeting with more than 20 (!) different candidate architectures (actually there were fewer architecture variations but they were multiplied by the possible technology mappings). Trying to decide which was the best option to follow we trying to conduct some sort of a QFD process where several members where in charge of the weights and the rest where in charge of evaluating and scoring the different categories (per option). Like most "design by committee" efforts this also proved a doomed from the start - and the option everybody disliked got the highest score. If you are wondering what happened - we scraped this effort and started from scratch in a more sensible way (which included a detailed prototype) - what's important for the purpose of this post is that it got me thinking that there must  be a better way to do evaluate architectures. Well, a lot of research and several projects later I think that there are few techniques that give much better results.

 

The first methodology I stumbled upon was ATAM (short for Architecture Tradeoffs Analysis Method), developed by SEI.

ATAM is a rather lengthy and formal method to evaluate architectures it requires a lot of preparation and commitment from the different stakeholders. You can get an overview of the process from the following (~130K) ATAM presentation  I prepared few years ago (While this is probably not  the best presentation in terms of presenting it to a crowd (I know better now :) )  it does provide a good overview of the 9 ATAM steps).

 

ATAM is explained in more details in "Evaluating Software Architectures", the book also details two more evaluation methods SAAM (which I'll let you read in the book) and ARID (Active Reviews for Intermediate Designs).

ARID, like ATAM, is a scenario based technique, meaning that as part of the evaluation process you need to identify scenarios where the system's quality attributes (see Quality attributes - Introduction ) occur or manifest themselves. The main idea in ARID is that for each (prioritized) scenario the participants try to draft code that solves that scenario utilizing/following the  design  under test. The results of the effort are then evaluated for ease of use, correctness etc.

There's a good introductory whitepaper on ARID in SEIs web site.

 

Note that ARID is more suited to agile/iterative development (compared with ATAM) since (as its name implies) it doesn't require the architecture to be completed and finalized up front.

 

While I was working for Microsoft, I stumbled upon another evaluation method called LAAAM (which is now a part of MSF 4.0 for CMMI Improvement). LAAAM which stands for Lightweight Architecture Alternative Analysis Method  is also scenario based and like ARID is more agile alternative to ATAM.

 

In LAAAM you create a matrix which has scenarios on one dimension and architectural approach, decision or strategy on the other dimension. Each cell is evaluated on three criteria:

• Fit - how viable is the approach to solve the scenario (including risk, alignment to the organization's standards, etc.)
• Development Cost
• Operations Cost

 

LAAAM was developed by Jeromy Carriere while he was working for Microsoft (he is now working for Fidelity Investments in Boston).

 

SAF works well with all of these techniques, as one of the basic steps is to identify the quality attributes and write down scenarios where these attributes manifest themselves in the system (see Utility Trees - Hatching quality attributes  )

 

To sum things up -

There are several ways to evaluate software architectures on paper - ATAM, ARID, LAAAM and few others (I didn't discuss here)

Scenarios based evaluations help verify quality attributes are being taken care of by the suggested architecture

Paper based evaluations can help reduce the number of options to few (hopefully one or two) leading solutions which can then be evaluated in code (as the previous post on this subject suggested)

 

For the first post for 2006 I'd thought I'd throw-in my .2 cents about SOA

(Note: this is 1.5MB ppt)

 

Today I attended a presentation on Service Orientation (SO) by Clemens Vasters. The presentation itself was very interesting, Clemens mentioned that SO is about business alignment and loose coupling (two claims I all heartedly agree with). He also made some claims about SOA which I only partially agree with (but that's not the point of this post).

 

Clemens, used the business practices of  the pre-computer era to build the case for service orientation - utilizing several metaphors such as Department = Service and folder+form = message etc.

 

From these metaphors we can basically  infer 2 types of service communication patterns:

• one way messages  - a service consumer can ask a service to perform some action
• Request/Reply - A consumer asks for a service and waits for the results (synchronously or asynchronously)

 

 

I guess there's a general limitation to metaphors - since a metaphor is (a kind of) a model of a problem, it doesn't catch/include all the properties or possibilities of the problem. Using a metaphor, we should be careful not to neglect options that are not covered by it.

 

In this particular case, it is hard to see other possible ways for services to communicate. Unlike business practices of the 1920s, things can be handles in parallel, a service may not care who deals with data it publishes e.g. in a publish subscribe  scenario or if there's an external (to the service) workflow that takes care of routing (read: orchestration in MS's lingo or Choreography in IBM's lingo) data to interested services. For example:

Using a publish/subscribe communication An Ordering service may publish any approved order and both the invoicing service  and a logistics service can trigger a JIT inventory process to ensure that supplies will be available to fulfill the order.

 

By the way, I think this point (regarding metaphors) is also valid for the "system metaphor" in extreme programming (but that's a point for another post)

 

Reading the November issue of Crosstalk I came across an interesting article on managing Architectural dependencies using a Dependency Structure Matrix (DSM) - rather than dependency graphs induced by UML. The article is called "Dependency Models to Manage Software Architecture" by Neeraj Sangal and Frank Waldman"

The article talks about the value of using DSM technique for managing architecture evolution as requirements change (analyzing ANT as an example) - Apropos "Architecture Evaluation",  I think the DSM technique can also be useful in early stages of the architectural design as a way to infer dependencies and to help promote loose coupling

 

 

In the introduction to architecture evaluation I said there are two approaches to evaluating a software architecture. This post talks about the first approach - evaluating an architecture in code.

 

POCs

The first evaluation-by-code tool is the  Proof of Concept (POC for short). Building a POC is about building a minimal amount of code implementing  a focused area of the architecture or the architecture's  technology mapping. The aim of the POC is to help weight alternatives (when you are contemplating which way to go), lower technical risks or lower stakeholders'  anxiety over an architectural choice.

POCs map quite well into XPs spikes

 

Lets look at a few POC examples (examples from my past projects)

 

Example 1: Validate the feasibility of an architectural direction

On one project we inherited this ugly application incorporating its own proprietary cgi web-server,  the architectural decision was to keep this as a black-box and develop the project using a better more scalable architecture (though we still needed to utilize functionality form the C++ server now and then). The challenge to make this happen was to be able to maintain and  pass the session from the rest of the application (JSP, Servlets and J2EE) onto the C++. A (successful) POC that tackled this issue allowed us to advance in the chosen architectural direction, reducing the risk significantly.

 

Example 2: Validate a technology mapping

On another project I worked for (when I was with Microsoft) we analyzed the project quality attributes and found that there's a need for near-fault tolerance (fail-over in 5 seconds or less). The architectural solution that we decided on was to use an active server and a semi-active one (on-line, ready to take over server that constantly applies state from the active server)* - for technology mapping we considered several options (e.g. fault-tolerant hardware ). One of the option considered was using SQL Server 2005 Database mirroring to keep the two servers synchronized (DB mirroring gives you a failover of the DB in about 5 seconds or less). In order to verify this direction I set up a small Proof of concept to verify that this direction is viable. I was told that after I left Microsoft, further investigation of the issues found led to Microsoft's decision to postponed Mirroring for now. 

 

Example 3: comparing alternatives

We wanted to compare MSMQ vs. using an existing distributed object middleware both in terms of performance and usability (is it developer-friendly). We crafted 2 POCs one for each technology which enabled us to compare the two approaches head-to-head.

 

POCs help evaluate alternatives and  lower risk in specific areas of the architecture (and design for that matter). However, POCs will not give you a feel on how the overall architecture  will play together - enter prototypes

 

Prototypes

A prototype is basically a working simplified model of the system. There are many characteristics to distinguish between different types of prototypes (hi-fidelity/low-fidelity, global/local etc.) - let focus on two:

• Horizontal prototype - which models wide aspects of a single layer, i.e. many features with little details. The most common example for Horizontal prototype is a  user interface prototype which is used to test the overall interaction with the system.
• Vertical prototype -Implementing some sub-system or a limited set of features across all layers /modules.

 

The Vertical prototype is useful way evaluating, getting a feel and understanding of how the different components, that makes  the architecture, work in unison without getting bogged down in all the fine details of the system's functional requirements.

 

Example: Using a prototype to evaluate an architecture alternative.

We were getting ready to embark on a rather large project (we did the prototype around the release of .Net 1.0 and the project is still going on…). We wanted to understand the capabilities  and limitations of .NET. We chose a limited aspect of the system (which we considered the most risky), chose some of the designated team-leaders and took an architect from Microsoft Consulting Services to help us build the "by the book" architecture.

We did a very extensive prototype, total effort of 3-4 man-years including all the preliminary work and the post-mortem analysis. We gained a lot of insights on what .NET can and cannot give us out of the box, we understood the limitations of the components we integrated (e.g. ESRI's limitations in displaying near-realtime moving objects) and we  also used the preliminary prototype (which was a performance hog) as a platform for running POCs for other architectural and technological directions. Additionally, once we solved the performance problems, we also used it as a demo for the client.

By the way, this experience also had some additional positive residual effects like, getting the team leaders up-to-speed on the (then) new technology, jelling the core team  etc.

Taking all the information gathered during the prototype we were able to design a better, more robust architecture for the project itself (which the architect, that came after I left the project, managed to mangle - However that's another story altogether :) )

 

I've found that in most cases exploratory prototypes, or "Throwaway prototypes" are more useful as they really let you get to the crux of the matter quickly, i.e. getting all the components connected the way the architecture dictates to test their interactions and usage. Again, the idea here is to focus on evaluating the architecture, not on the implementation details of the overall solution. Nevertheless, once the architecture is more mature you may choose one of the prototypes and evolve it into the actual system (sort of turning it into an architectural skeleton).

 

Architectural Skeletons

Once you've decided on a candidate architecture (i.e. the architecture you want to use for the project) your first iteration or two (This might not be the first iteration as  you may already done a couple or so prototype iterations) should be focused on creating the architecture skeleton.

Architecture skeleton is about implementing the minimal set (bare bones, so to speak) of the project's functionality that is needed to connect all the pieces in a meaningful, integrated way (for example it can includes an implementation of a single thread in a use case or a important story). It is somewhat similar to a prototype, with 2 differences :

• It has to  implement real functionality of the system (though the functionality is, usually, very thin)
• You don't throw it away (hopefully anyway)

 

Most current methodologies (RUP, MSF for CMMI Improvement, XP etc.) support the notion of architectural skeleton (though not using this name). In RUP, for example, you would have the architectural skeleton up and running at the end of the elaboration phase - a running architecture which you can expand and functionality to in the construction phase.

 

It is important to implement a skeleton (vs. starting to implement the different components and try to integrate them later) as it gives you a relatively early opportunity to actually test if your architecture holds, and it is much better to find errors, especially architectural ones, as early as possible.

 

Summary

I demonstrated  3 "tools" to enable evaluation of architectural decisions in general and the overall architecture in particular:

• POC - focused on a specific area
• Prototype - overall architecture with "simulated" behavior
• Skeleton - "barely running" implementation of the chosen architecture.

 

The problem with these approaches, especially prototype and skeletons is that they require a relatively long time as well as resources to implement. We need some additional tools in our evaluation toolset to allow us to focus on architecture alternatives that are most likely to match our needs.

I think that there are such tools, and on the next post on architecture evaluation I will try give my view on what they are and how to use them.

 

---

* other options are active-active and active-passive (e.g. Windows clustering)

 

Ok, so you've designed this grand glimmering SOA, and you have a few dozens services each assigned to a development team.

 

As the iterations spans the contracts have to be amended (be that WSDLs, plain old messages, object APIs or whatever) - do not leave the task of changing, aging or maturing to your developers, team leads or anyone else - I strongly believe that the architect, having defined the boundaries between the major components (services in this case) is the one also responsible for the space between the services. Anything that travels this space is in the realm of the architect*

 

In my opinion, failure to be in charge of the contracts can result in major breakage of the architecture (chatty interfaces, security problems, services dependency to name a few of the possible and likely problems)

 

I've been a little busy in the last couple of weeks and unfortunately, didn't have time to blog - to get my self started again - here is a short introductory post on the subject of Architecture evaluation:

I said  in "what's Software architecture" - architecture is both an early artifact and it also represents the significant decisions about the system - or to sum it up  "Architecture is the decisions that you wish you could get right early in a project.”  (Ralph Johnston*). That is exactly why I made evaluation one of the key steps in SAF. We want to raise the level of certainty that the direction(s) we are taking are indeed the right ones and we will not hit a wall later on. Especially considering most projects these days are iterative, which makes this even more challenging.

But, how do you evaluate an architecture? (on a give set of quality attributes) How can you tell if utilizing, say, SOA, will yield better results than distributed objects ? Is using fault-tolerant hardware better than using a database (on performance vs. robustness trade-off)? and so on and so forth.  Even if we say that flexibility and the ability to cope and embrace change is the most important trait (read quality attribute)for our architecture   we still need a way to evaluate which approach will give us the most flexibility

The way I see it there are basically two approaches for evaluating software architecture

• In code - as in proof of concepts, architectural prototypes and architectural skeletons
• On paper/discussion based -so called  "formal" methods: ATAM, ARID, LAAAM to name a few

 

I will try to use the next few posts on SAF to elaborate on these issues - where, the next post on SAF - Evaluation will explain and demonstrate the "in code" methods. The post following that, will talk about the paper based methods, and the third post on the subject, will try to contrast the two approaches and  talk about their respective pros and cons

 

---

*As quoted by Martin fowler in "Who needs an Architect"  - which, by the way, provides for a very interesting reading on architecture and the architect's role.

I've posted in the past about the importance of stakeholders for the architecture design process (in  "Stakeholders everywhere"). I recently stumbled upon an interesting article called "Understanding Organizational Stakeholders for Design Success" by Jonathan Boutelle that goes into depth discussing both the importance of stakeholders and the process of mapping thier interests and influence (which I also mentioned in my post)

The next step in SAF after Modeling is technology Mapping. While mapping  is not a part of the architecture per se, it is, in my opinion,  an important and sometime crucial step.

Before I rumble on explaining why I think this is an important step, let me try to define what exactly do I mean by "technology mapping"

 

Architecture in essence is technology neutral - it describes the major components (read objects/services/components etc.) and their interactions -  but it doesn't specify what technologies and what technologies, products or existing assets will be used to implement it - that's where Technology Mapping steps in.

 

For example, in a previous post (Architectural Modeling - a First Step ) I presented the following block diagram as a possible view for an architecture (Layer diagram):

 

 

One possible technology mapping for this (view of an) architecture  is depicted in the following diagram:

 

 

Before I continue any further, I should probably say that I think that technology mapping is basically  a design activity and not an architectural one. But wait, if that is true, why am I mentioning this as a step in a framework for building architectures (SAF ..) ?

 

Well, besides the obvious answer, that it helped me create a nice acronym for the process - being the second M in SAPMMED,  Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible.

 

For example, in one of the projects I am currently working on we made the decision to create an SOA (surprise, surprise*). We decided that the various services shall communicate using a service-bus and our intention was to map that to a messaging product (such as Microsoft's MSMQ  or Tibco's Rendezvous ). As it happens, the powers that be (i.e. upper management) decided that there is no room for investing in a new inter-process communication solution and that the project will be forced to reuse the existing solution. The existing solution, in this case, is a proprietary distributed objects solution developed in house. While our intention was to promote message-oriented development and contracts. The existing middleware, being a distributed object framework induce chatty RPC-like contracts (you can read more on RPC vs. message-orientation in "Mort gets the message" and few of the other posts on John Cavnar-Johnson's blog ). The implications of this technology decision is that the architecture team has to closely pay attention** (on daily level for the initial iterations) to what the designers and developers do so as to ensure that the architectural decisions are kept. 

 

On another project I worked for in the past we designed the solution to use an application server (multi-tiered hardware architecture) however the solution also had to incorporate ESRI's ArcGIS which (at the time) only worked in a "two-tier" client/server manner with its underlying data. In order to support the implementation of an application server (which we felt was really needed) we had to (decide and) develop an independent "entity layer" on top of ArcGIS. Failure to notice that the technology mapping implications  would have resulted in the architecture not being implemented (and serious scalability issues for that particular project).

 

The second reason for including Technology mapping in the architectural process is to help promote reuse in projects by making this as an activity at the architectural level (i.e. both early in the process, and making the decision to  reuse a component  a global decision governing the solution).

Making reuse a first-class citizen in the architectural effort can greatly help promoting product-line approach (examples for additional factors that can help promote product-lines include domain-driven design or concepts like software factories)

 

To recap:

• Technology mapping is about deciding which products, technologies and existing assets are going to be utilized for implementing the architecture.
• Technology mapping can have a significant impact on the ability to adhere to the architecture to the point where under certain mapping constraints you may need to reevaluate the architecture (i.e. is it worth-while to go "against the stream" and implement the architecture using a technology/components whose architecture.
• Deciding which existing assets can be reused at the architectural phase helps to create systematic reuse (vs.  opportunistic reuse) which improves time-to-market and solution quality (assuming you carefully choose assets for reuse )

 

I recommend either making technology mapping a required step in the architectural process or alternatively revisiting the architecture once this step occur as part of the design.

 

---

*when you get over the incredible amount of hype around it (I believe) SOA does have several distinct business and technological advantages - I'll try to elaborate more on this in one of the next posts

 

**I'll talk more on what happens once the architecture is "released" for public consumption when I'll coved D- Deployment phase of SAF

I recently read Weblog Usability: The Top Ten Design Mistakes by Jacob Nielsen  [found via Jeff Tash's IT Scout blog]

The first 2 mistakes Jacob discusses are  missing author photo and bio - since these are relatively easy to fix, I went on and mended the situation. I've added a photo  on the sidebar and I've posted a short bio in a new "about me" section.

 

Avoiding the other 8 mistakes require more effort - but I think (read: hope) I have most of them covered.

 

I'll take this opportunity to make a quick note on the architecture posts. I see that there's a lot to say about architecture modeling (oh my, what a surprise). I want to cover SAF at a certain level before I get lost in the details - so if you found SAF interesting thus far, watch out for posts on Mapping, Evaluation and Deployment soon :)

The PDC 05 videos are up @ http://microsoft.sitestream.com/PDC05/

 

I claimed in the past (on my "what's software architecture" post) that Architecture is a type of design - if that is true, an interesting question is do we also have architectural patterns ?

 

I think the answer is yes - there are architectural patterns they are also called architectural styles - I actually like this term better as it is helps differentiate them from design patterns; for example I agree with  Harry Pierson's observation  that many of the patterns in Martin Fowler's PoEAA are indeed technical patterns at an engineering or design level and not architectural patterns.

 

The difference between architecture styles and design patterns is similar to  the distinction between architecture and design - architectural styles effect the solution globally or at least it affects major parts of the solution and not solve local issues. Although it is interesting to note that some architecture styles have been well known well before the notion of design patterns for software was introduced.

 

SEI's architecture glossary defines Architecture Style as

"A specialization of element and relation types, together with a set of constraints on how they can be used."

That's a good start but it might be a little hard to understand we can basically say that an architectural style  defines a family of solutions in terms of a pattern of structural organization using a vocabulary of  components and connector types (plus constraints on their use). It is also worth mentioning that different styles can be combined to create compound or derived styles.

I'll try to illustrate what a style is using an example:

 

One of the basic architecture styles is "Layered architecture":

 

The layered style is composed of layers (the components) which provides facilities and has a specific roles. The layers have communication paths / dependencies (the connectors).

In a layered style a layer has some limitations on how it can communicate with other layers (the constraints). Typically a layered is allowed to call only the layer below it and be called only by the layer above it (but there are variants e.g.  a layer can call to any layer below it; vertical layers that can call multiple layers; etc. - as long as the layers communication paths are limited by some rules)

 

You can see the application of layered style all over the place for example: logical software layers (e.g. presentation component, UI Controllers, business processes, business components, data entities, data access layer) , SOA layers (fundamental , Intermediate, Process), physical tiers (e.g. database server, application server, web server and clients)  etc.

 

There are many other architecture styles including for example pipe and filter, push based, peer-to-peer, blackboard, MVC, PAC, or the more recent Adaptive Object Models, REST and SOA (some will probably disagree that SOA is a style - but I'll try to explain why I think it is on another opportunity). I'll try to talk a little about some of them in the next posts.

I just read Paul Kimmel's very interesting article on "truisms" of software development called "Un-Dynamics of Software Development, or, Don't Bite the Flip Bozo" . The article is written in an amusing/cynical tone, however a lot (if not all) of it is really true -

"Flipping the Bozo bit " by the way is "to make a mental note that a particular person is a bozo and everything they say in the future should be ignored or looked upon as the meanderings of a slightly annoying, occasionally amusing child or a drunken uncle"

Paul comments that the mean time for someone in the crowd to flip the bozo bit on you when you start speaking is 10 seconds.

The article also mentions stuff like

• When someone says the schedule is going to be missed, they are never lying.
• If a manager says I am not technical, be prepared to spend a lot of time explaining things to them so they can make decisions they shouldn't be making.
• Managers hire experts and ignore them all the time.

[found via Mitch Barnett's blog on software industrialization - which provides for interesting reading by itself (focusing with Software Factories and DSLs and general software development). his company developed a Biztalk appliance, which you may want to checkout if you are using Biztalk*]

---

* I have my views and some reservations regarding Biztalk - but I guess that a subject for another post :)

Ok, so we've identifies stakeholders, set principles and guidelines, found out what are our architectural requirements and we want to start modeling already - especially considering architectural modeling is great fun (for techno-geeks like myself anyway).  However, before we just start to create an endless  flurry  of blocks, boxes, arrows and whatnot, it is probably worthwhile to take a few minutes to think which Models we want to create and what views do we want to use to communicate them otherwise we may end up doing a very good job thinking about and  describing something that doesn't interest anyone and gives no real value.

 

IEEE 1471 defines (an architectural) view as a representation of a whole system from the perspective of a set of concerns (where concerns are the key interests of the different stakeholders)

A view is comprised of parts of one or more models to demonstrate the how the concerns are covered

A view is (sort of) an instantiation of the pattern defined in a viewpoint (see more below)

 

So for an example if our concerns are concurrency and timing issues looking from this viewpoint can  be looking at the threads and process model of the solution and we can use views like process diagrams and timing diagrams to visualize it.

 

One thing we can do (over time) is to create a viewpoint repository and then when faced with a set of concerns  we can easily choose which views to create.

Rich Hilliard suggests that a viewpoint would hold the following information:

• Viewpoint name
• The stakeholders addressed by the viewpoint
• The stakeholder concerns to be addressed by the viewpoint
• The viewpoint language, modeling techniques, or analytical methods used
• The source, if any, of the viewpoint (e.g., author, literature citation)

  A viewpoint may also include:

•  Any consistency or completeness checks associated with the underlying method to be applied to models within the view
• Any evaluation or analysis techniques to be applied to models within the view
• Any heuristics, patterns, or other guidelines which aid in the synthesis of an associated view or its models

 

If you are just starting out you can turn to various frameworks to see which views they have and use that as the basis of your repository for example:

RUP suggest a 4+1 set of viewpoints:

 

Microsoft is currently moving from the MSF 3.0 model (4 spheres - contextual, conceptual, logical, physical crossed with 4 "views" -business view, applications view information view and technology view totaling in  16 viewpoints) to the following set of viewpoints:

 

Other examples are DODAF (3 main viewpoints with 26 sub-viewpoints), RM-ODP (5 viewpoints)  and Zachman Framework (with 36 viewpoints)

          

         Lastly the software architecture document  template  I've posted also contains a list of possible views

 

 

What about a minimal set of viewpoints? Well, I don't know about a minimal set - there are however few viewpoints that  usually get used:

Some sort layer view (usually block diagram)

A logical view (main classes/packages)

A deployment diagram (tiers, zones etc.)

A view to show concurrency and timing issues.

On SOAs there's also a service view (services, policies)

 

To wrap-up -

• Choosing which views to create (viewpoints) is an important step before embarking on modeling.
• Viewpoints are chosen according to the stakeholder's concerns (to help make sure the concerns are addressed and to communicate the architecture better).
• Growing a viewpoint repository is a way to reuse your knowledge of concerns to views mapping.

 

Harry Pierson writes in his blog a very interesting piece investigating the tenets of good models.

 

While I tend to agree that there is a place for precise formal models that can be transformed easily to lower levels.

I would also like to argue that

• I think that imprecise models are also very useful, since at different points in time during the development you cannot fully specify all the finest details (even if for the "current" level of abstraction) esp. since most projects these days are iterative.
• Which brings me to the next point - for imprecise models - I don't necessarily think that there's a need to keep (all of) them updated during the development life-cycle. The high-level designs can be replaces by detailed designs and they in turn can be replaces with the code itself - good code explains itself beautifully :) .
• You should carefully weight the ROI for creating such a precise model. For example I happened to work on a large (hundreds of man-year) project where the initial thought was to use a tool (Vitech's Core) to for requirements analysis. The benefit was that (if done right) the model created can be "run" using their built-in simulator. After spending more than half a year (of a rather large team) we finally decided to drop this precise model for a much less precise model of use-cases which allow for a varying level of abstraction. It should be noted though, that (cross-subsystems) use cases are later refined into a DSL  which is actively used for generating cross-subsystem interfaces and simulate  missing system during  integrations.
• Another point from the former example is on the timing of requiring the precision. Modeling tools should allow several levels of precision, since in earlier stages you (usually) cannot determine all the bits and bytes that will allow for a "deterministic transformation"

 

 

Just my 2 cents

 

I've added a sample skeletal  template for a Software Architecture Document  on the SAF page.

 

Few notes on the template:

• It is based on formality level required for large safety critical projects - most projects do not need this level of formality and can (and should) do with fewer views
• When there isn't a specific requirement by the customer to create an "official" architecture document - you probably want to have it just barely good enough
• The view list in this template is not an exhaustive list - these just the views I've used most often. (more on views on the next post :) )

 

 

Any feedback (either as a comment or to my email) is more than welcome

Michael Platt talks   about the basic questions related to architects (what is architecture?; why are architects needed?; what are architects?; what skills do architects have?; and what are the types of architect). I agree with some of the things he says - especially the discussion on the architect skills.

I think Michael's definition of architecture is too simplistic (see more below) and I don't agree  with his classification of architects

 (here    is what I think on architecture types). For example he bundles solution architects under technical architects  I believe solution architect also have a lot to do with the problem domain and not just the technical or technological sides of the solution.

 

He also defines a "product architect" :

"Product level architects have an in depth understanding of the use of a specific product in a technical architecture domain such as Lotus notes in the messaging domain. Typical product architects are Exchange, SQL Server (normally as a DBA), Windows, and Networking etc"

I don't think that people under these roles are actually architects - they are definitely specialist and they may very well be experts but the breadth of their designs is very local in the scope of a complete solution and their skills will never be used on their own -  for example even if you do a data warehouse  project designing the database is (a very important) part of the project but there's still a lot to do with getting the data into there and deciding what data you want to store.

Architecture talks about things in the global (in the context of a project/solution/product line/enterprise) and design deals with local issues (how to model the UI, optimize the DB, set up Lotus Notes etc.)

 

What do you think?

The next step in SAF (after "quality Attributes") is Modeling.  Webster's dictionary defines "Model"  as (among other things) : "3 : structural design; 7 : Archetype ; 10b : a type or design of product (as a car);11 : a description or analogy used to help visualize something (as an atom) that cannot be directly observed" - as I mentioned in my what's software architecture post there's no single structure that IS the architecture - this means that we'll have to look at the architecture from different angles (viewpoints) - for example a block diagram such as the diagram below  (accompanied with some documentation that explains it) may tell us something about the layers that will be used to solve a problem it tells us nothing about the business that we are trying to solve.

 

 

We need to augment with more views on the system (such as the next diagram) so that we can better visualize and convey the architecture of the system.

 

 

 

IEEE 1471-2000  "Recommended Practice for Architectural Description of Software Intensive Systems"  defines the relationship between the Stakeholders, their concerns, viewpoints, views and models:

(IEEE 1471 model adapted from a presentation by Rich Hilliard )

 

By the way, the fact that we need to look at the architecture from different viewpoints doesn't  necessarily  mean that the documentation isn't just POW("plain old whiteboard") - The formality of the documentation is driven by the project style (agile/formal) and other stakeholder constraints (company standards, customer requirements  etc.)

 

Since models participate in views (which in turn conforms to viewpoints -  which address the stakeholder's need/interested in) I consider choosing the views a prerequisite for modeling. Thus, on the next post on modeling I am going to talk a little about choosing views (suggested minimal set,  viewpoint library etc.). Once I'll get that off the table I'll try to talk a little on architectural styles and patterns and follow that with some strategies for dealing with quality attributes before moving to the next SAF phase (Mapping).

 

 

 

Rich Turner writes about architecture & change. I agree with a lot of what he says.

I often find myself explaining to stakeholder (PMs, developers, etc.) that the only constant thing in the project is that it is that it is changing :)

 

 

One project I am currently working on,  exhibits this inherent change - For one the  customers don't really know yet what - they want (think) that they would be able to use the exact same solution for a  several very different configurations (ranging from standalone computer to a server farm with dozens of clients) etc. 

What do we, as architects, can do to handle these situations? Well, what I usually do is add a lot of modifiability related quality attributes (see utility trees) these can include requirements for interoperability, adaptability, changeability etc. etc. (An example scenario may be: provided with a verified algorithm,  replacing an existing navigation  algorithm shall take less than 3 man-weeks).

 

While I will (probably) talk a lot more on strategies for handling quality attributes in posts regarding modeling (as part of my SAF posts), SOA (sans hype…) is a good strategy to cope with flexibility (i.e. changing requirements). The explicit boundaries and contract first approach help localize changes  also the resulted loose coupling help to replace, add and remove services easier, Lastly the basing communication on policy helps postponing issues that has to do with the network (QOS, security etc.)

 

 

A completely different angle on the issue of changes -  is that sometimes it can be problematic to allow it, even at the expense of missing some of the customer's changing requirements. An example (maybe the only one) is for critical systems, where a defect can result in loss of lives  (two polar examples are medical systems on the one hand and weapon systems on the other). In order to be able to ensure software safety (Identifying hazards, proper fault tree analysis etc.) there's a need to affix the requirements for longer periods compared with other types of projects.

It seems the PDC presentations are available here ...

(like everyone else ?) I've finished reading the C#3 Spec and the LINQ project overview.

with samples like:

public void Linq43() {
   List customers = GetCustomerList();

   var customerOrderGroups = 
         from c in customers
            select
               new {c.CompanyName, 
                  YearGroups =
                     from o in c.Orders
                        group o by o.OrderDate.Year into yg
                        select
                           new {Year = yg.Key,
                              MonthGroups = 
                                 from o in yg.Group
                                 group o by o.OrderDate.Month into mg
                                 select new {Month = mg.Key, Orders = mg.Group}
                                }
                    };

ObjectDumper.Write(customerOrderGroups, 3);
}

Hey - that's de ja vous all over again :)

Actually it is (er..will be) much stronger than SQL - for one, you can use a single LINQ query to combine data from arrays and data and XML and whatever. Also combined with lambda expressions (next gen for anonymous methods ) you'd have even more power.

While I am on the subject of what's new in C# 3.0 - I also found Extension methods very interesting - it kind of lets you have a look and feel of multiple inheritance without the associated headaches

consider the following

namespace MultipleInheritanceSimulation
{
   public static class Extensions 
   {
      public static void DoSomething<T>(this T var) 
      {
         //do whatever
      }

   }
}

And then:

using MultipleInheritanceSimulation;

namespace Tester
{
   public class Foo
   {
      public void Bar()
      {
         MyClass x;
         MyOtherClass y;
   
         x.DoSomething();
         y.DoSomething();
      }
   }
}

 

oh well,enough playing - back to work...

In previous posts (here  and here) it seems I downplayed the importance of functional requirements (vs. quality attributes) on the architecture. Nevertheless the functional requirements do have few  important roles in shaping/looking at  the architecture. One aspect of the functional requirements role was demonstrated in the scenarios that describe the instantiation of quality attributes within the system. Lets look at a couple of other aspects.

 

As I mentioned in a previous post no single structure is the architecture - this means that in order to present an architecture it is needed to describe it from several different angles or viewpoints (I'll spend some time taking about these in the next few posts on SAF). One of these has to do with the domain architecture for the solution. This can include identifying business areas trying to identify services (in an SOA), identifying major components for a component based architecture, or even identifying major use cases in a "use case view" (as part of a Unified Process 4+1 approach )

 

Listening to "Almost Cut My Hair" by Crosby, Stills & Nash, in the background kinds of brings me to the 3rd area where  I see functional requirements meets architectural decisions. When there are extreme/hard/important requirements there are many times where you have to decide whether to cut your hair and  bend the entire design to answer that requirement (i.e. make that a global decision - hence an architectural one) or satisfy it by a specialized local solution (i.e.  Postpone it to the design phase). For example, I once worked on a system where we identified one area of the solution that needs a (relatively) high update rate (low latency, high throughput for updates coming from an external system, processing it within the system and  all the way to the user's workstation window). While this was both predicted to be frequently used and an essential requirement for the success of the system the majority of the system's functionality did not have these characteristics. The decision I've made was to treat it as a local issue (to be given a specific solution @ design time). (Unfortunately ?) I moved to another company before the project ended, and the architect the followed me took the decision the opposite decision (i.e. to make the solution for that problem an architectural solution for all the system's "entities") - which resulted in making all the interactions within the system (even the simplest ones) asynchronous. I recently  had a chance to see the effects of this decision on the system's schedule, robustness and complexity, well let me just say that the lesson here is that while cutting your hair (making an architectural decision) is not an irreversible decision, you cannot just undo it instantly and it can take quite a long time (=money) to correct things.

 

To sum things up

• Functional requirements manifest themselves as part of the utility tree (as the scenarios),
• It can also be important to view the architecture from the functional perspective
• Significant/important functional requirement should be weighted for their influence on the system's architecture (should they get a local treatment (as part of the design) or affect the system globally)

Grady Booch (one of the "three amigos" - fathers of UML) added a gellery of Architecture views from the literature (jpg's) to his Architecture Handbook site

Note: registration needed

In the previous post  about SAF I introduced the concept of quality attributes. I wrote that using a "utility tree" approach is a very good way to identify, document and prioritize quality attributes. The purpose of this post is to expand on this issue

 

As I mentioned before, MSF 4 for CMMI improvement make use of LAAAM (developed by Microsoft's Jeromy Carriere )

) for assessing the architecture (it is used there for assessing the architecture, which is also a good place to use it - but I'll talk about that when I get to E(valuation) of SAF.). LAAAM also builds on a "utility tree, below are the sub-activities mentioned in the MSF beta bits:

 

• Examine quality of service requirements and product requirements to determine the key drivers of quality and function in the application.
• Construct a utility tree that represents the overall quality of the application. The root node in the tree is labeled Utility.
• Subsequent nodes are typically labeled in standard quality terms such as modifiability, availability, security. The tree should represent the hierarchical nature of the qualities and provide a basis for prioritization.
• Each level in the tree is further refinement of the qualities. Ultimately the leaves of the tree become scenarios.
• For each leaf in the utility tree, write a scenario. The scenario is in the form of context, stimulus, and response. For example, "Under normal operation, perform a database transaction in fewer than 100 milliseconds."
• Open the assessment matrix template. Enter each scenario as a row in the assessment matrix.

 

ATAM (by SEI) - (another architecture evaluation methodology) talks about a similar process with the addition of prioritization:

 

• Select the general, important quality attributes to be the high-level node
  • E.g. performance, modifiability, security and availability.
• Refine them to more specific categories
  • All leaves of the utility tree are “scenarios”.
• Prioritize scenarios
• Present the quality attribute goals in detail

 

This post is going to cover writing the scenarios, their prioritization and what's missing from both these methods (since they are evaluation methods) - ways to help us identify which quality attributes to use in the first place.

 

First, before we delve too much into details, here is an example for what the end result might look like (taken from http://www.akqit.ch/w3/pdf/bosch_atam.pdf - I am trying to see what I can publicize from project's I've been involved with - but I guess this will have to be later, i.e.  in a separate post)

 

It is hard to explain exactly how you would go about eliciting the quality attributes and their refinements (I think that the best way to do that would be through a workshop - but it's hard to do that over a blog :) - it does, however, include the same techniques you would use to elevate any other requirement -either by building on your past experience from similar systems but  mostly by working closely with your stakeholders:

• Interviews - meeting with individuals stakeholders to discuss their view of the system
• Brainstorming - meetings with multiple stakeholders trying to come with attributes and scenarios
• Reading written requirements (if available) - e.g. RFPs, use cases , project risks document etc.

 

To help with the elicitation, I'll try to give you some list for the first two levels (Attributes and refinements) that can serve as a repository or checklist when you are working with the stakeholders.

 

I already provided  a relatively long list of quality attributes to draw from to create level 1 of the tree (though the list is not an exhaustive one) in the previous post .

 

For the next level 2 of the tree (refinement) consider the following lists for the common quality attributes (most from A Method?< Analysis Tradeoff Architecture the to Scenarios General of Applicability)

 

• Performance
  • latency
  • deadline
  • throughput
  • jitter
  • miss rate
  • data loss
• Availability -
  • time period when the system must be available
  • availability time
  • time period in which the system can be in degraded mode
  • repair time
  • boot time
• Modifiability / Replacability / Adaptability /Interoperability
  • difficulty in terms of time
  • cost/effort in terms of number of components affected
  • effort
  • money
•  Efficiency
  • Resource X (CPU/Memory/…) usage on average per unit of time
  • Max usage of Resource
  • Availability of resource over time
• Usability / Learnability  / Understandability / Operability
  • task time
  • number of errors
  • number of problems solved
  • user satisfaction
  • gain of user knowledge
  • ratio of successful support requests to total requests
  • amount of time/data lost

 

The scenarios are the most important part of the utility tree, the main reason is that the scenarios help us understand the quality attributes needed, and more importantly, by tying the attributes to real instances in the system the scenarios help make these goals both concrete and measurable.

 

A couple of things that are important to note about scenarios

• First and foremost - Scenarios should be as specific as possible.
• Scenarios should cover a range of
  • Anticipated uses of the system (“use case” scenarios) - what happens under normal use
  • Anticipated changes to (growth scenarios) - where you expect the system to go and develop
  • Unanticipated stresses to the system ("Soap opera scenarios" or exploratory scenarios , pushing the envelop etc.) 

 

 

Scenarios are basically statements that have a context a stimulus and a response and describe a situation in the systems where the quality attribute manifests itself.

Context - under what circumstances

Stimulus - trigger in Use case lingo

Response - what the system does.

 

 let's look at few examples to try to clarify this:

 

• Under normal operation, perform a database transaction in under 100 milliseconds (Use case)
• Remote user requests a database report via the Web during peak period and receives it within 5 seconds (Use case).
• Add a new data server to reduce latency in scenario 1 to 2.5 seconds within 1 person-week. (Growth)
• An intrusion is detected, and the system cannot lock the doors. The system activates the electromagnetic fence so that the intruder cannot escape (Use Case)
• For a new release, integrate a new component implementation in three weeks. (Growth)
• Half of the servers go down during normal operation without affecting overall system availability (Soap opera)
• Under normal operations, queuing orders to a site which is down, system suspends within 10 minutes of first failed request and all resources are available while requests are suspended. Distribution to others is not impacted. (Use case)
• By adding hardware alone, increase the number of orders processed hourly by a factor of ten while keeping the worst-case response time below 2 seconds (Soap opera)

 

If we take one of these (e.g. "An intrusion is detected, and the system cannot lock the doors. The system activates the electromagnetic fence so that the intruder cannot escape ")

The stimulus - An intrusion is detected

Context - the system cannot lock the doors.

Response - the system activates…

Or another one (Half of the servers go down during normal operation without affecting overall system availability)

Stimulus - Half the servers go down

Context during normal operation

Response - without affecting overall ...

 

 

The last step is prioritizing the scenarios, it is common to use 2 criteria  (though you can use more)

• Importance  to system success
  •  High, Medium, Low
• Risk/difficulty in achieving
  • High, Medium, Low

 

The interesting scenarios (where you would focus) are the ones with high priority (H,H);(H,M) and (M,H) - these will be used as input for the modeling step of SAF

 

I'll try to provide  samples based on my experience in one of the future posts.

 

I just found this on the new CodeGallery on GotDotNet (via Brad Wilson's blog). The paper provides for an interesting reading and discusses some of the issues I was going to cover regarding Architecture Evaluation (I will probably blog about them anyway, but that's just because I like to write :) ).

Also note that the document is labeled "Microsoft Confidential." on the first page - I am guessing that the document status changed and they forgot to remove this notice - but it might also mean the document will not be there for long...

Harry Pierson talks about architecture and software architects. He quotes and adopts Alan Cooper 's* view of an architect : "...The architect is responsible for determining who the user is, what he or she is trying to accomplish, and what behavior the software must exhibit to satisfy these human goals..."

 

I agree that customer focus is a very important aspect of the architect's work - To quote from WWISA's definition of the architect's role:

"Client advocacy is the cornerstone of the architect’s role ... An architect ceases to be an advocate if tethered to a prescribed set of technologies, tools, or methodologies, only narrowing the solutions available to the client…". Marc Sewell in "The Software Architect's Profession: An Introduction" expands this view comparing software architects to construction architects, i.e. the architect role is to represent the client vs. the construction (development) organization.

 

Nevertheless - I believe that the only way for the architect to accomplish these feats is to do design - no - not "low-level" design of local issues, but yes the design of the overall system, be that partition into business-aligned services advocated by SOA or identifying strategies needed to cope with fault-tolerance (an availability issue).

Yes "Coming up with the lists of functional requirements and non-functional constraints is the architecture problem" (as Harry wrote in a previous post) is an important part of the architect's job - but it is far from being the only part, this is just the preface to the actual work of laying out a solution that can support these requirements and especially the quality attributes (non-functional reqs.). (To use Marc Sewel's analogy) Just as building architects design blueprints for buildings, Naval architects design blueprints for ship building  - the software architect has to draw the blueprints which the development teams will use.

Lastly, Jack Greenfield et. al. pointed out in "software factories" the model of one level of abstraction are the specifications for the next level of abstraction thus the requirements  are the specification of what the system does (without specifying what the solution is) and architecture is the specification of what the solution is (without specifying how it should be implemented)

 

It is also important to note that the customer is not the only stakeholder whose concerns has to be considered (and balanced) by the architect (see a previous post I made on stakeholders) - This aspect  is even more intensified in "real-life" since the architect is more often than not a part (or hired by) the developing organization and not the client (i.e. you don't usually see a client that hires an architect to represent it vs. the development contractor).

 

Having said that, I would also like to comment on the quote "architecture is design but not all design is architecture" - what I meant to imply by that is not that architecture is some sort of "good design", but rather, it means that architecture is a certain level of design that takes into account global decisions which affects the whole system (and identifies key local decisions) and that there are other levels of design which are not architecture (what Harry calls engineering)

 

---

* Alan Cooper is considered as the father of VB. He is  also the author of very enlightening books on interaction design: "About Face 2.0" and "The inmates are running the asylum" 

 

I've just read  Klaus Aschenbrenner's  Service Broker article on DevX. While the article provides a nice overview of the mechanics of service broker, I think there's a need to elaborate more on the essene of Service Broker (esp. vs. MSMQ) - so here's my 2 cents on the subject.

While MSMQ is a general purpose messaging engine, Service Broker is focused on (some would say limited to)  transactional messaging -And in this area it does a better job than most if not all the other messaging solutions out there. Here are two examples:

• Message order - MSMQ transactional messages only guarantees ,message order inside the transaction - it does not guarantee order between transactions also messages from two transactions can be interleaved (see MSDN for illustrations). Service broker solves both these problems.
• Multiple readers - If you have a large message load and want to set up multiple readers. One problem that might happen (which Service broker solves) is of  two related messages being handles by different readers at the same time - this can occur if you have a long running processes between services or if you utilize service broker for message exchange inside a service. Also in the realm of multiple readers - Service broker will do some form of load balancing the readers for you - if you are willing to let the reader be stored procedures.

Another important trait of Service Broker is that it seems it was built with service orientation in mind (and not just because of the name..). For one you have to define messages and contracts (which party can send which messages) in order to establish a conversation (even though you can trivialize it to send any message you want in any direction). Furthermore, you send messages on a conversation and not to a queue. This means that the sender doesn't really have to know who would get the message (well some layer has to know in order to set up the plumbing - but the business logic that decides to send a message is free from all this).

The other use for Service Broker is to introduce asynchronous operations into the database. It lets you create short transactions updating a single table or view and have the effects of the change ripple as a continuing transaction or transactions. for example you can update a view (that has a lot of underlying tables) and in its "instead of " triggers put the updated information in a queue. You can then read from the queue in another transaction and update the underlying tables. since everything is transactional (the send from the view's trigger, as well as the read by the updater stored procedure) you can be sure that the updates will occur (by the way, this view scenario is something that can be used for implementing insert-only databases, but that's a matter for another blog post :) ).

Arvindra Shemi posted a diagram with the main concepts of service brokers. Note that he also mentions there Monologs, which, as Pat Helland once told me (before he left Microsoft), was originally intended to be part of the product, but was dropped (postponed ?) somewhere along the way. It should be noted that you can implement the behavior of monologs using dialogs, however you need to do a some work to make that work. Another concept missing from that diagram is conversation group, which is a way to treat several conversations as if they were a single conversation.

One technical remark on the DevX article - The article shows the TSQL - it isn't too difficult to wrap these with C# code. I didn't check the latest beta of SQL  but the samples of previous versions already had such an implementation (prepared by the service broker team).

[minor updates]

If you are following this or Udi Dahan's blog, you've probably already read that we are currently working on a project together.  The project we're working on has a lot (and I mean a lot) of flexibility requirements (adaptability, replacability, interoperability etc.) which pretty much pushed us to work service-orientations.

Naturally (?), we invested  a lot in partitioning our services around logical boundaries, which will allow autonomy and loose coupling. we also tried to make sure this partitioning is aligned with the business goals of both the current project and the future road-map (the same architecture should grow to serve a product line).

The more uncommon (and probably more interesting) design decision we made is  regarding the internal structure of the services. We have an additional level of services (i.e. Micro services) that partition the service internally. The big difference however is that the alignment of the services boundaries is technical/design driven and not business driven. For example we can have a micro service to deal with persistence and another to do number crunching (for important algorithms) etc.

What are the benefits of this architectural decision?

• Unified semantics both inside and outside the service
• Unified solutions for availability
• Fine grained control on scale-out strategies and bottleneck handling (scale-out in the service level or the micro-service level is relatively easy)
• Built in solution for increasing concurrency (remember, partitioning along technical decisions)
• Contract based negotiations allow for increasing flexibility in identified areas (e.g. set important algorithms as a micro service and it will be easier to upgrade them later to a more advanced version) 

The main cons are additional complexity and increased latency inside the service. also there's the risk of partitioning the micro services to too fine-grained services and increase the latency further (watch out for chatty contracts)

Other things you to pay attention to - You probably don't want to have fine-grained micro services (just as much as you wouldn't want them in the services level). Also, it is probably not a very good idea to have too many levels of micro services (it seems that a single  level below the service is just right , but I don't have enough data to validate that yet).

By the way, it was Udi that dubbed the term "Micro Service" - so if you decide to use it, sent the royalties to him :)

 

 

Udi Dahan and me are currently architecting an interesting little-big project (little as it has a very limited scope for a first deliverable on a very tight schedule and big as the stakeholders require the architecture being drafted now to fit/scale for some very aspiring plans for the future - past this deliverable). It is very interesting working together seeing how different people have different styles, approaches and even directions looking at the same set of requirements and problems (and that's when we basically do have a common approach).

Yesterday Udi posted on one such difference - in the area of documenting our architectural designs. Udi thinks it is more simple and understandable to use sequence diagrams while I opted for using UML 2.0 communication diagram 

The main reason for me to choose this diagram )over sequences) was to show off my UML skills (?!) - well, not really - rather the reason was that we were trying to describe to overall context of the UI. Using a Communications diagram helps alleviate the responsibilities of the classes in the (UI) context by showing both  the relations of the main classes (actually class instances) and the main flows of messages between them. Using a single diagram allows for understanding (once you get yourself familiar with the syntax) how all the object collaborate to achieve the mutual goal of making the UI tick (I guess a better name for this diagram is collaboration diagram -but that means something else in UML 1.x).

Also as Scott Ambler mentions in The Object Primer 3rd Edition: Agile Model Driven Development with UML 2 collaboration diagrams are useful when use cases aren't the primary requirements artifact (which is the case in the particular project we're working on).

As for "visual shock" - well  I think the class diagram is less intuitive and straightforward compared with the communication diagram. what exactly is the difference between aggregation and composition and how is that different from association. Also when would you use association over dependency etc. Class diagrams seems to be "simple" because we are used them. Communication diagrams

Lastly talking about the "white-board" approach  - compared with sequence diagrams,  using communication diagrams it is much simpler to rearrange message and/or to insert a new messages in the middle of a sequence (just draw a new message and renumber the other messages)

 

I read today a post by  David Ing called "An Overly Long Guide to Being a Software Architect" . David talks about different aspects of a software architect - among those things he mention two important soft skills for architects namely Organizational Politics and communications. Two additional soft skills (or competencies) that an architect needs are  strategic thinking and Leadership (There may be some others but I think these 4 are the main ones).

Dana Bredemeyer measures competencies from 3 viewpoints - what you know, what you do and what you are.
For example looking at Organizational Politics

• What you know - Who the key stakeholders are, what they want from the business and personal perspectives
• What you do - Communicate, network, build relations, sell the vision/architecture
• What you are - comfortable with compromise and conflict, able to look at situations from several viewpoints, articulate, patient, resilient, sensitive to power flow within the organization.

Or if we look at leadership

• What you know - yourself
• What you do - mentor others, motivate others, build teams and set their vision, influence decision makers
• What you are - committed, passionate, credible.

Brdemeyer also supplies competency elaborations (levels for each competency and and guidelines on how to advance yourself between the levels) for Strategic alignment, Organizational Politics  and Leadership .

Another interesting source on architecture competencies is a book called "Software Architecture - Organizational Principles and Patterns"   by David M. Dikel, David Kane and James R. Wilson.
The authors detail a reference model of the organizational aspects of the software architecture process (vs. for example the SPAMMED Architecture Framework (SAF) which details the more technical aspects of the process).
The model takes about 5 principles :

• Vision - Deals with the mapping of future value to the constraints on the architecture and how well understood, flexible etc. are the architecture's structure goals
• Rhythm - the predictability and recurrence in the exchange of deliverables between the architecture group and the architecture consumers.
• Anticipation - The extent to which the architects predict, validate and adapt the architecture to changing requirements (as well as technologies, competition etc.)
• Partnering - the extent to which the architects interact with the architecture stakeholders to allow maximizing the value delivers or received by the different parties
• Simplification - achieving the "zen" (clarification and minimization) so to speak of both the architecture and the organizational environment where it lives.

An example for the patterns and antipatterns that relates to Stakeholders (first step of SAF)

• Criterion : The architect continually seeks to understand who the most critical stakeholders are, how they contribute value and what they want.
• AntiPattern - Phone Doesn't Ring
• Pattern - Know Thy Stakeholders
  
• Criterion: Clear compelling agreements exist between stakeholders
• AntiPattern: Lip-Synching
• Pattern: Reciprocity
  

The book naturally continues to describe what these patterns are :).

The important takeaway from this is that while knowing every nook and cranny of "the framework formerly known as Indigo" (WCF) will probably won't harm you -  technical competency alone will only take you so far as an Architect and  you can not afford to neglect growing and working on the aforementioned soft skills .

 

 

Udi Dahan talks about the merits of agile methods vs. (heavy) process methodologies (countering a post by Joel Semeniuk on CMMI).

I can't say I totally agree with either (what else :) ) - There are cases where agile methods are a better git and there are cases where you'd be lost without a plan. While agile methods optimize for change (which is indeed a grim reality we all live with) plan driven methods optimize for complexity. Barry Boehm and Richard Turner detail the various people related issues that can make you choose one over the other. The diagram below (taken from that article) sums it up nicely.

It is always important to strike a balance between the level of process employed and the project at hand. That's why even the more heavy processes (like RUP or MSF for CMMI) are tailorable. I guess a lot of organization don't bother to take the effort to tailor the processes to their need and rely on the "out of the box" experience which is not tuned to their needs. and thus suffer less than optimal results.

Another important issue is tool support - If you are going to employ a more plan-driven (heavy) process you really want it to be supported by your tools to help alleviate that "document oriented development" feeling Udi mentioned in his post. This is where this upcoming release of MSF shines (especially vs. the previous release of MSF).

I just upgraded the blog from to DasBlog 1.8 (RC), I also took the chance and changed the theme for the site.

I guess this is a good opportunity to thank Scott Henselman and Omar Shahin for all the time and effort the put into maintaining and upgrading this software

I recently saw a post by James Gosling (via David Strommer ) called the Eight Fallacies of Distributed Computing . These are eight assumptions on network almost anyone new to distributed computing assumes  which proves to be wrong in the long run (and thus cause big problems and headaches).

I thought I'd try a to complement this list by adding few realities on distributed systems and data

1.  Expect a certain level of entropy in the system -  Sites are never fully synchronized (unless you stop new data from pouring in)
2. You can only afford to cache immutable data
3. It very very hard to be able to scale indefinitely
4. Observing global state is only possible via control messages
5. It is hard to achieve distributed consensus (membership in a cluster, total order, commitment etc.)
6. Expect to debug by log-files

There are probably many others - but these are the first few that came to mind :)

 

Architecture Description Languages (ADLs) sound like a great idea when you look at the "spec sheet"

ADLs represent a formal way of representing architecture
ADLs are intended to be both human and machine readable
ADLs allows simulation and analysis of architectures – completeness, consistency, ambiguity, and performance
ADLs can support automatic generation of  the system (as it is represented by the architecture)

There are quite a few ADLs out there xADL , ACME/ADML,  SSEP, Rapide,  Wright  and the list goes on and on.

I am guessing most of you didn't hear about any of those (well ACME appears in all those Willy E Coyote vs. Road Runner cartoons but that doesn't count). I think that the culprit lies in the fact that most (if not all) these come from the academic world where the focus lies on the models (in terms of semantics, completeness, rigorness etc.) and not on the practical use and applicability to day-to-day issues. Another problem lies in the fact that main-stream tools used by the industry.

By the way, UML isn't considered an ADL for several reasons for example the weak integration between the different model that inhibits automatic analysis.

The reason I am bringing the issue of ADLs up is that looking at the new architecture designers in Visual Studio 2005  (the Application Designer , the logical data center designer ,the System Designer  and the Deployment Designer) actually form a set of DSL (Domain Specific Languages) that together can be treated as an integrated ADL. Furthermore Microsoft also provides an SDK   for the"System Definition Model" that is the underlying model behind all these DSLs which lets interested parties extend and build additional designers.

While this model is not complete in several aspects ( some designers like the logical data center designer are too limited or there aren't enough views to cover all the architectural description) it is a good starting point in bringing the ADL concept into more practical and usable form.

 

This is part II of P is for Principles (and Guidelines and Constraints...) - Iteration I

In the previous entry I've said that it is helpful to provide a list of constraints and principles as it helps in limiting the scope of the solution and directing it toward good (and/or required) practices (Architectural, technological or business aligned). However, I also claimed that a simplistic list is problematic - I'll try to demonstrate this through a couple of real-world examples:

 I recently reviewed the software architecture document (SAD) of a rather large software project, I saw that they it is mentioned that the project uses "service oriented architecture" - reading on, I saw the architecture builds on a distributed "shared memory" where every client and server has a full image of all the data entities, further more data entities are intertwined without any boundaries whatsoever. when I asked what's service oriented in that, I was told that the underlying distributed "shared memory" engine provides several services like dissemination, scheduling etc. The point here is that a catchy name can mean different things to different people.

On another project some of the stakeholders mentioned it is important that the solution would have an "Open Architecture" - sound good to me, but what the hell does that mean? based on open standards? promote extensibility (easy to add features)? promote replacability (easy to replace components)? all of the above? something else?

Furthermore if you don't understand the implications behind each of the principles you name -  it gets very tempting to create a "Buzzword Oriented Architecture"- we want SOA, AOP, Software Factories, Smart clients, GRID, fault tolerance and whatnot... (Note - It is recommended to proceed with caution if you see too many buzzwords in your guidelines/goals. You might be trying to accomplish too much and/or you have too much marketing influence).

So what makes a good (or at least better) description for a principle?I currently use the following template:

Name - something easy to remember (e.g. SOA, Layered Architecture etc.)
Description - What does it mean
Rationale / Benefits -  Why do we want to apply this principle
Implications - What does it mean to use it 
Alternatives - What else - What are the other options we considered and why we didn't use them.
Scope/Exceptions - when and where does it apply

Note: I used to use a simpler template (whiteout Implications and Scope) the current version is based on a template by Ilia Fortunov from Microsoft UK. I can explain that further but it will probably be easier to understand through examples - so here are a couple from projects I worked on:

Principle Name: Code Generation
Description:Generate specific implementations and allow users to configure generated code via designers.
Rationale/Benefits: Increase longevity of the domain model (help separate from technical implementation). Reduce bugs via use of design tools.
Implications: Need to "templatize" solutions and develop code generators (need to check commercial solutions)
Alternatives: 100% object orientation and generic implementations. rejected due to tight coupling to technology, performance implications and impact on code readability.
Scope/Exception: Domain entities and any "aspect" related implementation (logging, security etc.)

Note: this is for a solution that has to use .NET 1.1 - had it been for a solution that relies on .Net 2.0 Another option - using Generics  might have been a viable solution

Another example:

Principle Name: Distributed Database
Description: Each site shall have a separate independent copy of the DB and will have to synchronize its data with connecting sites (Note that I use the term synchronize and not replicate - as replication is a specific technical solution)
Rationale/Benefits: The inter-site communication medium is unreliable plus sites has to maintain autonomy.
Implications: The system has to cope with partial data. There's a need for conflict resolution policy. We need to consider idempotent messages for inter-site communications. Need a distributed primary keys management scheme.
Alternatives: Federated database - problematic since (some of the) data will not be available when sites are disconnected. Another solution weighted is a combination of centralized server and "off-line" capabilities upon disconnection - rejected as it would be more complicated (based on past experience) and have high-dependency on bandwidth (for on-line work)
Scope/Exceptions - Database layer and inter-site communication.

The next thing we have to deal with are constraints.As I've said in the previous post constrains originate from the different stakeholders and limit the scope of the solution. To document the constraints in a meaningful way I use the following template:

Name: something easy to remember
Definition: What does it mean
Implications: What does it mean for the architecture? what are the limitations it places.
Scope: where will we feel the impact
Origin: who placed this constraint and why

Again, lets look at a couple of examples:

Name: Use .NET 1.1
Definition (pretty obvious...)
Implications: use tools/products compatible with .NET 1.1; don't relay on .NET 2.0 capabilities (important for the mapping stage of the SAF)
Scope: All the system
Origin: using .NET is a company policy; we need .NET 1.1 since all the existing tools (e.g. ClearCase, XDE) only support 1.1

And another example:

Name: Deadline
Definition: We must have a working deliverable in 5 months.
Implications: Strive to reuse existing assets; try to migrate code from legacy version; try to model (an extensible) simple architecture (don't try to solve everything now - but try to leave flexibility for future growth)
Scope: Mapping stage; Architecture documentation; quality attributes of the solution
Origin: Customer  (time-to-market)

One thing important to remember is that we are not interested in all the project's constraints (we are- only not in this context). The meaningful constraints are those that have implications on the architecture.

To summarize. Principles and Constraints help you limit the scope of the intended solution architecture. Principles are based (mostly) on past experience, constraints must be followed (and are originated from the different stakeholders). Using only a catchy phrase to describe either of these (Principle or Constraint) can prove to be problematic (creates confusion, doesn't really add anything etc.) and it is better to think about the implications of applying the principles and constraints. Lastly, principles (at the first stage) should be taken with a grain of salt - as they may not be suitable for the current requirements - you should be ready to reiterate and update them once you know more about the requirements (The end result would be to have a list of guidelines which are actually used for the solution's architecture).

Both templates (principles and constraints) are available for download on the template section of the SAF page.

The next PDC is comming to Los Angeles Sept. 13th  a (nearly complete ) list of the sessions and abstracts can be found here:

http://commnet.microsoftpdc.com/content/sessions.aspx

 

What if there was a new platform for .NET that has the following characteristics

• lets you run .NET 2.0
• supports web-services (something like IIS)
• integrated message-queue (a-la MSMQ)
• high-availability (better than clustering performance)
• strong management tools

well there is one - it is made by Microsoft and goes by the name SQL Server 2005.

SQL Server 2005 lets you run .NET 2.0 code in process (an interesting point here is that DB2 already lets you run .NET code and Oracle will also add this capability with 10g Release 2  but both do it out-of-process).

It also  lets you expose stored procedures as web-services

It has a performant message queuing mechanism called service-broker. Service broker guarantees message ordering as well as solves the problem of multiple readers (what happens if one reader takes the order header and another takes the order line) and as I said it has good performance. by the way Roger Walter who is the group program manager for service broker is going to publish a book on the subject. Having worked with him for a short while (He was our "Redmond connection" on one of the projects I consulted) I can say without hesitation that he is the authority on service broker and a smart guy to boot :)

SQL Server 2005 also has (almost) instant fail-over capability - using Database mirroring you can have your (application?) server fail over in less than 5 seconds (compare that with clustering fail over times of 60 seconds or more). Note that each database can only fail over to one other database, but a server can be both the backup database of one server and the active server (vs. another server).

And you get all the management capabilities of SQL server (oh yeah, you also have a great DB there :) )

Taking all this into consideration there are (a limited set of) situations where you can actually use SQL Server as a hosting platform (instead of COM+ or IIS) for your application. True, this is not something I will generally recommend but I can think of one or two projects where scale-out was less of an issue where it could have been used (had it existed).

 

 

Whenever you start a new project, even if you think you start for scratch - you don't really start from scratch. You always bring your past experience into the fray. Developing an architecture for a project is not different. the "Principles"  stage of the SPAMMED Architecture Framework (SAF) is about bringing in your "lessons learned" and "best practices" as  baseline rules or starting point for the architecture you are trying to build. laying down a good set of principles will help you limit the scope of the solution and focus on proven tactics.

It is important to note that principles you set should be treated as recommendations - the main reason for that is that they are based on past requirements and experience and not on current requirements.

An example for a principle may be "use layered architecture"  Layered architecture is the practice where you define several level/areas of processing and limit the communication paths between them (common patterns are: layer can talk to the layer just below; layer talks to any layer below it etc. - the important characteristic is that communication is restricted). Layered Architecture brings a lot of benefits especially in promoting flexibility in deployment, modifying implementation etc. some of you reading this may think this is a trivial principle - however, sometimes it is good to put even trivial assumptions on the table, furthermore not everybody agrees that it is always needed see this  for example. Lastly using a layered architecture has its risks for example the different layers can't scale to the same extent you may find yourself with scalability issues down the road. (I'll give more detailed examples for principles in the next post on this subject)

A similar notion to principles, in the sense of limiting the solution scope is constraints. However, unlike principles, these are not recommendations but rather these are limitations you have to follow. Constraints are set by stakeholders (their origin may be company standards, customer standards).There are several types of constraints:

• Technical - limit platform, reuse existing system/solution/component, follow a particular standard  e.g. use Windows, .Net (alternatively use Linux, Java), use Web-services
• Organizational - follow a particular process, availability of the customer e.g. use RUP , MSF, company standard # 15 ..
• Business - time/money (deadline, budget). Another interesting example for this is "application freeze" - when an organization forbids change for a period - something some organizations did just before Y2K. [thanks to Andrew Johnston from www.agilearchitect.org for this one]

So how does it works - well, you bring in your architecture team along with some of the technical stakeholders - go through one of those "brainstorming" meetings and come up with a list - e.g.:

• Build on Open standards
• Reuse
• SOA
• Layered Architecture
• support scale-out
• .
• .
• .

Wait, what's wrong with this picture - well, for one it is too simplistic view (it doesn't say much) and even more importantly it not accurate (i.e. it can mean different things to different people) - on the next post on this subject I will show a better way to document and understand principles along with a few examples from real projects.

 

[thanks to Eliaz Tobias from MS Israel for the link]

The beta version of MSF 4.0 formal is available for download. This is an interactive process which is compliant with CMMI level 3.0 (see http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr029.pdf chapter 7 - maturity levels)  and supported by the tools (i.e. Visual Studio Team System 2005). This can be great news to organizations like the one I work for these days who are certified for CMMI 3.0 and can (hopefully) stay compliant with less bureaucracy.

Looking at this process from my perspective (i.e. as an Architect) it also looks interesting. It defines several roles for architects (e.g. in domain technical level as Subject matter experts and in the solution level as an Architect).  The process suggested is tailored/aligned with VS2005 capabilities (and thus somewhat limited) however many of the steps are both viable and important. For example it has parts that do with quality of service requirements (what I call Quality Attributes in SPAMMED).  My particular favorite step is "Assess Alternatives (LAAAM)"  which I helped introduce ( :) ) in my previous job as an Architect for Microsoft Consulting Services

 

It is not a secret that user involvement increase the success rates of software projects. We can basically look at the architecture as a mini-project inside a project. The users in that case (as my entry on Stakeholders shows (or tries to)) are the stakeholder.

Richard Demers talks about Architecture Control Board (on his smalltalk blog - oh my :) ) as a way to increase the involvement and of stakeholders for large software projects. The idea is to engage as many stakeholders groups as possible on a periodical basis and to set up a review and change board that approves changes in requirement and also review and approve  the architecture (I'll talk more about something similar when I'll get to E - Evaluation in SPAMMED).

Richard lists 13 points in regard to the Architecture Control Board - the most important ones (in my opinion) are

• the ACB set the requirements scope for the architecture (what requirement/quality attributes should be accounted for)
• the ACB  review and criticize the architecture - they don't, however, design it or vote for its correctness.
• the architecture team has final say on architectural decisions (though an escalation path to upper management should exist)
• the documentation approved by the ACB is the ultimate deliverable of the architecture team (i.e. the "Software Architecture Document")
• go read the rest :)

While establishing a "formal" ACB in smaller-scale projects is probably an overkill you may still want to follow some of these tactics on a less formal basis to increase your stakeholders involvement and more importantly cooperation.

 

 

There's a lot of confusion on what Service Oriented Architecture (SOA) is and isn't. Martin Fowler sums it up nicely in his Bliki (Blog+Wiki). Clemens Vasters blogged something similar back in May. Whether you agree or not - one thing is sure - there's so much hype around SOA these days that it is hard to understand the realities.

 

The Composite UI Application Block (CAB) is an interesting architectural solution for smart client development. It is essentially a plug-in architecture that brings the WebParts concept to the desktop.

The CAB has the following elements:

• WorkItems. These are the classes which represent use cases in your application and contain the business logic for those use cases.
• SmartParts. There are the building blocks of a Windows-based application - similar in concept to WebParts.
• Workspaces. These are helper classes that can display SmartParts with a uniform style.
• UIElements. These are elements such as toolbars and menu bars that are shared by SmartParts within the application.
• Support services. These include:
  • Event broker service to manage the publishing and subscribing of events between SmartParts.
  • State management service to hold shared state for areas within the application. This provides an option to encrypt the state before it is stored.

The CTP is now available on GotDotNet (needs .NET 2 beta)

Don't worry  I didn't switch to blogging on Greek mythology (well, not yet anyway) this is still a  technology blog.

Ajax is an old/new web development paradigm. It is basically a combination of several existing client-side technologies to achieve responsive and "rich" web applications (a good example for that is Google maps - just pan the map and you'd see what I mean). specifically Ajax builds on XHTML, cascading style sheets (CSS),Javascript, DHTML and XMLHttpRequest (for asynchronous communication with the server). By the way the reason it was dubbed AJAX is that it is an acronym for Asynchronous Javascript and XML.

Microsoft has downplayed Ajax importance for quite sometime - claiming that basically the technologies has been available for years in Internet Explorer. Nevertheless Microsoft has recently decided to join the fray and provide structured tools to help realize Ajax development on the .NET platform  - The code name for this initiative is Atlas. Scott Guthrie from Microsoft published some of the details in his blog

Here are a few highlights of what Atlas will include:

• Client Side Framework - an extension for Javascript with UI controls, network stack, eventing etc.
• Server Controls - server side-controls that interact with the client side controls (based on async callbacks)
• Web-service integration
• Few Server-side and Client side Services - e.g. Profiles, Personalization, Authentication and Roles on the server and local browser cache on the client

It seems Microsoft will release a technology preview of Atlas at the PDC later this year.

Meanwhile, if you don't want to wait for Atlas - you can download Ajax.net  - Ajax.Net will generate client-side scripts that interact with server-side methods decorated with [Ajax.AjaxMethod] attribute - it isn't everything, but it is a very nice start (and it is available today)

It should come as no surprise that the first pillar of the SPAMMED Architecture Framework (SAF) are the stakeholders - after all at least some of these are the people/organizations that are the cornerstone of the software project itself.

What exactly is a stakeholder - EIA 632 , a standard for System Engineering, defines a stakeholder as "An enterprise, organization, or individual having an interest or a stake in the outcome of the engineering of a system".

Sounds good enough to me :) - but before we delve into more details on how to identify these stakeholder, what do we do with that information, we first have to understand why it is important to us as architects. The short answer was already mentioned stakeholders (most obvious examples are Customer, Project Manager) are what makes the project tick. That, however is just the beginning of it. One of the primary responsibilities of the software architect (much like the project manager by the way) is to balance the stakeholders interest to ensure the success of the project. In the SAF sense the stakeholder are important for several reasons:

•  The solution is developed to serve their needs or goals (at least some of them i.e. customer, end users, management)
•  They serve as the source  for constraints (and sometime principles)
•  Their concerns can help elevate needed quality attributes
•  Stakeholders can (and should) help evaluate the architecture
•  The documentation of the software architecture is targets at stakeholders

Several stakeholders are pretty common to any project . The following list shows them along with samples for some of the concerns (or vested interest) they have regarding the project:

• Customer - Functionality, price 
• End-User - Ease of use, performance
• Project Manager - On time delivery, development costs
• Management - Price, reuse
• Developers - structure and dependency between components, interesting technology
• Maintainers - ease of debugging, modifiability
• Testers - Testability, Traceability
• Security Analysts - security
• Project New comers - structure and dependency between components, traceability
• Customer’s IT group - ease of installation, stability

This list is a nice starting point but it is just that - a starting point. There are still a few things we may want to do

• Identify additional stakeholder - sometimes there are less common or obvious stakeholders (e.g. System engineers, shareholders, safety analysts, the general public etc.)
• Map stakeholders relevance - Jaap Schekkerman from  suggest prioritizing stakeholders by power vs. interest. I believe the privatization should also include the (stakeholders) concern importance:
  

• Lastly you may want to document your stakeholders so as not to forget what they are interested in. This documentation can include the constraints they place their concerns (translated into quality attributes) and a list of viewpoint that are needed to satisfy them (i.e. explain the architecture to them). A template for documenting stakeholders is available from the SAF page.
  

I've added a new area on the site for the SPAMMED Architecture Framework (SAF)

www.rgoarchitects.com/saf

There's nothing much there (yet...) except links to the blog entries on the SPAMMED process, however, I am going to add there presentations, whitepapers, a workshop etc. in the future (some of these are already under development)

Everybody talks about "Software architecture" these days (your humble servant included...) - but what the hell is it?
I mentioned in my first  SPAMMED Architecture Framework post. that Software Architecture "deals with the major components or structures, their relationships and interactions. It encompasses the major (read hard to change) decisions and their rationale and every system has an architecture (even if it is a default one)" - OK sounds nice, but is it enough?

Probably not (If I thought it was enough - I wouldn't have wrote this post...). There are literally dozens! of definitions for what (solutions) Software architecture is ( you can see many of them here). I am not going to quote all of them, instead, Lets spend some time looking at some of the more prevalent characteristics found in most definitions

• Architecture is Early - It represents (well at least, should represent) the set of earliest  design decisions which are both hardest to change and most critical to get right.
• Every system has an architecture - even if it is just a default one (i.e. it can be described using reverse engineering) it still there.
• Architecture is about breaking a system into components and setting boundaries. It doesn't describe all the components - it usually deals with the major components of the solution. Also it doesn't describe the complete characteristics of components - it mainly deals with their interfaces or other aspects that has to do with their interactions.

which brings us to the next point:

• Architecture is about the relationships and interactions of components. Again we are interested in the  behaviors of the components as it can be discerned from other components interacting with it.
• Architecture explains the rationale behind the choices (vs. the choices not made). It is important to understand the reasoning as well as the implications of the decisions made in the architecture since their impact on the project is large. Also it can be beneficial to understand what alternatives where weighted and abandoned (for future reference, when/if things needs to be reconsidered, and for anyone new to the project that needs to understand the situation).
• There isn't a single structure that is the architecture - there's a need to look at the architecture from different directions or viewpoints to fully understand it.

There's a very interesting standard called IEEE 1471-2000  "Recommended Practice for Architectural Description of Software Intensive Systems"  which defines the relations between the system stakeholders and the different viewpoints of the architecture. It serves as a good reference for understanding how to document an architecture - it also means that identifying the needs of the different stakeholders will tell us a lot what views (details of the architecture from a specific viewpoint) we need to have.

• Architecture is the first design artifact where a system’s quality attributes are addressed

Stakeholders are also the main source for these "quality" requirements and It is the architect's responsibility to balance the quality attributes of the system. This and the previous point are the main reason that the SPAMMED process first step has to do with identifying stakeholders  (And I'll elaborate more on this on the next SPAMMED related post - "Stakeholders Everywhere")

Apart for the quality attributes side of the last point - it also basically state that:

• Architecture is design (but not all design is architecture) 

Which raises another interesting question - what's the difference between architecture and design. But this will have to wait for another post as well

Microsoft has released  the Connected Systems Toolkit. A very nice intorduction to the SOA concept (as Microsoft sees it). The toolkit (DVD) includes:

• a connected system's course (with 4 labs)
• an extensive sample application (with code and documentation)
• few whitepapers
• few presentations (ppt & video)
• some marketing hype (well, nothing is perfect :) )

Assuming the previous post made you curious  - here are a few links to get you started:

• An article that talks about extending XP's system methaphor  
• Scott Ambler's thoughts on the matter
• A site dedicated to Agility and architects

 

The architect doesn't talk, he acts.
When this is done,
the team says, "Amazing:
we did it, all by ourselves!" (17) (The Tao of software architect - Philippe Kruchten)

On the surface - When it comes to agile development the role of the software architect is a little more blurred.

The most obvious aspect where architects is for the technical architecture. An experienced technical architect, can greatly enhance any project by steering the designs into the "best" directions (under the chosen platform constraints). The technical architect can also promote reuse etc.

Additionally while the requirements change a lot and not fully defined, the quality attributes of a system are more stable - if you need performance, then you need performance!. More so,  there are some qualities that are inherent to agile process - for example you want to put an emphasis on flexibility and maintainability - if your developed solution does not have these it is going to do refactorings. Thus, depending on the size of the project you may want to use an architect to help set the ground rules in the first couple of iterations.

Another area where architect involvement can be very beneficial is when you try to scale an agile project a good choice is to try to break the project to smaller loosely coupled projects (see this paper  for example)  - well, this is just  what we (architects) live for...

By the way, few agile processes, define the architect role up-front, one such process is MSF 4 Agile. Note that MSF (Microsoft Solutions Framework) version 4 comes in two "flavors" one that is aligned with CMMI  and the other a (much) more light version, the afford mentioned MSF 4.

 

How does the SPAMMED process fare with an Agile project - (surprisingly enough) I would say pretty well

First of the architect should be hands-on i.e.  part of the development team (most likely the technical lead)

• Stakeholders, you would probably want stakeholders on-site for any architecture related  meeting (just as much as you would want the customer on site for other activities)
• Principles would include things like TDD, Simple Implementations, Refactoring
• Quality attributes would hold Flexibility and maintainability and a couple or so of the important project qualities (performance, availability..)
• For the modeling you would chose very few views and would try to focus on ones that have manifestation in deliverables
  A good example for this is the Application designer  in Whidbey (see screen shot below) - the result of which is the projects structure for the solution

• Mapping doesn't really change
• Evaluation would be focused on proof of concepts and/or skeletal architecture.
• Deployment - well,  being part of the team... you would notice if your decisions were wrong or circumvented

To sum up - yes Agile projects can and many times should use an architect to help it stay on track - hey, even XP has an architectural spike...

 

The presentation for the paper mentioned in the former post can be downloaded from here

First the disclaimer :)

I wrote the paper below about 2 years ago, summarizing my experience with requirements engineering using use cases. The problem was that it got to be too long to be a magazine article and too short for a book plus it needs a ton of editing.

Nevertheless, Now that I started blogging and considering that I think it still has some very useful information to anyone trying to make use cases work in a mid sized or large project - here it is for your viewing pleasure:

Methodology for building Use Cases for large systems.pdf (206.64 KB)

[fixed link]

I am writing an entry on the architect's role in an agile project and I noticed I am using the term "Technical Architect". My first intention was to go on and explain that within that  post, but I then thought it would be better to give a specific entry that paints the complete picture (well, at least, as I see it :) )

There are three basic classes of software architectures:

• Infrastructure Architecture - Has to do with presenting an workable infrastructure solution (types, deployment and configuration of servers, LAN, etc.) it basically deals with the overall layout of the  "out of the box" solutions that help solve the problem. An example for this is MS's Windows Server System Reference Architecture 
• Business Architecture - Concerned with the business model as it relates to an automated solution. It has to do with the structural part of requirements analysis, and it is usually domain specific (sometimes the job of a business analyst). Domain architecture is ideally technology neutral, although more often then not it is cluttered by technical constrains.
• Technical Architecture - Specific to technology and the use of this technology to structure the technical views (esp. Technology Mapping) of an architecture. Technical architects usually have a very good understanding of a technology (e.g. .NET, J2EE etc.) and how to best solve problems using that technology.

Additionally there are two compound classes

• Solutions Architecture - Specific to a particular business area (or project) but still reliant on being a technical focal point for communications between the domain architect, business interests and development.
• Product Line Architecture - The architecture that has to do with families of related solutions. It is basically the same as solutions architecture only with an extended scope. A product line architect has to promote reuse and identify commonalities between several solutions while on the same time providing each of the solutions with enough specifics as to make it a good viable solution in its own right.

Lastly there's the more comprehensive architecture class which is Enterprise Architecture (EA). EA deals with the governing logic and strategy of the a firm's core business processes and IT capabilities. It is a set of recipes (policies and principles) along with technical constrains (that are set for the different solutions within the enterprise). The EA is concerned with cross project/solution architectures and tries to lay the rules needed to achieve the business standardization and integration requirements of the firm’s operating model.

One of the most interesting things that MS is building these days is the set of tools around their software factories initiative.

Software Factories, is actually a methodology that builds on the concepts of Domain Specific Languages (DSLs) along with frameworks, patterns and guidance.

The idea is to bring the notion of domain modeling and DSLs from horizontal markets (e.g. the form designer in any modern IDE) to the everyday use of vertical markets (i.e. your next application) and help realize the promise of product lines. Many of the ideas in software factories are not new (see for example this article from 1993) - however, today,as the Jack Greenfield et al  state in their book, a number of needed technologies has (finally) matured enough to make it feasible. Furthermore  What's unique in this effort is that Microsoft is making the effort to back the ideas with tools  that will enable architects and developers to put them into use.

A software factory needs 3 elements a schema, a template and tool to support them (a development environment). The schema is basically the recipe containing the overall information on how things should work together. The template is the set of DSLs, samples, frameworks that are used to create the factory and lastly we need  the IDE that supports them (e.g. Visual Studio "Whidbey" - although, at least technically this can also be achieved in Eclipse etc.)

I am, personally, very interested in this initiative. We already use code generation all over the place (DAL, serializations, Data Entities etc) and employ a domain driven approach even today. I would love to have "user-friendly" tools that will let me as an architect achieve these goals more easily.

One final note - yes I am aware of MDA (Model Driven Architecture) but all the MDA tools I've encountered thus far are lacking in their usability. This, by the way, brings me to one caveat with software factories - it is not based on UML 2.0. I would rather have a DSL that is based on stereotypes, tagged values and OCL. This would allow for a specific view using the appropriate designer but also for backward compatibility (and a downgraded view) on other tools. I do agree though, that UML 2.0 has its own share of problems - but that's a matter for another discussion altogether

 

Just saw an entry @ Kevin Hammond's blog that when you download the Indigo/Avalon beta you have to read through to download the SDK rather then the redistributable - I have to admit that I've made the same mistake myself...

The previous post on SPAMMED introduced the different pillars of the process. This one focuses on their interactions

The state chart below (modeled using Sparx System's excellent Enterprise Architect) shows the possible transitions between the different process steps

Eliciting stakeholders is usually the first step to take - the reason is that stakeholders serve as the base for several of the following steps for example stakeholders concerns serve as a guideline for deciding which views to document during the modeling step.

The next step is documenting principles and goals - which are based on past experience. The real reason however that this step follows the Stakeholders elicitation is that the step also includes documenting constrains, and most of these are set by the different stakeholders (e.g. use .NET/Java because it is a company standard).

Quality Attributes builds on the former two steps, deciding which quality attributes are important and balancing them is based again on the stakeholders concerns and the principles you've set.

You would then follow this with some modeling and then technology mapping.

Up to this point the process has been pretty much "waterfall-like", however after you evaluate your former steps, you may find that you need to revise any of the preceding steps. This is why you want to get here as soon as possible. do not try to complete and "finalize" the architecture and only then perform the evaluation. Developing the architecture, is very much like the other parts of the development life-cycle, and can benefit greatly from a short feedback loop. Evaluating early helps prevent the architecture from being a bottleneck holding all the development process and even more importantly, building too much structure based on false assumptions/decisions.

Deploying the architecture, i.e. releasing it to the designers and developers is the next step (assuming the evaluation was OK). Once the architecture is "out-there", the realities of the chosen technology/product, changing requirements, budget/talent/ time constrains and most likely the mere iterative nature of modern software development will probably still make you evaluate and retrace some of your steps as a result. I believe this refactoring is a healthy procedure -  in the first few iterations (the actual number depends on methodology used, iterations length as well as project size) of your project. Nevertheless if the number of times the architecture has to be reevaluated and changed or alternatively the number of changes is large - it is probably a sign that your architecture is in need of a serious overhaul.

Microsoft has recently went public with their Architect certification program  - While (naturally) they expect knowledge of the Microsoft platform ("about a quarter of the emphasis" to quote the site), it also has a lot of of emphasis on architect competencies and general know-how. Way to go MS :)

On the Java side of the fence there are already architect certification programs e.g. Sun's or BEA's.

There are also two (that I know of) non-vendor specific certifications - one from the Open Group (they also brought us TOGAF - which is worth its own discussion) and the other from the Software Engineering Institute (SEI @ Carnegie Mellon university)

I 'd be happy to learn of any other architect's certification programs if you know them

Well, I haven't heard about that one yet - but there are few other groudps and organizations out there. Here are the ones I know about:

• Word Wide Institute of Software Architects - I am a member of this one :)

• Enterprise Architect Community

• Global Enterprise Architecture Organization

• International Association of Software Architects

• Enterprise Architect Interest Group

• Institute for Enterprise Architecture Developments

I almost forgot another commercial GRID implemrntation  is Gigaspaces. Gigaspaces is interestig since it is actually based on  Java standards  JINI  and JavaSpaces but also has support for C++ and .NET

GRID computing, in the sense of distributed computing with a focus on large-scale resource sharing and high performance computing, is starting to emerge as a viable direction for several computing problems (you can also see What's the grid  )

Below are links to several  projects that lets you create Grids using the .NET platform

• MS.NETGrid -   - An open source .NET implementation of  Open Grid Services Architecture and Infrastructure
• OGSI.NET - Another open source implementation of the same standard
• Alchemi.NET - Yet another open-source project - this time a part of the GridBus project
• BFC - Base one Foundation Class - a commercial GRID implementation

Happy Gridding

General overview of the SPAMMED process for developing software architectures

Introduction