Earlier today Microsoft announced it newest SOA initiative codenamed Oslo Here are a few observations I have on this announcement.

Let's start with "what is it?" Well it isn't an "it" per se, since Oslo is a bunch of initiatives within the Microsoft offerings.

For one, it is some of the libraries within .NET 4.0 -- specifically the next versions of WCF and WF.

Secondly, it is a bunch of designers and tools that will be part of Visual Studio (beyond VS 2008).
The most interesting component of Oslo will be a new repository to allow version management of models and services. I guess it is safe to say it will be built upon Team Foundation Server (or a subset of which which will be used by both products).

The last part of the puzzle is of course V.Next of Biztalk and something currently branded as "Biztalk Services '1'". As far as I know Biztalk sells pretty, but I think it is both too bloated (e.g., think about the hardware needed to run this in high-performance solution) and builds on the wrong architecture (hub vs. bus). I hope Microsoft makes major updates this time (Biztalk 2004 to Biztalk 2006 mostly innovated around the business activity monitoring. While that's important I think more work on the engine was/is due).

Biztalk services would offer an implementation of some of the SOA patterns I talk about -- service host, workflodize, etc. -- to provide an infrastructure for building services. The relation between "Biztalk 6" and "Biztalk services 1" is not clear from the information provided by Microsoft; hopefully this is just a branding issue and not a tight relation between the products.

On the upside, one of the key persons working on this is Don Ferguson who, before joining Microsoft, was chief architect for IBM's software group. About a year ago I had the chance to hear him talk about SOA and all I can say is that Don is someone who really knows his stuff.

PS: It's amusing to see the press release talks about "model-driven" approach rather than software factories, but I guess that's just nitpicking.
 

Let's assume I convinced you that some projects need architects (see part I). Convinced, you go and hire an architect. now what?

Let's start by looking at  "architectural decisions" - which is sure sounds like something we'd want an architect to do. I read once (I think that was Martin Fowler) that an architectural decision is a decision that in hindsiight you wished you made right. if we look at a formal definition of software architecture (say from IEEE 1471) we see that the architecture embodies the fundamental decisions about the system its components, their relations and their properties. Using this definition an architectural decision is a fundamental decision about the system (which pretty much explain why we want to make them right etc.)

Well, here are two observations on what I've said thus far. One is that we would want to postpone architectural decision as much as we can, since changing them will cause us a lot of headache. The problem is that in order to postpone an architectural decision we need to build flexibility into the system which is an architectural quality in itself - which might not be the top of the list if we prioritize it vs. other architectural qualities we need.

The second observation is that if we "refactor" the pretty language out from both of these definition - we can see that an architectural decision is basically a guess, hopefully that's an educated guess but it is a guess none-the-less. and as Albert Einstein once said it is  hard to make  predictions - especially about the future.

This is why architects  breadth of knowledge - which helps explain the architect training program I posted about a few weeks ago (see Architect training program Part I and Architect training program Part II). Another aspect is experience. And to get a wider perspective it can be helpful if this experience includes other roles besides developer such as project manager or business analyst etc. Another important component is domain knowledge and understanding of the business.

Using all these you (as an architect) may come up with a reasonable architectural decision (e.g. use MVC pattern) and a design to match it and that's it.

Well, actually, not quite since as I said earlier it is still a guess. Remember  an architectural decision (and any design for that matter) is a mirage no matter how beautiful the power point slide looks (or white board or UML sketch etc.)

Alas, power point compilers are still in the making. Which means that as an architect, you must be able to prove your point in writing - that is coding. While you are at it, you also need to know a thing or two about the technology you are using because it too has an architecture, features etc. which can have a significant effect on the end result. (You can read a little bit more on this in the "Architecture Deployment" paper I published a while ago).

The result of trying to postpone architectural decisions, ever changing requirements along with adding details as we unfold the  architectural abstraction level to a working system, is that the architect can't just appear at the inception of a project and disappear afterwards - they need to stick around for the game. This is especially true if you want to have an evolving architecture

An architect needs to do more than "architectural decisions". There are also additional reasons why the architect should have continuous interaction with the rest of the development team. However that will have to wait for Part III. :)

 

I received a number of inquires regarding PaperLnx following the help we rendered to Yediot in enhancing the quality of the video of Yitzhak Rabin's assassination (The link is to a site in Hebrew).

Our business is not video enhancement per se. What we do is use this and other similar proprietary technologies to provide a form of visual search for pictures taken on mobile phone cameras. "surfing" on a mobile phone is not a good user experience, typing URLs and search terms is cumbersome and lengthy. Using this image understanding capabilities we enable end-users to get at the relevant multi-media content for the object they are interested in (the object in the picture taken). This type of application are called "physical world connection" solutions. Naturally we are not alone in this space, but think that the technology we have gives us a competitive edge in the robustness of  our solution.

We are now at the stage where we the technology is pretty solid and we "only" need to turn this into a product, which is as I mentioned in another post, why we are looking for a few good men (and/or women) to join us.

 

I am currently interviewing people for developer positions for my startup (PaperLnx) and as I presented my plans to use multiple languages to one of the candidates he immediately retorted with "why on earth would you want to do that?"

Why indeed?
 
As I mentioned in the previous post about F# and Erlang it seems to me that the age of "one size fits all" is ending  in a lot of areas. I think that it is also happening for languages.

When .NET was announced, one of the messages by Microsoft was that .NET is the platform and that you can choose any language you want to develop in it. Microsoft also worked with partners to back up this statement and  release a lot of languages for the platform (e.g. Eiffel#, Fujitsu NetCobol  etc.) vs. Java which was a single language with a "write once, run everywhere" vision. Microsoft itself released 2 manages languages VB.NET C# and managed extensions for C++. The C++ managed extensions were very awkward to work with  and only people  who really had to use it. the non-MS languages seemed to be niche languages for legacy integration and not much more. VB.NET (we're talking .NET 1.0 here) seemed like something to try to distract VB heads while Microsoft was pulling the rug beneath their legs. Which basically left C# as the only viable language to develop in if you were serious about .NET (at least in my eyes). The "many languages" vision seemed like something to hide the (obvious) fact that Microsoft had to come up with an answer to Java (especially since around the same time Sun also sued Microsoft over the J++ and JVM implementations)

5-6 years later the situation is a little different. For one, we see that the JVM is not just about Java anymore. Just like the original .NET vision, the JVM has become a platform supporting many different languages Jython, JRuby, Scala, Groovy etc.
The .Net platform is also getting more interesting  languages like  Boo or   Gardens Point Ruby.Net . .NET is also being extended by Microsoft itself (with IronPython, IronRuby, F# ).

Two interesting things we can see are one, suddenly the languages are cross platform. e.g. I can program in python and deploy it natively, .NET or on a JVM - so in a way you get some platform independence.

The other and more significant issue (in my eyes) is that the languages available on each platform are different enough in meaningful ways - to a point where it is getting worthwhile to use more than one language in projects. Here are few examples from around the blogsphere that demonstrate this:

• Neal Ford shows an example using the (excellent) Mocha mocking library (in JRuby for testing Java classes. So the dynamic properties of Ruby help make the testing simpler for the statically typed Java.
• The BOO Build System (via Ayende) - which is about building a DSL for builds (a la rake). again we see taking advantage of the properties of a dynamic language this time to help with .Net languages.
• (not) Dennis Byrne shows how you can call Erlang from Java - which demonstrate taking advantage of the concurrency features of Erlang from Java.

When it comes to the difference between general purpose languages (i.e. Java, C#, VB.NET) we mainly see syntactic nuances. When it comes to dynamic languages vs. static languages vs. functional languages etc. the differences are more meaningful. What has changed now is that both platforms and open standards (e.g. REST over HTTP/WS-*) enable better integration and increase the motivation to use the "best tool for the job" approach over "the everything looks like a nail" approach .

I think the time for language pluralism is ripe - In my solution this would probably translate to a mix of C/C++, F#, C# and Ruby - What about your projects?

 

A couple of thoughts on functional programming in general and on F# and Erlang in particular

Erlang - is it worth it?
I read Armstrong's Erlang thesis and other Erlang related material with great interest. After all Erlang claims some very compelling architectural benefits in the areas of scalability and availability.  However one thing that prevents me from digging too deep into it is the utter ugliness of the language vs. for example, ruby which is very compelling. 
i also tend to agree with Otaku that at least today you can achieve this tauted resilience with other languages as well.

On the other hand we start to see some really interesting things built on Erlang like RabbitMQ which is an AMQP implementation (AMQP - The Advanced Message Queue Protocol), MNesia persistence engine . Yaws web server etc. (you can see here one guy that even likes Yaws + MNesia better than Ruby + Rails)

Well, I am still on the fence  for this one, but I guess I will take a deeper look at Erlang eventually..

By the way, while other functional languages also have irritating syntax (e.g. Scheme ) - some look more intereseting (like F# or OCaml)

 which brings me to the next tidbit...

Functional Languages moving mainstream.

Functional programming is an old paradigm (see for example this paper from 1984 "Why Functional Programming Matters" explaining the advantages of functional programming over structured programming..).

For a decade or even more and even today the object oriented paradigm has ruled the software development world. In a way, and as part of the end of the "one size fits all" paradigm (I also mentioned this trend here) we also see more pluralism for languages so we get more dynamic languages vs. static typed one and we find a place for functional languages and not just object oriented ones. (I'll expand more on that in another post).

Anyway, Yesterday S. Somasegar (MS VP of DevDiv) announced on his blog that F# will be "productized" :

"We will be partnering with Don Syme and others in Microsoft Research to fully integrate the F# language into Visual Studio and continue innovating and evolving F#. In my mind, F# is another first-class programming language on the CLR. "

Microsoft is of course not the center of the universe, but when a company like Microsoft chooses to bring something closer to main stream it is a significant move which, in my opinion, shows that functional programming is getting more traction.

 

One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?

The current answer I have is that the code, the tests and the acceptance tests all test each other so if one fails we'll spot the problem in at least one of the others. I hope that this  it is a good enough answer... :)

What do you think?

 

Well it seems setting up a new company can keep you busy at times - which is my official excuse :) for the quiet last week. Hopefully I'll have a little more free time this week

Note: I am talking in this post about roles and capabilities. i.e. when I say architect I mean someone who has the capacity to be an architect (e.g. as demonstrated in the architect training suggestion I made) and takes that role. On a specific project you may have a person that performs multiple roles such as an architect and project manger or an architect and developer while  other project warrant one or more full time architects.

Not all projects need architects. There, I've said it. Not all projects need architects and I am not talking here just about trivial projects. There are cases (maybe even many cases) where you can get by with what I call "off-the-shelf" architecture - maybe with a few adjustments that any master developer (i.e. seasoned and experienced developer) can handle. For instance a lot of web-sites can do pretty well by using Model-View-Controller (or a derivative of that) along with a simple O/R mapper such as active-record. In fact a lot of them do when they use a framework like Rails that made these architectural choices for them*. Another example is the vanilla 3-tier architecture provided by vendors (such as this one by Microsoft). Yes, when you take something off-the-shelf the result might not be optimal but that doesn't mean it isn't sufficient. You just have to be aware for the tradeoffs...

Another point is that "design" is not an exclusive architect thing. A developer is not a good developer unless she also known about  proper design. mastery of technical details of the language without understanding the wider context of design will just help you code a lot of crap faster.

Having said that we need to consider a few issues - When do you need architects? What do they Do? What's their relation and interaction with the developers ?

When do you need architects?

It is sometime a fine line between a project that can get by with an "off-the-shelf" architecture and one that needs an architect. It would be nice if we could have something like a litmus test that would tell us if architects are needed or not. I don't have one. The closest thing I could get to this is something I call the SCLR test (pronounced scaler). SCLR stands for Size, Complexity & Limited  Resources.

• Size - well if you are going to have something  estimated at 1000 man-years or dozens of teams. I think it is pretty obvious that you can't just use something  which isn't made to fit. If anything there's a need to divide the work between the teams in a way that would make sense so that you wouldn't get a big-ball of mud. There's also a lot of need to coordinate the efforts and keep the big-picture inline. Personally I think that it doesn't  have to be a huge project to warrent some architects involvement. Since as Fred Brooks notes the number of interactions grows exponentially as we add more people.  In my experience trouble starts even with more modest numbers - more than 4 or even 3 different teams working concurrently is probably a good number to start thinking  about architects
• Complexity - There are many signs for complexity in a project. The vision statement can provide a hint. "Let's design the software to support the next mars mission", "best CRM platform ever"  - an ambitions project will not make-do with "average" architecture. Size (which  I already mentioned) is also a sign of complexity, while previously I talked about size of the project, the size of data, users etc. is also relevant when we're thinking about complexity . A lot of external interfaces is another sign. Integration doesn't seem very complicated, until you actually try to pull it off. When you have to do a lot of that in a project that's complex. And there are many other signs
• Limited Resources - Naturally every project has limited resources, but limited resources should be considered as a sign for architect involvement if the resources are extreme. When resources are extremely limited the tradeoffs that have to be done are more meaningful, which is why wed want people who can help with that (i.e. architects). For instance in a projects I worked on in the past we had a lot of availability and performance requirements on one hand but only so many "U"s in the rack and even limited electricity to make all this magic happen. This turned something which otherwise was a relatively standard IT project into something a lot more challenging.
  

Assuming I manage to convince you that some projects can't just choose one of the available blue-prints and need some more work - the next step would be to convince you that architects are better suited to solve this than developers. I'll try to do that in the next post on this series where I'll explain what (I think) architects do and their place in the development team

---

* Rails has more than just MVC and Active-Record but that isn't an important point for this discussion

 

Pete Lacey has a post called "What is SOA?" where he defines SOA as follows:

"

• Network Oriented Computing (NOC): An approach to computing that makes business logic available over the network in a standardized and interoperable manner.
  • Service Oriented Architecture (SOA): A technical approach to NOC that has a non-uniform service interface as its principle abstraction. Today, SOAP/WS-* is the chief implementation approach.
  • Resource Oriented Architecture (ROA): A technical approach to NOC that has an addressable, stateful resource as its principle abstraction. Today, REST/HTTP is the chief implementation approach.
• Business Service Architecture (BSA): An unnecessary term (also not an architecture) that tries to make the obvious something special. Aka, business analysis. Aka, requirements gathering"

I am sorry but I beg to defer.

The first thing to note (again) is the architecture vs. architecture style differentiation I mentioned in a previous post (You can see a similar definition by Stuart Charlton) Here is a quick reminder :

Software architecture is the collection of the fundamental decisions about a software product/solution designed to meet the project's quality attribute requirements. The architecture includes the main components, their main attributes, and their collaboration (i.e. interactions and behavior) to meet the quality attributes. Architecture can and usually should be expressed in several levels of abstraction (depending on the project's size).

An Architectural style is a blue print that can be used when you desing an architecture. An architectural style defines some of the components and thier attributes as weel as place constraints on how they can interact.

My claim is that SOA is an architectural style for distributed computing which puts extra emphasis on the interface (and hence gets the easier interoperability). Ok, if SOA is indeed an architectural style, we should be able to define it as a set of components, interactions and attributes. Well, I already did that a while ago (in a paper called "What is SOA anyway?"). And while it may not be perfect, I think it is a reasonable definition all the same:

"SOA is an architectural style for building systems based on interacting coarse grained autonomous components called services. Each service expose processes and behavior through contracts, which are composed of messages at discoverable addresses called endpoints. Services’ behavior is governed by policies which can be set externally to the service itself. "

You can see the above mentioned paper for a little more detail on each of the components.

ROA, in my opinion, is just a re-branding of REST so that it would be easier to discuss it as an architectural style and not connect it to the HTTP implementation - which is what  a lot of REST proponents are doing.

By the way, as I pointed out before, there are a few other important architectural styles that are related to distributed systems like Event driven architecture, Spaced based architecture, peer-to-peer etc.

As for "Business Service Architecture" - I personally like to think about that as "SOA initiative" as in the strategic decision to try to implement an SOA in an organization while trying to achieve the more nebulous traits like business and IT alignment etc. (which is why it is nether architecture nor architecture style)

 

In a recent post Steve Vinoski said:

"Frankly, if I were an enterprise architect today, and I were genuinely concerned about development costs, agility, and extensibility, I’d be looking to solve everything I possibly could with dynamic languages and REST, and specifically the HTTP variety of REST. I’d avoid ESBs and the typical enterprise middleware frameworks unless I had a problem that really required them (see below). I’d also try to totally avoid SOAP and WS-*."

It is easy to dismiss this as just another yahoo who goes against conventional wisdom until you remember that Steve spent more than a decade working in Iona in leading roles like Chief Engineer of product innovations and helped develop some of the middleware standards for OMG and W3C.

Well, I guess that's becoming an epidemic  now :)  just recently we had Michael Stonebreaker talking about the RDBMS demise, Pat Helland talking about life beyond distributed transactions.  and now Steve on ESBs.

That trend aside, I think Steve is doing throwing the baby out with the bath water. The dream of a single infrastructure for an enterprise is ludicrous enough (Remeber Peter Deutsch and the "The network is homogeneous" fallacy). but if you drop the "E" from the ESB moniker you get a valuable middleware which is very usable in many situations and not just legacy system integration. For instance one thing that is missing form "HTTP variety of REST" implementation is reliable messaging. location transparency is  harder to solve with HTTP etc.

Another problem I have with the current approach of Steve is that he is replacing one dogma (EBSs are good) with another (ESBs are bad use Ruby, REST) - this is not a healthy approach. The solution should match the problem, that's probably the primary reason why we need architects after all

 

In the previous post on the subject I promised to expand a little more on the suggested content for "Distributed Systems Architectures Workshop" so here a short drill-down:

Even though most of the time should be spent on working, designing and evaluating architectures there's probably a little room for theory .

Module 1 The basics (probably not more than half a day)

What's software architecture
The software architect role
Activities
Scenario based architectural design
(documenting software architectures)
Agile SDLC and architects

Module 2 Distributed Systems background

Understanding the Fallacies of distributed computing

Distributed architectures styles - it is important to understand the different architectural styles that can be used to implement distributed system - whithin this topics like clustering, computation and data grids, messaging , publish subscribe etc should also be discussed

• Client-server - The most basic distributed architectural style. It is based on the  N=1 premise and isn't fit for most of today's challenged. However it is still an option for some types of projects.
• Pipe and Filters - not necessarily a distributed style, but it can be applied in distributed space
• N-Tier - That's actually a moniker to anything where N>2 but usually it pertains to 3-tier architecture (front-end, server, database) or the internet 4-tier version (client, webserver, application server, database). 
• Event Driven Architecture
• Service Oriented Architecture
• REST 
• Space-based architectures - like JavaSpaces  and its implementations like Blitz (open source) and Gigaspaces (commercial)
• Peer-to-Peer - you know that's what all those file sharing tools use

Distributed Consensus

• 2 phase commit - used by XA and COM+ distributed transactions
• 3 phase commit - considered a non-blocking protocol (vs. 2PC which is a bloging protocol) 
• Paxos commits
• Sagas
• Eventually consitent (BASE) - Basically Available Scalable/Soft state & Eventually Consistent. An alternative to distributed transactions used by a lot of internt-scale companies (see a post I made on ebay's architecture )

Module 3 - workshop - most of the days should be focused on actually working to design architectures.
I would think that this would be handles best by working  in groups. e.g. having each group focus on one architecture style.
 
The groups would be given a scenario which covers some architectural concern (integrity, performance, scalability, availability etc.)  and would try to design strategies to handle the scenarios within the constraints of the architectural style. Present that to the other groups and then have a facilitated discussion on the pros/cons of each strategy. The  scenarios should be based on a large enough story to allow meaningful architectures to emerge (e.g. you can see the  10 scenarios in my SOA Patterns presentation )


Any comments or other ideas for what's needed for this kind of a workshop are welcomed

 

It seems that even the smartest people can get the difference between architcture, architecture styles and technology wrong
For instance Anne Thomas Manes points out the Roy Fielding makes this mistake in his REST and Relaxation presentation by mixing an architectural style with technology:

 "Roy is equating SOA with web services. Although a lot of folks use web services to implement services, that's simply an implementation decision"

But then procede to make the exact same mistake 

"So when watching Roy's presentation, replace the term "SOA" with "WS-*", and the discussion will make a lot more sense."

REST is an architectural style you can implement it with WS-* which is a technology. It is not the most natural way to use WS-* standards but it is doable.

Looking at the same context (i.e. Roy Fielding's presenation) Steve Jones makes a similar mistake confusing Architecture and Architecture style.

My definition for software architecture is

Software architecture is the collection of the fundamental decisions about a software product/solution designed to meet the project's quality attribute requirements. The architecture includes the main components, their main attributes, and their collaboration (i.e. interactions and behavior) to meet the quality attributes. Architecture can and usually should be expressed in several levels of abstraction (depending on the project's size).

An Architectural style is a blue print that can be used when you desing an architecture. An architectural style defines some of the components and thier attributes as weel as place constraints on how they can interact.

For instance, the REST constraints (taken from Anne's post mentioned above) are:

"Uniform Interface:

• Resources are identified by only one resource identifier mechanism
• Access methods (actions) mean the same for all resources (universal semantics)
• Manipulation of resources occurs through the exchange of representations
• Actions and representations are exchanged in self-describing messages

Hypertext as the engine of state:

• Each response contains a partial representation of server-side state
• Some representations contain directions on how to transition to the next state
• Each steady-state (page) embodies the current application state"

Architecutre Styles can be combined to create new architectural styles. Roy Fielding demonstrates this in his famous dissertation  where he demonstrate how REST is a composition of several styles such as  Client/Server, Layered system, Stateless etc. As another example (which a lesser degree of precision) I take about enhacing SOA with EDA in "bridging the gap between BI and SOA"

The last piece of the puzzle is technology. Technology (in the software context) are set of tools provided by a vendor to enable and support building software solutions. As I've said here numerous times, technologies has their own internal architectures (as they are software solutions themselves) which is why different technologies support different architectural styles and why the alignment of the technology with the architecture chosen for your solution is important.

Yes this post is all about semantics - but clear meanings are important to prevent confusion, at least in my opinion anyway