Blog Home  Home RSS 2.0    
Arnon Rotem-Gal-Oz's Cirrus Minor - Everything
Archive
 
 Tuesday, August 14, 2007
One of the most interesting presentations in Architecture & Design world was the eBay Architecture presentation by Randy Shoup and Dan Pritchett. The presentation was only one hour long, so Randy and Dan didn't cover all the topics in the slide. Here are some of the insight I took from this presentation.

Architecture evolution  -  eBay actually went through several architecture revolutions. Their initial architecture cannot even begin to scale to their current loads. It was, however, a very good fit for their initial quality attributes - specifically, the emphasis on time to market and costs.  This shows the importance of balancing quality attributes. Sure an architectural change is painful but if they'd future proofed too much I doubt they would ever get something working.

V2 demonstrated that traditional 3-tier architecture would only scale so far. It was nice to see how it evolved though. Also with the move from version 2.4 to 2.5 and later to 3 we see eBay learning about  CAP - the hard way. In its final (current) incarnation eBay's data architecture prefers partitioning and availability over consistency. This doesn't mean they forgo consistency altogether - just that they trade the comfort zone of ACID transactions with the BASE approach. Where BASE - stands for Basically Available, Scalable/Soft state & Eventually Consistent. .
eBay partitions thier data in two levels one is a SOA like division by business areas (users, items etc.) and the second level is an horizental partitioning based on access paths.This BASE approach to data was dubbed by Dathan Pattishall (from Flicker and Friendster) as sharding (via HighScalability). This approach means things like high partitioning, no distributed transactions (also see below), denormalization etc. (you might also want to read the item I wrote on denormalizaiton in InfoQ yesterday).
The more major implication here is that when it comes to internet scale, the database looses its importnace - or as Bill de Hora nicely puts it:
The use of RDBMSes as data backbones have to be rethought under these volumes; as a result system designs and programming toolchains will be altered. When the likes of Adam Bosworth, Mike Stonebraker, Pat Helland and Werner Vogels are saying as much, it behooves us to listen.

As I said the data architecture of eBay is SOAish -  partitioned their components and data along business lines, and they apply many of SOA principles. They don't however unite data and components to create a service and  they don't (seem) to have the same contract boundaries that SOA promotes (Randy told me that they are currently contemplating SOA).

Returning to the  eBay do not use transactions. "no transactions" which seems very controversial  - but if we just consider some of the points I made on transactions between services in previous posts - it is the only logical way to ensure scaling. By the way, as can be expected  they do use transactions - when they are local e.g. if the users table is spread over a couple of table both will be updated together).

The application layers also follow the segmentation by business areas. eBay cacse metadata/immutable data as much as possible. keep the application stateless (i.e. state comes from client/db) e.g. they don't use sessions. The DAL virtualized the horizontal partitioning mentioned above for the rest of the code.

It was also interesting to  that eBay developed its own messaging infrastructure - though Randy and Dan did not provide alot of details on that

Development process - It seems that eBay is using some hybrid of feature driven development with waterfall (i.e. the development is feature by feature - but the development of a feature is waterfallish). The do have a constant delivery rate which they synchronize using the concept of a train. if you have a features that is it will be added to the train which is scheduled to arrive around the time your feature will be ready. Several features are delivered as a package which gives a predictable (weekly). I guess it also gives them some nice metaphors to use such as a feature that doesn't make it - misses the train or the train leaves on time etc.

The slides of the presentation can be downloaded from Dan Pritchett's site (They not from the same event but they are pretty much the same slides. Also you can read Elliotte Rusty Harold's account of the presentation.
8/14/2007 11:17:39 PM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   A&D2007 | Everything | SOA | Software Architecture  | 
 Sunday, August 12, 2007
If the previous post made it look like I think everything is just perfect with Ruby - then the answer, in my opinion, is no, it isn't perfect.

For instance, Let's take a look at a basic class like the Array class, which in an effort to be "Humane" violates several OO principles.
For one it violates Don't Repeat Yourself (DRY) by including several methods that do the same (#length and #size for instance) more importantly it violates the Single Responsibility Principle by exposing Stack (#push and #pop methods) and Queue (e.g. #shift).  Making an Array double as a queue is not just violating SRP is also implementation revealing rather than intention revealing.

Another, even more annoying problem is the sometime confusing scoping :
if you do something like
class Foo
    attr_accessor :bar # create accessor methods for a bar attribute
    def barTender(value)
       bar=value
       # do something with bar
    end
end

What happens here is that the bar in bar_tender is a private variable of the bar_tender class so setting it doesn't affect the bar attrribute. If you want to  set the bar property you need to call that with self.bar. Fortunately, the IDE I am using (Netbeans) has recently added a warning for this issue so I won't be doing these again.

I also find it annoying that a dynamic language like Ruby is also case sensitive so if you call something like barTender and bartender you'd get two different methods/variables etc. I guess that's why the Ruby convention is to use all lowercase i.e. bar_tender ( maybe I'll switch to that as well, you know, when in Rome..)

oh yeah and why the hell does  elsif misses the "e" ? :)

And there are of course other things. I do have to say though, that on average I find Ruby code to be more readable than Java or C# - I guess it is easier to define small code DSLs in Ruby (or other dynamic languages) than it is in Java or C# but I'll write about that in a different post

8/12/2007 10:14:43 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   Design | Everything | ruby  | 
 Thursday, August 09, 2007
I've been taking a look at ruby for a few month, I am finally getting to a stage where (I think/hope) I can actually say something intelligent about it. Last time I had an "aha moment" about a language was the fist time I saw Java.  C++  was oh-so-powerful, but Java was (is)  much more elegant and nice. Now Ruby changes the rules of the game again.
When I first heard about ruby I thought it was just a fad, something that the cool kids are using but its just another language. I getting more and more convinced that it isn't so. I am trying not to get too "silvery-bulletey"  here but working in ruby seems to actually increase productivity.
Let's take Dependency Injection(DI) as an example. DI is one of the most important and powerful tool I've learned in regard to Object Oriented development. Instead of classes depending directly on other classes classes depends on interfaces. And external classes (assemblers) provide them with their dependencies. This allows for loose coupling, increase testability and a lot of other such goodies (you can read a concise explanation in a paper I wrote on OO principles or get a more thorough explanation in a paper Martin Fowler wrote on "Inversion of Control Containers and the Dependency Injection pattern".  The .NET and Java worlds are filled with a lot of frameworks to help solve this elegantly. Spring (and Spring.Net) is probably the most known one.

How do you do DI in Ruby? in two words - you don't
If I am to join the "Define DI in one sentence" challenge by Jim Weirich I would say that
"DI is a powerful and good workaround to  the collaboration coupling problem between objects which is best addressed at the language level"

Why doesn't ruby need DI?

Well, I would say that it all starts at the basics. I remember when I learned OO, I was told objects communicate using messages. I never really understood why they call "method invocation" messages - it doesn't make any sense. The point is that in ruby you really don't "call a method" you "send a message"*.  When you make a call like someVariable.SomeMessage - the ruby interpreter  doesn't really care about the type of someVairable just that the object it holds (and everything is an object) has some entry which can handle SomeMessage.
Let's start with a simple example
consider the following code:

class Foo
  def bar
    puts "Foo-bar"
  end
end

class Foosa < Foo
end

class Baruser
  def baruse(b)
    b.bar  #dependency
  end
end

bu= Baruser.new

bu.baruse(Foo.new)
bu.baruse(Foosa.new)   # sub-class

Well, nothing particularly exciting here. if you run this  you get foo-bar printed twice. That's very much like the dependency injection you see in .NET or Java
It gets a little more interesting when we consider that the following classes would all work as well

class Foz
  def bar
    puts "Foz-bar"
  end
end

class NoBar
  def method_missing(methodname, *args)
    puts "NoBar" if "bar" == methodname.to_s
  end
end

class MakeBar
   define_method(:bar) {puts "madebar"}
end

bu= Baruser.new

bu.baruse(Foz.new)     # another type altogether
bu.baruse(NoBar.new)   # a class that doesn't have bar method
bu.baruse(MakeBar.new) # a class where the bar method is created programatically
 
We can see from the examples that what ruby does is searching for a handler for the bar message. The handler can be a method (symbol) called bar or a generic handle like Missing_Method - ruby doesn't care as long as the message get handled

I think that's pretty nice, we have a lot of flexibility on the dependency side but the depending class still  essentially gets the dependency by injection (the call to baruse)
Well, ruby can help us flex the dependent side as well. The answer is in the last example which uses (an overly simple an uninteresting example of) meta-programming Consider the following code example using Mocha which is a Mock object library for ruby
 
Lets say we modify our Baruser class to the following.

class Baruser
  def initialize
    buildfoo()
  end
  def baruse
    foo = buildfoo()
    foo.bar  #dependency
  end
 
 private 
  def buildfoo
    Foo.new #dependency
  end
end

Note that buildfoo is private - if we wanted to test this in.NET we need to have Foo around. i.e. we can no longer test Baruser by itself
if we use Mocha in ruby we can do the following:
foo = mock('foo')
foo.expects(:bar).at_least_once.

bu = Baruser.new
bu.stubs(:buildfoo).returns(foo) # basically what happens here is that the instance we have is changed
bu.baruse

I can't believe they invented it  - You can get  ruby for just the price of download and if you call within the next 15 minutes we'll throw in a copy of gems free of charge :)

In any event what we see here that using meta-programming and other ruby constructs we can forgo using DI altogether - no wonder Neil Ford defined DI as :
"Dependency Injection enables a vitally important but nevertheless weak, limited, syntactically confounding, and dauntingly complex form of one of the kinds of meta-programming that should exist in the language."
 Not to mention that the resulting code is much more elegant. which is actually what I like best about ruby, the code is much cleaner (but that's for another post)

Some closing thoughts

  • .NET 3.5 bring some of the ruby goodness to C# - but as the previous post demonstrated it is just a move in the right direction but not the whole thing
  • While I am on the subject of the previous post the whole interface vs. class thing is, of course, a moot point in Ruby since there are no interfaces. Interfaces, like DI, are another thing that is very important in C# and Java and not needed in Ruby
  • I've read some complaints on Ruby's performance. performance is important but there are two things to remember. First, the fact that a solution is not the fastest doesn't mean that it isn't fast enough. Second, I can still vividly recall the performance benchmarks for our Java code before we got the first hot spot compiler installed. The point is, that if it is important to enough people it will get better.




* I know, I know, Smalltalk had it since the beginning of time. However in Smalltalk everything is an object, in ruby you can also write plain scripts and (more importantly for me) - I never really took more than a cursory look at smalltalk  so I never  saw that.

 

8/9/2007 1:01:43 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [5]   .NET | A&D2007 | Design | Everything | ruby  | 
 Monday, August 06, 2007
One of the interesting presentations I attended in Architecture & Design world was "User Interface Principles in API design" by Elliotte Rusty Harold.
Elliotte started off by mentioning that developers are people too (I guess...) and that designing an API we need to think about the people (developers) who are going to use it.  I totally agree, in fact, when it comes to API design is one place I insist developers use TDD even if they don't do it on their day-to-day development. Dogfooding their own API is probably the only way to really ensure you get a usable API. Elliotte thinks you should go further than TDD and write some sample programs first.

The challenges of API or library design come from the fact that you cannot be sure about Elliotte mentioned some of the fundamental principles of UI design that apply to API design.
  • The importance of consistency
  • Simple is better (and for god sake please keep complexity internal) which translated to YAGNI and "when in doubt, leave it out")
  • Smaller surface (API) is easier to use.
To demonstrate the point of proper API design Elliotte compared the complexity of JMidi vs. the elegance of JFugue. While in JFugue you just use a couple of lines to play a few notes:
Player player = new Player();
player.play("C D E F G A B")
JMidi required more than 20 lines of setting all sorts of things related to its internal structure.

Elliotte also had a lot of practical advice on varios aspects like maintenance, specific advice for Java, specific advice for .NET etc. - Instead of me reiterating it I suggest you just take  a look at the presentation which is available on his site.

One point Elliotte made which I don't completely agree with was that you should prefer classes over interfaces. we discussed this issue after the presentation and I said that it is true for libraries but I am not sure that it is always true for frameworks (see Frameworks vs. Libraries for more details on the difference). in IoC scenarios interfaces allows extending the framewok for use with user defined types. One real-life example I offered was a plug-in framework we developed internally a few years ago. This framework incorporated a fixed set of components based on certain 3-rd party components (grid, tree etc.) - This worked very nicely, until in one project we wanted to use other 3rd party components (let's say a ribbon or another grid) which left us with either waiting for the infrastructure guy to find the time to make the changes, fork the infrastructure code, or build something new (we went with building something new since working with end-users we found we actually  didn't need most of the capabilities the framework offers like security/roles, role tayloring capabilities etc.)

On the other hand one argument for using classes over interfaces that Elliotte is right about is that when you use interfaces and then make a change in that interface. The chage  breaks all clients. If however all the clientes used a subclass they would automatically inherit the new functionality.

I thought I could work around this limitation in .NET 3.5 (or whatever microsoft will end-up calling it) by using extension methods so I did something like the following (tests etc. are excluded for brevity)
  class Program
{
static void Main(string[] args)
{
testInterface tc = new testClass();
tc.DoSomthing();
tc.DoSomethingElse();

}
}

interface testInterface
{
void DoSomthing();
void DoSomthingElse(string test); // compilation error....
}

static class testInterfaceDefaultBehaviour
{
public static void DoSomethingElse(this testInterface ti)
{
System.Console.WriteLine("Doing something else");
}

}

class testClass : testInterface
{
public void DoSomthing()
{
System.Console.WriteLine("Doing something here...");
}
}
I was wrong....

Main does  recognized the DoSomethingElse pretty well - however adding it to the Interface definition caused a compilation error.
This is a shame since the MSDN documentation says that
 " In effect, extension methods make it possible to extend existing types and constructed types with additional methods."
If it had worked that would have let us have something like method_missing in ruby in C#. Well, .NET 3.5 is still in beta so maybe it would change (not counting on that)

8/6/2007 1:06:12 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   .NET | A&D2007 | Design | Everything  | 
 Wednesday, August 01, 2007
I won't say anything about my presentations (that's for others to say :) ). The point of this post is just to let you download them. So here they are:
  • SOA Patterns (2.14mb) - Takes a look at different strategies (patterns) to solve common SOA pitfalls
  • Getting SPAMMED for architecture (4.56mb) - Takes a look at the activities architects can/should do when they think about software architectures. The presentation also covers architecture in agile projects.

8/1/2007 10:52:49 PM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   .NET | A&D2007 | Agile | Everything | SOA | SOA Patterns | Software Architecture | SPAMMED Process  | 
 Monday, July 30, 2007
Just got back from Chicago, where I attended Architecture & Design World. The event was great. it had many interesting talks and sessions  which I'll blog about in  few of the following posts (Also I'll upload the slides from my sessions later this week). One session however, was very disappointing for me, Ivar Jacobson's Keynote on "Next Generation Process with Essential Modeling".

I was really looking forward for this keynote; After all Ivar is one of the "Three Amigos", father of use cases and all. Also I had the chance to attend a couple of his presentations several years ago and he had a very interesting and convincing appearances at those times. Unfortunetly this session was nothing like those previous events.
Ivar talked about his new methodology. The basic idea behind the essential software process is (in my opinion) correct. Instead of big bang prescribed processes - it is better to tailor a process from a set of practices to get something that fits the organization/project at hand. It might be a little unnerving to hear this from one of the people who brought (upon) us the RUP, but, I guess it is good to see people who are constantly learning.

The problem was that while the main idea can be summed up in one line (see above) Ivar went on fro the better part of the hour to explain how cool it was that his base set of practices comes on a set of que cards that can be sorted and arranged. He also went on to explain the game board (or something to that effect) where you can lay different cards to both describe your current process as well as the bunch of practices you want to use for your future process.
If that wasn't enough, when he (briefly) shown us the cards that talk about the architecture and architecture description practices the advice/guidance there was very mediocre and general
I expected much more from someone of the stature of Ivar Jacobson.
On the up-side this was only one keynote and the other sessions I listened to where much better - as I already said, I'll blog about few of them in upcoming posts.
7/30/2007 11:01:20 PM (Jerusalem Standard Time, UTC+02:00)  #    Comments [1]   A&D2007 | Agile | Everything | General  | 
 Monday, July 23, 2007
While I am getting ready to fly to A&D world 2007 where I'll present both SOA patterns and the SPAMMED architecture framework, I thought I'd throw in a little update on the book as well.

I've made a small change to the way chapters 5-7 are organized. They are now grouped under a separate part called "Service Interaction Patterns" (and chapters 2-4 are grouped under "Structural Patterns").
  • Chapter 5 is focused on Message Exchange Patterns (MEP): synchronous, asynchronous, events and transactional  - The patterns there are not new for SOA, instead the focus is on the meaning of implementing the usual MEPs under SOA constraints. I sent it to manning early last week so hopefully it would be available on MEAP soon.
  • Chapter 6 is called "Consumer Interaction patterns" and includesthe UI interaction patterns as well as interaction pattern with other types of consumers. This is the chapter I am currently working on.
  • Chapter 7 is unchanged for now
Lastly,  as you may remember,  I publish online one pattern from each chapter so I'd be happy to get comments on which of the following three patterns (from chapter 6) you like  to see on-line: Reservation pattern (making partial commitments), Client/Server/Service (integrating Legacy or thin clients with SOA) , Client/Service (integration Rich clients with SOA) - if you want to vote just send me an email or leave a comment

7/23/2007 11:45:51 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   Everything | SOA | SOA Patterns  | 
 Sunday, July 22, 2007
I've got a few emails from people who asked about the commit and consensus protocols (Paxos, 3PC etc.) mentioned in the previous post.
I was going to write something up but then I find a very good overview by Mark MC Keown which provides "A brief history of Consensus, 2PC and Transaction Commit". This short writeup provides links to the most important papers like the Fischer, Lynch and Paterson (FLP) "Impossibility of distributed consensus with one faulty process", Jim Gray (who, unfortunately, was lost at see earlier this year)  and Leslie Lapmort's "Consensus on Transaction Commit". In addition to Mark's post and all the papers there you may want to check out the presentation Jim and Leslie made on paxos commits

By the way, if on the other hands, you already know all this (and a little more) than check out Werner Vogel's (Amazon CTO) blog as there's a interesting  job opening for you there

7/22/2007 9:16:23 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   Everything | Software Architecture  | 
 Thursday, July 19, 2007
Following the previous post I had a chance to exchange a few email with Mark Little (the director of engineering in the JBoss division of Red Hat). Mark thinks the topic of transactions and SOA has been beaten to death already and wonder's why does it need to resurface (see his post "Is anyone out there?") - I don't see a problem with discussions resurfacing when new people are faced with situations others already solved (but that's a matter for another post)

Anyway, the reasons we're here is  that I think that during this conversation  mark made a few interesting observations and I think the end result is pretty interesting. I decided (with his permission) to post it here ( It is only minimally edited: no deletions, few additions (in []) and a few time shifting to make it more coherent as a single conversation)

Mark
: From what I can see it's [the arguments on transaction and services - are] the same old arguments that have gone round and round, ignoring the important fundamental issues and not doing enough background research.
Sagas are transactional - it's just an Extended Transaction model and not an ACID transaction model. Don't get hung up on the word "transaction", which is way to overloaded in our industry to actually mean anything by itself. Plus, 2PC is a consensus protocol too; it does not impose any other aspect of ACID than the A. Even the D is optional until/unless you want to tolerate failures.

Arnon I know this is an old argument - but that doesn't mean it isn't worthwhile

Mark It isn't worthwhile if people aren't going to listen ;-) I've been involved in these debates so many times over the past 7 years (for Web Services transactions) and longer for extended transactions, that it gets a bit old after a while. Maybe we should create a wiki page and point people at that ;-)?

Arnon I guess, but you should keep in mind that people who are solely in the .NET camp only got WS-AT recently with Windows Communications Foundation so you can expect the issues to resurface. By the way a wiki might not be a bad idea 

[regarding 2PC.] 2PC is a distributed consensus protocol and in principle doesn’t have to be related to ACID transaction. But I think the common view and use of it is for ensuring distributed ACID behavior. Looking back at my experience with XA and COM+ transactions it seems it does a good job at achieving this ACIDness


Mark This is an education issue. The literature is clear on this. People who know and understand transactional protocols don't make the mistake of equating 2PC to ACID properties.

Arnon Yes it is an educational issue . But I am not sure that it is that common knowledge. It is expected that middleware vendors who build the tools to support these protocols to understand it better - I don’t think it is that widely known outside these circles. Most of the architects I’ve don’t (maybe It time to look for new friends ;-))

By the way as 2PC is not resilient to failures of the coordinator so in a highly distributed environment like SOA it might have been a better idea to go with paxos commits if at all you go down that path.

Mark The reason WS-AT and WS-ACID chose 2PC is: interoperability. All TP monitors support it. Try getting IBM, MSFT, Oracle, BEA etc. to change to Paxos, 3PC, flat-commit, or anything else and you'll be waiting for the heat death of the universe.

Arnon Can’t argue with that

Mark [also] 2PC is resilient to failures if the coordinator eventually recovers. Paxos has its own failure assumptions too: Jim never disputed this. Same as 3PC and other consensus protocols. As with *any* fault tolerance approach (transactions, recovery blocks, replication, etc) it's always probabilistic. All we're doing is making it highly unlikely that the system cannot complete, but we can never make it entirely safe. Even in the airline industry they can "only" go to a probability that failures happen .000000001 ;-)

Arnon You are probably right that in an SOA situations the chances of not getting an ACID transaction are worse than in a controlled environment - which actually make the situation even worse since people using WS-AT perceive it as allowing them ACID interaction (e.g. Juvals podcast) .


Mark WS-AT is *all* about ACID, in the same way WS-ACID is about ACID transactions. It is *nothing* to do with SOA though. Web Services are not purely the domain of SOA implementations!

Arnon I totally agree that Web-services and SOA are not directly related and can each exist independently of each other. Again this is an educational issue but,  SOA==Web-services is a very common misconception (I guess the word “service” in web-service doesn’t help ;-) )

[in any event] I think distributed transactions in general should be used carefully period.


Mark Absolutely. They are not a global panacea and people who push them as such do more harm than good.

Arnon WS-AT is more problematic than regular distributed transactions as by definition in an SOA you do not know who and how many other services will participate in your transactions so you are much more likely to run into problems.

Sagas which embrace the temporal shift don't give an illusion of ACIDness and allow to focus on achieving distributed consensus while keeping all parties involved consistent. I think that it is a much better option if you need transaction-like behavior

Mark For SOA, yes. Although Sagas are only good for a certain type of use case. That's why we've always tried to develop "live documents" that allow people to add new models when/if needed. With a couple of exceptions during the BTP days, there has always been consensus that one size does not fit all (http://www.webservices.org/weblog/mark_little/blackadder_and_the_micro_kernel_approach_to_web_services_transactions).

Web Services *anything*, whether it's WS-AT, WS-Sec, or WS-Addressing all have their non-SOA aspects because Web Services aren't developed purely with SOA in mind. If that were to happen then Web Services as a technology would lose some of their important benefits immediately.

Arnon This whole discussion is in the context of SOA (at least from my side ) – naturally there’s a place for ACID transactions for other uses.

Regarding Sagas - calling them "Extended transactions that are not ACID" is just semantics - my point was that they are not ACID transaction. I think most people equate transactions with ACID transactions as well (but I may be wrong)

Mark Many people do and that again is an education problem. The term Extended Transactions (don't need to say "that are not ACID") has a well defined meaning in the R&D community. There have been many good models and implementations around Extended Transactions. They really took off in the vendor community through the Additional Structuring Mechanisms for the OTS, back in the 1990's. If you check that out you'll see that it formed the basis of WS-TX and WS-CAF. Even in Jim's original technical report he discussed relaxing all of the ACID properties in a controller manner to get more flexibility. That was the first extended transaction. In fact, ACID transactions are just one type of extended transaction. There are many many others, including nested transactions, coloured actions, epsilon transactions, sagas etc.

Unless I qualify it beforehand, I try to never use the term "transaction" in isolation because it has different meanings to different people. For example, when talking to developers working in trading infrastructures, a "transaction" isn't an ACID transaction at all. In telecos it's different again.

7/19/2007 12:19:49 AM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   Everything | SOA | Software Architecture  | 
 Friday, July 13, 2007
Evan H asked a question about distributed transactions and services in the MSDN architecture forum:

Are distributed transactions (ie.. WS-Transaction) a violation of the "Autonomous" tenant of service orientation?   Yes or No and Why?  Kudos if you can address concurrency and scalability (in an enterprise with multiple interacting services).

I answerd this questions back in april when I wrote a couple of posts that explained why cross-service transactions are a bad idea:cross service transactions and some more thoughts on cross service transactions.
Roger Sessions also agrees with this view (well, it seems actually, he wrote about it well before I did :) ):
When the WS-Transaction specification was first proposed, back in 2002, I wrote an article explaining why I thought the idea of allowing true transactions to span services was a bad idea. I published the article in The ObjectWatch Newsletter, #41: http://www.objectwatch.com/newsletters/issue_41.htm. Nothing since then has changed my mind. Atomic transactions require holding locks, and spanning transactions across services requires allowing a foreign, untrusted service to determine how long you will hold your very precious database locks. Bad idea. Just because IBM and Microsoft agreed on something doesn't make it good!

The reason I am bringing this issue back is that Juval Lowy (who wrote the article that triggered my first post on the subject) has recorded an Arcast with Ron Jabobs. Where he re-iterated the idea that "Transactions is categoricaly the only viable programming model" and you should strive to use it whenever you can. It seems Juval admits you sometimes need to use Sagas (which he called "long running transactions" - you can see in my link why I think that's a wrong name). He also agrees that you can also use a transactionable transport and then only do internal transactions from each service to the transport (a pattern I call "Transactional Service"). However, at the end of the day, he still thinks you should use WS-AtomicTransactions whenever you can.

I agree that transactional programming is important. I think it is the simplest programming model (from the developers side). I would probably never write an interaction with a database that is not transactional; I look very favorably at initiatives for in-memory ACI (no Durability) transactions such as the one Ralf talks about.  Until we get to Distributed Transactions...

First, we should note that transactions are not "the only viable" option.As Martin Fowler notes Ebay seems to be doing fine without distributed transactions. Not only that, they abandoned distributed transaction and went "transactionless"because they needed one simple thing... Scalable performance .

In most COM+ scenarios you have a single server or a few internal servers where the distributed transaction happen - and even there you should plan your transactions carefully if you want to get any kind of decent performance. In SOA scenarios the situation is more complicated as the distribution level is expected to be higher (even if you don't involve services from other companies). More distribution means longer times to complete transactions (especially if a participant can flow the transaction and extend it). It also means increasing the chances of failure (see Steve Jones series of posts on five nines for SOA). In my opinion, the more distributed components you have the more you want their interaction to be decoupled in time - i.e. the opposite of transactions.

Juval also said he doesn't buy the denial of service problem I mentioned (supporting a transaction means you allow locks - if an external party doesn't commit you retain the lock..). Juval said he assumes that a solutions has both authentication and authorization so this shouldn't be an issue. For one, I have seen too many projects where security was something that was neglected or quickly patched in at the latest moment - so I would hardly assume security. Even with security on - you increase your attack surface.
But that's just the half of it. Even if all your service consumers have good intentions - you still don't know anything about their code. SOA is not like the "good old days" where you owned the whole application  - this means you cannot trust their security to be ample. Also you don't know anything about their code quality. Services are likely (in the general case) to be deployed on different machines, even if they start co-located. I think that a Service boundary should be treated as a trust boundary just like a tier boundary. I strongly believe you should have reduced assumptions on what's on the other side of the service's boundary - transactions are not reduced assumptions

SOA and distributed transactions do not go hand in hand - it isn't just autonomy at stake here. It is a problem for performance and scalability and even security period.

To finish this post - I would also highly recommend looking at Pat Helland's paper "Life Beyond Distributed Transactions: an Apostate's Opinion" and a post he recently made called  "SOA and Newton's Universe", where he explains more eloquently than I ever could why SOA is not a good fit for distributed transactions.


7/13/2007 10:47:03 PM (Jerusalem Standard Time, UTC+02:00)  #    Comments [3]   .NET | Everything | SOA | SOA Patterns | Software Architecture  | 
 Monday, July 09, 2007
Steve Jones has (yet another) great post called "Le Tour SOA - why support services are critical, but not important".
You should go read the article - but in a nutshell, Steve explains that important services are the ones that bring business values and critical services are the supporting ones that help keep the light on for the important services to function properly. 

While the post has SOA in the title. I think it is more general and is also applicable to applications or any other IT generated components. In fact it can also be applicable to IT itself as Nicholas Carr noted in 2003 when he published his paper "IT doesn't matter". Nicholas argues that IT will become akin to electricity and as such be critical for the business to continue operating but not important. As a side note I'd say I think this is might be true for traditional businesses but not for businesses where the IT is the business (such as banks, insurance companies, etc.)

Back to critical vs. important - I think this is an important for architects to make this distinction to be able to prioritize work and not confuse business value with semblance of business value due to criticality for operations. This doesn't mean you can neglect critical tasks (after all they are critical...). It is the important stuff that will bring your business the competitive edge.

7/9/2007 11:55:22 PM (Jerusalem Standard Time, UTC+02:00)  #    Comments [0]   Everything | SOA | Software Architecture  | 
 Saturday, July 07, 2007
Coming back from vacation I saw that Jeff Atwood (coding horror) wrote a post on "rethinking design patterns"
a few days ago. Jeff criticizes the GoF design patterns book
Jeff says his t