In one of my previous posts (Rest: good, bad and ugly), I made a passing comment, about how I think using CRUD in RESTful service  is a bad practice. I received a few comments / questions asking why do I say that – so what’s wrong with CRUD and REST?

On the surface, it seems like a very good fit (both technically and architecturally), however scratch that surface, and you’d see  that it isn’t a good fit for either.

REST over HTTP is the most common (almost only) implementation of the REST architectural style - to the point REST over HTTP is synonymous with REST. I would say most of the people who think of REST in CRUD terms, think about mapping of the HTTP verbs.

CRUD which stands for Create, Read, Update and Delete, are the four basic database operations. Some of the  HTTP verbs, namely POST, GET, PUT and DELETE (there are others like OPTIONS or HEAD) seem to have a 1-1 mapping to CRUD. As I said earlier they don’t. The table below briefly contrast HTTP verbs and CRUD

Verb	CRUDdy Candidate	Actually
GET	SELECT (Read)	Get a representation of a resource. While it is very similar to SELECT it also has a few features beyond an out-of-the-box SELECT e.g. by using If-Modified-Since (and similar modifiers) you might get an empty reply.
Delete	Delete	Maps well
PUT	Update	Put looks like an update but it isn’t since: 1. You have to provide a complete replacement for the resource (again similar to update but not quite) 2. You can use PUT to create a resource (when the URI is set by the client)
POST	Insert	It can be used to create a   but it should be a child/subordinate  one. Furthermore, it can be used to provide partial update to a resource (i.e. not resulting in a new URI)
OPTIONS	?	Get the available ways to continue considering the current state or the resource
HEAD	?	Get the headers or metadata about the resource (which you would otherwise GET)

The way I see it,  the HTTP verbs are more document oriented than database oriented (which is why document databases like CouchDB are seamlessly RESTful). In any event, what I tried to show here is that while you can update, delete and create new resources the way you do that is not exactly CRUD in the database sense of the word – at least when it comes to using the HTTP verbs.

However, the main reason CRUD is wrong for REST is an architectural one. One of the base characteristics(*) of REST is using hypermedia to externalize the statemachine of the protocol (a.k.a. HATEOS– Hypertext as the engine of state). The URI to URI transition is what makes the protocol tick (the transaction implementation by Alexandros  discussed in the previous post shows a good example of following this principle). 

Tim Ewald explains this  nicely (in a post from 2007…) :

… Here's what I came to understand. Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I'm not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressableg by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you're moving to, making that your new state. A state's representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app's protocol can't work that way too. (The ATOM Publishing protocol is the canonical example, though its easy to think that its about entities, not a state machine.)

If you are busy with inserting and updating (CRUDing) resources you are not, in fact, thinking about protocols or externalizing a State machine and, in my opinion, miss the whole point about REST.

CRUD services leads and promoted to the database as a service kind of thinking (e.g. ADO.NET data services) which as I explained in another post last year is a bad idea since:

1. It circumvents the whole idea about "Services" - there's no business logic.
2. It is exposing internal database structure or data rather than a thought-out contract.
3. It encourages bypassing real services and going straight to their data.
4. It creates a blob service (the data source).
5. It encourages minuscule demi-serices (the multiple "interfaces" of said blob) that disregard few of the fallacies of distributed computing.
6. It is just client-server in sheep's clothing.

The main theme of this and the previous post is that if we try to drag REST to the same old, same old stuff we always did we wouldn’t really get that many benefits. In fact, the “old” ways of doing that stuff are probably more suitable for the job anyway since they have been in use for a while now. and they are “tried and tested”  (“You can’t win an argument with an idiot, he’ll just drag you down to his level and beat you with experience” …). REST is just  a different paradigm that RPC, ACID transactions and CRUD.

---

* I know I sound like a broken record on that but our industry has a history diluting terms to a point they almost stop being useful (SOA comes to mind..). The way I see it you can have 3 levels on your way to REST over HTTP:

• You can be using HTTP and XML/JSON – this is level 1 or “Using standards”.
• You can be using the HTTP verbs properly and/or applying document oriented communications – this is level 2 or “Rest-like” interface
• You can conform to all REST constraints and be at level 3 or “RESTful”.

All levels can be useful and bring you merit but only the 3rd is REST

 

 

Yesterday I read an interesting paper called “RETRO: A RESTful Transaction Mode”. On the good side, I have to say, it is one of the best RESTful models I’ve seen thus far. The authors took special care to satisfy the different REST constraints, unlike many “RESTful” services (e.g. twitter that returns identifier and not URIs). On the downside is I think a distributed transaction model is bad for REST or in other words I don’t see a reason for going through this effort and jumping through all these hoops.

Why?

For the same reasons transactions are wrong for SOA and  why WS-AtomicTransactions is wrong for SOAP web services:

• Service Boundary – RESTful or otherwise is a trust boundary. Atomic transactions require holding locks and holding them on behalf of foreign service is opening a security hole (makes it much easier to do a denial of service attack)
• You cannot assume atomicity between two different entities or resources. Esp. when these resources belong to different businesses.
• Transactions introduce coupling (at least in time)
• Transactions hinder scalability – It isn’t that you can’t scale but it is much harder

For rest it is even worse - Since using hypermedia as the engine of state change means that the hypermedia actually  describes the protocol, we clutter the business representations (the representations of real business entities like customer, order etc.) with transactional  nitty-gritty as the authors say:

“our model explicitly identifies locks, transactions, owners and conditional representations as explicit, linkable resources. In fact, every significant entity in our model is represented as a resource in order to comply with this constraint.”

This also means the programming the resources themselves will get much more complicated

I think that if you want to reap the benefits of REST you should keep the protocol simple and focus on the business and technical merits you can get not bog it all with needless complexity. It seems to me that RETRO is a good mental exercise to show transactions can be RESTful. I think, however that it is an overkill for RESTful implementations.

RESTful architectures will be better off with BASE (Basically Available, Scalable, Eventually Consistent) and/or ACID2 (Associative, Commutative, Idempotent and Distributed) models –or at least the Saga model (which the authors intend to tackle next) which  is a better candidate (IMHO) for achieving distributed consensus.

 

I recently read a post by  Tim Bray where he states that building on web technologies let you get away with believing some of the fallacies of distributed computing.

I personally thinks he is a little optimistic in that claim.

On “The network is reliable” – Tim says that that the connectionless of HTTP helps (it does) and that GET, PUT and DELETE are idempotent helps as well. I say that GET, PUT and DELETE only if the people implementing the server side make them so – i.e. consider the fallacy. The fact that the HTTP says they should be idempotent doesn’t automatically make each implementation compliant

On “ Latency is Zero” – Tim says the web makes it worse – but, he claims, users got used to that. Even if they did I think that users are just part of the picture since the programmable web is also making strides. Also as Tim says it is actually worse. Not to mention that “Latency isn’t constant” either

On “Bandwidth is infinite” – Again Tim agrees that it is worse but people learn to note it. Again learning that it is there doesn’t mean the fallacy is gone just that people are less likely to presume it

On “The Network is secure” – Tim says its probably the “least-well-addressed by the web” – no argument here

On “Topology doesn’t change” – Tim says URIs help mitigate it – Again Tim is assuming people make URIs permanent or will always return a temporary redirect/permanent redirect when a URI change – good luck with that.

On “There is one administrator” – Tim says that yes that’s the case but who cares. Well, an example I usually give is that time when I deployed an ASP.NET which worked for a while – until the hosting company decided to change their policy to partial-trust (the app. needed full-trust) – when that happens to you. You care. If you mashup with someone else, you care etc.

On “Transport cost is Zero” – Tim says it is the same as for Bandwidth – i.e. worse.

On “The network is homogeneous” – Tim says that that’s this is the “web’s single greatest triumph”. I actually agree to that as long as all of you stick to using the web’s ubiquitous standards (http, XML/JSON ) if you have parts of your application that can’t use that you still need to pay attention

One thing I am really  puzzled by is Tim’s conclusion :

“If you’re building Web technology, you have to worry about these things. But if you’re building applications on it, mostly you don’t.”

Since even according to him only 4 fallacies are covered by the web… (I think only 1)

In any event, I agree that the web standards and REST in particular, do contain guidelines that take into consideration the fallacies. However it is still up to developers to understand the problems they’ll create if they don’t follow these guidelines. Assuming that that is indeed the case, is well, overly optimistic in my experience.

You can also read a paper I published a few years ago which explains the fallacies  and why they are still relevant today.

 

I recently got a request from Alik for my opinion on REST. I think  this might be interesting for a wider audience and decided to blog my answer here.

Note: I also have a REST presentation I prepared awhile ago, which is downloadable from here (ppt)

The good

As you probably know REST is an architectural style defined by Roy Fielding for the web which is built on several foundations (client/server, uniform interface etc.) which gives it a lot of strength in affected areas. The top three in my opinion are:

• (relatively) Easy to integrate – a good RESTful API is discoverable from the initial URI onward. This doesn’t suggest that a any application calling on on your service will automagically know what to do. It does mean however that the developer reading your API trying to integrate it has an easier life. Esp. if since hypermedia provides you the roadmap of what to do next.
• Another feature for ease of integration which has to do with REST over HTTP (THE most common implementation of REST ) is the use of ubiquitous standards. Speaking HTTP which is the protocol of the web, emitting JSON or ATOMPub means it is much easier to find a library that can connect to you on any language and platform.
• Scalability – stateless communication, replicated repository make for a good scalability potential.

do note that, as with any architecture/technology – a bad implementation can negate all the benefits

other REST goodness are things like the notion of the URI, idempotance of GET in  REST over HTTP etc.

The Bad

Some of the  problems of REST aren’t inherent problems of the architectural style but rather drawbacks of the REST over HTTP implementation. Most notable of these is what’s known as “lo-rest” (using just GET and POST) – While technically it might still be RESTful, to me a uniform interface with 2 verbs is too small to be really helpful (which indeed makes a lot of the implementation unRESTful see “The Ugly” below)

One problem which isn’t HTTP specific is handling REST- programming languages are not resource oriented so the handling code that maps URIs to tends to get messy. Actually Microsoft did a relatively good work with implementing Joe Gregorio’s idea of URI mapping which helps alleviate  some of the problem. On the other hand it is relatively hard to make the REST API hyper-text driven (Which is a constraints of REST)

Lastly and most importantly REST is not the answer to everything (see also another post I made on using REST along with other architectural styles) – e.g. most REST implementations I know do not support the notion of pub/sub (Roy did suggest a REST implementation called WAKA that enables this but most people never even heard of it). be weary of the “Hammer” syndrome, REST is a good tool for your toolset but it isn’t the only one. 

The Ugly

In my opinion there are 2 main ugly sides for REST. The first is Zealots. That isn’t something unique to REST any good technology/idea (Agile, TDD etc. ) gets its share of followers who think that <insert favorite idea> is the best thing since sliced bread and that everybody should do as they do or else.

The real ugliness comes from the misusers – There’s a lot of mis-understanding. The fact that REST over HTTP has become synonymous with REST leads people to think that HTTP is REST. I recently read a REST book review on Colin’s blog where “the author states that although hypermedia is important in REST it isn't covered in the book because WCF has poor support for it” i.e. a book on REST which ignores one of the important constraints of the style..

Other mis-uses include building an implementation that is GETsful  (ie. does everything with http GET) or doing plain RPC where the URI is the command, doing CRUD with HTTP verbs etc. etc.

The point is that REST seems simple but it isn’t – it requires a shift in thinking (e.g. identifying resources, externalizing the state transitions etc.). However, as noted above, done right it can be an important and useful tool in your toolset

 

If you read this blog regularily you've probably heard/read about the 8 fallacies of distributed computing once or twice ... you know the assumptions architects and designers tend to make when designing distributed systems which prove to be wrong down the road, causing pain and havoc in the  project.  (indeed my paper explaining them is the second most poplar download on my site with just about 50K downloads)
Originally drafted in 1994 by  Peter Deutsch (with one more added by James Gosling in 1997). These fallacies still hold true today. I still see designers make these same old mistakes in modern  SOAs, RESTful designs and whatnot - but that's not the reason for this post.
What I want to talk about is the second fallacy "Latency is zero".

The more I think about it the more I think this fallacy should be updated to "Latency is zero or constant" (or add another fallacy for "latency is constant" on its own).

What's the difference?

Well, "latency is zero" fallacy means treating remote "things" as if they are the same as local "things". We can't do that - we need to build the API of remote things to take the fact the information takes time to get there into account (e.g. chatty interfaces vs. chunky interfaces). You can see more on that in a post called "Why arbitrary tier-splitting is bad" i wrote about a year ago

The "latency is constant" fallacy means thinking that if we send several batches of "stuff" to a remote "thing", they may arrive late but at least they'll arrive in order. Or to move from "things" and "stuff" to more concrete terms if you send messages over a network from one service to another they won't necessarily arrive in order.

But wait isn't it only true for  asynchronous messages? if we make synchronous calls we don't really care about this, now do we? That's only true if you and the service you are consuming are alone in the world. In all other cases (i.e. most of the time) even if you make all your calls synchronous, you can't know what other messages (from other senders) will arrive in between your messages - and how it will affect its state.

Unreliable latency can also mean we'll retry a message because we think it is lost and find out that the reciever gets it multiple times later.

These are things you really have to take that into account when you make multiple related calls - like,say, in a saga. One thing you can do to help is make messages idempotent (which also helps with the "network is reliable" fallacy). You can also increase latency even more and order the messages something that happens, for example, when  streaming video or audio.

What you really need to think about is  ACID 2. No, I am not talking about the database transactions ACID but rather on another term I first saw in "Building on Quicksand" (paper (pdf)/ppt) by Pat Helland. In this paper Pat talks about some of the implications of unreliable conditions (such as inconstant latency, failure etc.) on fault tolerance. ACID 2 (which apparently was  coined by Shel Finkelstein) stands for Associative, Commutative, Idempotent and Distributed. i.e. messages can be processed at least once , anywhere (same machine or across several machines), in any order.

That's harsh but I think that If you are building distributed systems today (SOA or otherwise) you can't ignore it.

 

I am (finally) writing some new stuff for my SOA book - working on a few Anti-patterns

• The Knot - The distributed version of "big ball of mud" basically point to point integration
• NanoServices - designing/building fine grained services (methods != services)
• 3-tiered SOA - dressing up 3-tier architecture in SOA clothing (e.g. database as a service)
• Whitebox Services - exposing internal structure - comes in two flavors exposing technology and allowing access not through contracts
• Transactional Integration - inter-service transactions (use Sagas instead)
• RESToid- combing SOA and REST without understanding the full implication of either

I am going to publish one of them (probably the "knot") in a few days but I thought I might be able to get a little feedback before that. I chose to describe anti-patterns in the following format:

•  Context - Presenting the problem (probably through an example)
•  Consequences - Explaining what the problem is. i.e. what happens when the anti-pattern is prevalent
  
•  Causes - discussion on the forces that lead to the anti-pattern
  
•  Refactoring - The patterns (and/or other tips) that can be used to fix the design
  
•  Known Exceptions - Are there any contexts where using the anti-pattern is acceptable
  

I'd be happy to hear any comment you have on the anti-patterns listed above as well as comments on the structure for describing them

Thanks
Arnon


 

I wrote in the past how WCF defaults limit scalability but this thing (which had cost me two days of head scratching) is even worse.
Consider the following scenario:

 You have a WCF service/resource. when you get a message/request your codes needs to send another message to another service.

Sounds common enough now doesn't it? and it is - unless you happen to use a service with   WebHttpBinding (e.g. if you try to develop a RESTful WCF service or want to use POX services).  When you use WebHttpBinding and try to make a call within a call you are likely to find yourself starring at a ProtocolException with a 405 error - Method not allowed. Turns out WCF finds itself confused by the Operation Context (OperationalContextScope) of the incoming request so if you want things to work properly you need to create a new one for the request

var webBinding = new WebHttpBinding();
var channel = new ChannelFactory(webBinding, controlUri);
channel.Endpoint.Behaviors.Add(new WebHttpBehavior());
var proxy = channel.CreateChannel();
using( new OperationContextScope((IContextChannel) proxy))
{
     proxy.Dostuff()
}

I already spent the time figuring this bugger out- I hope this post will save you the trouble

 

Following my latest post on evolving the architecture Dru asked me for more details on our RESTful control channels.
For one you can take a look at slide 25 of my presentation on REST which talks about the Sessions resource. The session resource returns an AtomPub feed of the current active sessions and then if you follow a link to a session you get the current status, the URIs of the participating resources etc.
I guess the more interesting questions are (especially in the light of all the on going REST debate we now see)

1. Why rely on REST for the control channel
2. Why not use REST for the whole system

So, why is REST a good option for the control channel?


 

Microsoft recently released SP1 for .NET. While the SP brings some nice stuff it seems it also has some bugs and a few less than inspiring components
Another example for a less than stellar idea is the "ADO.NET data services" component. Before I go on to explain why I think that. I should probably mention that this isn't just a Microsoft thing as IBM also mentions similar ideas as part of their (broader and sometimes even worse) view of "Information as a Service"

So why is exposing the database through a web-service (RESTful or otherwise) is wrong? let me count the ways

1. It circumvents the whole idea about "Services" - there's no business logic
   
2. It makes for CRUD resources/services
   
3. It is exposing internal database structure or data rather than a thought-out contract
   
4. It encourages bypassing real services and going straight to their data
5. It creates a blob service (the data source)
6. It encourages minuscule demi-serices (the multiple "interfaces" of said blob) that disregard few of the fallacies of distributed computing
   
7. It is just client-server in sheep's clothing

When it comes for ADO.NET data services you can add a few other problems like

1. it isn't really RESTful - you can also "enhance" the services with operations like example 18 in "Using ADO.NET data services" : http://host/vdir/northwind.svc/CustomersByCity?city=London (though it does support caching and hypermedia )
   
2. Also it doesn't really externalize a state machine it externalizes a relational model
3. It is built on Entity Services.

 

Retrospectives, every "agile" team does retrospectives.What are retrospectives anyway?

A retrospective is a meeting where the team takes a look and inspect the past, in order to adapt and improve the future.

Agile or not, our team does a retrospective at the end of each iteration (every two weeks in our case). We try to look at what worked, what didn't , how we are meeting our goals etc, how is the product going etc.. These meetings provide a lot of value for steering us at the right direction.
On going retrospectives that look at the near past allows for suppleness and change adaptation and they are very powerful at that - However it is sometimes worthwhile to reflect over longer periods of time.

One area where longer perspective is important is the architecture of the project. Evolving an architecture you run the risk of accepting wrong decisions - mostly because architectural decisions have long term implications, while YAGNI, time constraints and life in general drive you toward short term gains.

Again, taking an example from my current project, working towards the first release, we took a few major decisions during the development e.g.

• federated resource management - Taking into consideration the fallacies of distributed computing we decided that we'd have local resource managers that will take care of resource utilization and allocation. The resource managers will have a hierarchy where they'd communicate with each other to gain the "bigger picture"
• Introduce Parallel Pipelines - handle image understanding by dividing the work between specialized components.
  
• RESTful control channel - to use a "lingua franca" between all component types so that we can easily integrate across platforms and languages
  
• local failure handling - resources and components handle failure by themselves
• Communication technology (WCF in our case) is isolated from the business logic by an Edge Component
  
• etc.

Once we finished delivering the first release. We took a few "days off" to consider what we've done thus far. updated our quality attribute list per our knowledge working with the system and looking at some customer scenarios. studies the things we liked/didn't like in the design and architecture of the working system. and revised a few of our decisions for instance

• We found that rushing to a working system we introduced some excess coupling to a specific technological solution (for video rendering). We initiated a few proof of concepts and found out how to both isolate the technology from the rest of the system as well as allow more technology choices.
• We found that the some of the data flows were not as clean as we thought they'd be - adding new features caused more resource interactions than we thought when we partitioned the resources. We redefined some of the resource roles to get less message clutter (and higher cohesion)
  
• The federated resource management works well, but introduce needless latency in session initiation. We now opted for introduce "Active services" which are more autonomous.
• Add a blogjecting Watchdog in addition to local failure handling to both increase the chances of failure identification and recovery as well as get a better picture in a centralized Service Monitor.
• RESTful control channel worked well and will continue for later release
• Some of the scale issues will be handled by introducing "Virtual Endpoints" while some would continue to use autonoumous endpoint creation and liveliness dissemination (hopefully learning from the mistakes of others)
• etc.
  

The result of these and the other decisions we've maid is a rework plan that will (hopefully anyway) make our overall solution better.
What we see is that we evolved our architecture as we went forward. While all the the decisions we made seemed right at the time we took them, only through reviewing them in a wider perspective (architecture retrospective) we identified the decisions that we need to change and the ones that we have to enhance. The insight you gain after working on a project for awhile are much better than the initial thoughts you have or the understanding you master in the initial interations.
I think it is essential to review the architecture once you've gained more experience with the realities of the system you write (vs. the precieved realities you have on the get go)

By the way if you work with a waterfall approach your situation is worse. Since in this case you take your decisions before you write any code so, you don't even have the benefit of POCs, and working code to enhance your insights


PS
if you have the MEAP version of SOA Patterns you can read more on the patterns I've mentioned here: Active service in chapter 2, blogjecting watchdog in chapter3, Service Monitor in chapter 4, Parallel Pipelines in chapter 3, Edge Component in chapter 2


 

Every Thursday we have this "happy hour", you know beers, snack etc. Every other week or so we also try to make it educational and after socializing for a while hear a presentation  or a webcast.

I used this week's slot to present the REST architectue style. I think the presentation turned out pretty well so I thought I'd share it online (note it is a 6M ppt)

 

If you recall what I currently work on is a type of a visual search engine. In a nutshell when we get a request (image) we allocate a bunch of algorithmic engines in a grid like manner to process the image  (e.g. try to perform OCR or whatever). As it happens, we are developing the different components using several different environments(*) - e.g. the control bits run on windows (.NET) and most algorithms run on Linux (mostly C++).
The need for easy cross-platform communications and extensibility, the resource nature of the solution and a few other tidbits led us to design our solution in a RESTful manner.

If you are a .NET developer/architect and wanted you may know that to implement a RESTful application in Windows Communication Foundation (WCF) you really have to jump through hoops.For instance  you have to go back to basics and use the HttpRequest and HttpResponse, handle the breakdown and parsing of URI hierarchies yourself not to mention  fight  with the  bindings .

Fortunetly this all changed with WCF 3.5. True, .Net doesn't have (to my knowledge anyway) something like RESTlets, but at least building REST on http is pretty straightforward.

Consider for example the following excerpt:

    [ServiceContract(Namespace = "http://paperlnx.Contracts/2007/12", Name = "ISessions")]
    public interface ISessions
    {
        [OperationContract]
        [WebGet(UriTemplate = "/Sessions/{sessionId}")]
        [ServiceKnownType(typeof(Atom10FeedFormatter))]
        SyndicationFeedFormatter ListSessionStatus(string sessionId);
	.
	.
	.

With these 6 lines of code you see the essence of  the .NET 3.5 REST goodies

1. Integrated support for HTTP verbs  - The sample above shows the support for GET. You can get the other verbs almost as easy with the WebInvoke Attribute. To do that simply specify the verb you want e.g.   [WebInvoke(Method = "PUT")] , [WebInvoke(Method="DELETE")] etc.


2. Support for URI templates -  In a way not too far from Joe Gregorio's IETF draft , WCF supports the notion of providing a way to describe families of URIs. This is done using the UriTemplate class. The WebGet and WebInvoke attributes also accept URI templates as variables and map the variable values (the curly brackets ones {}) to parameters of methods.
   
3. Support for standard  formats - you can use plain XML or you can choose to use RSS and ATOM syndication formats. In its most basic form you just create a syndicationfeed and format it to atom feed. Which is what we do for error messages:
   

   
           public static SyndicationFeedFormatter GenerateAtomError(string errormessage, string description,Uri location)
           {
               SyndicationFeed feed = new SyndicationFeed(errormessage, description, location);
               return new Atom10FeedFormatter(feed);
           }
   

   Naturally you can also add items and element extensions to all elements (e.g. the feed or items)
   

All in all, I am a happy camper :) After all, when you make an architectural decision, you always need to review it once you opted for an underlying technology. Even when a decision is right. The friction caused by a  technology which doesn't accommodate it well can both make your life miserable and make a good decision bad. .NET 3.5 with its newly added support for REST increases the architectural freedom and that's always a good thing

---

* Among other things, it helps us avoid the "Network is homogeneous" fallacy - but that's another story :)

 

This post is part of a series of posts trying to define SOA as an architectural style. In the previous post I talked about how SOA builds on the Client/Server architectural style. In this post I'll talk about how SOA builds on the architectural style of Layered System.

Layered System or Layered architectural style is one of the most basic and widely used architectural styles. Here is a definition of Layered architecture I posted in the past

The layered style is composed of layers (the components) which provides facilities and has a specific roles. The layers have communication paths / dependencies (the connectors).

In a layered style a layer has some limitations on how it can communicate with other layers (the constraints). Typically a layered is allowed to call only the layer below it and be called only by the layer above it (but there are variants e.g. a layer can call to any layer below it;  etc. - all is fine as long as the layers communication paths are limited and restricted by some rules)

SOA takes the strict layers definition and restricts the knowledge of one service only to the service interface/contract of the other services. This means the services cannot be aware or care about the internal structure of other services. Services don't mind the internal structure of other services. This helps with introducing the  "boundaries are explicit" tenet  (although, it build on more than just layering)

The layered nature of SOA means you can also add additional layers between the services. One very common example is adding a servicebus (e.g. using an ESB or tools like NServiceBus) other examples can include load balancers, firewalls (see Service Firewall pattern) etc. Naturally, When you add intermediary layers  services don't talk to each other directly rather accept the services (such routing , message persistence etc.)  from the intermediary layer.

It should be noted, that in the context of SOA the layers are, in most cases, actually tiers. The difference is that tiers provide (potential) physical separation where as layers provide logical separation . When a layer is actually a tier it has extensive implication on the level of trust between the tiers (see my post "Tier is a natural boundary" for more details)

The next post in the series will talk about the "Pipe and Filters" style  and SOA. This is the first place where the REST architectural style and SOA diverge.

 

In the previous post  on defining SOA I claimed that SOA is an architectural style building on 4 other architectural styles. The first one of these is Client/Server.
Describing client/server is easy - not because I am such a genius (far from it) but it has already been done before numerous times. Let's take a look at the definition from  Roy Fielding  in his famous dissertation (The link is to chapter 3, REST is defined in chapter 5 if you are interested)

The client-server style is the most frequently encountered of the architectural styles for network-based applications. A server component, offering a set of services, listens for requests upon those services. A client component, desiring that a service be performed, sends a request to the server via a connector. The server either rejects or performs the request and sends a response back to the client. A variety of client-server systems are surveyed by Sinha [123] and Umar [131].

Andrews [6] describes client-server components as follows: A client is a triggering process; a server is a reactive process. Clients make requests that trigger reactions from servers. Thus, a client initiates activity at times of its choosing; it often then delays until its request has been serviced. On the other hand, a server waits for requests to be made and then reacts to them. A server is usually a non-terminating process and often provides service to more than one client.

Separation of concerns is the principle behind the client-server constraints. A proper separation of functionality should simplify the server component in order to improve scalability. This simplification usually takes the form of moving all of the user interface functionality into the client component. The separation also allows the two types of components to evolve independently, provided that the interface doesn't change.

The basic form of client-server does not constrain how application state is partitioned between client and server components. It is often referred to by the mechanisms used for the connector implementation, such as remote procedure call [23] or message-oriented middleware [131].

SOA takes from the Client/Server style the two roles - ie. in each interaction one party is the client (what I call service consumer) and the other is the server (service) which  handles the request coming from the client*. Unlike traditional client/server, the roles are held only for a particular set of interactions - a given interface that the service exposes. In another set of interactions the roles can be reversed and a component that once was a server can now act as a client even working with the very same component that was previously its client.

Like REST, SOA takes the constraint of separation of concerns which allow the service and its service consumers to evolve independently (as long as the interface is kept).
In order to support this, services should takes care of all its internal state without exposing its internal state or its internal structures outside of the service. This also allows the service to scale behind the interface but for that we also need constraints and capabilities from the next architectural style layered system, which I'll discuss in the next installment on this subject.

---

* You can compose SOA with other architectural styles to get different behaviors. E.g. compose SOA and  EDA and you can have the service also push data.This t isn't, however,  something SOA ,manifest in its basic form

 

A few weeks ago I posted a reaction to a post by Pete Lacey that asked what is SOA. In a comment to my post Pete said that my definition isn't good since

"...even according to your definition, an architectural style contains constraints, and to date neither SOA nor web services have been shown to exhibit any constraints"

The idea behind this series of posts is to try to take a little more formal view at what I think SOA is. It is based on my thinking for the past few weeks but it is also still a work in progress (so any comments are welcome)

The way I see it SOA is an architectural style which is derived from the following architectural styles:

1. Client/Server
2. Layered System
3. Pipe and Filters
4. Distributed Agents
   

Note that if you add to the above statelessness, uniformed pipe and filters and a cache you can get a RESTful SOA. This is not REST as REST itself does not require distributed agent or even pipes and filters (but it does build on client/server and layered system). In other words not all RESTful systems are SOA, you can build SOAs which are not RESTful and you can build RESTful SOAs.

The main components of SOA are Service,Message, Contracts and Consumers. Policies also exists but now I tend to think they are optional. The four architectural styles mentioned above affect the definitions of the different components and the way they interact together

In the following posts on this subject I'll first take a look at each of the contributing architectural styles and how they affect SOA and later try to provide a definition that builds on them

 

Pete Lacey has a post called "What is SOA?" where he defines SOA as follows:

"

• Network Oriented Computing (NOC): An approach to computing that makes business logic available over the network in a standardized and interoperable manner.
  • Service Oriented Architecture (SOA): A technical approach to NOC that has a non-uniform service interface as its principle abstraction. Today, SOAP/WS-* is the chief implementation approach.
  • Resource Oriented Architecture (ROA): A technical approach to NOC that has an addressable, stateful resource as its principle abstraction. Today, REST/HTTP is the chief implementation approach.
• Business Service Architecture (BSA): An unnecessary term (also not an architecture) that tries to make the obvious something special. Aka, business analysis. Aka, requirements gathering"

I am sorry but I beg to defer.

The first thing to note (again) is the architecture vs. architecture style differentiation I mentioned in a previous post (You can see a similar definition by Stuart Charlton) Here is a quick reminder :

Software architecture is the collection of the fundamental decisions about a software product/solution designed to meet the project's quality attribute requirements. The architecture includes the main components, their main attributes, and their collaboration (i.e. interactions and behavior) to meet the quality attributes. Architecture can and usually should be expressed in several levels of abstraction (depending on the project's size).

An Architectural style is a blue print that can be used when you desing an architecture. An architectural style defines some of the components and thier attributes as weel as place constraints on how they can interact.

My claim is that SOA is an architectural style for distributed computing which puts extra emphasis on the interface (and hence gets the easier interoperability). Ok, if SOA is indeed an architectural style, we should be able to define it as a set of components, interactions and attributes. Well, I already did that a while ago (in a paper called "What is SOA anyway?"). And while it may not be perfect, I think it is a reasonable definition all the same:

"SOA is an architectural style for building systems based on interacting coarse grained autonomous components called services. Each service expose processes and behavior through contracts, which are composed of messages at discoverable addresses called endpoints. Services’ behavior is governed by policies which can be set externally to the service itself. "

You can see the above mentioned paper for a little more detail on each of the components.

ROA, in my opinion, is just a re-branding of REST so that it would be easier to discuss it as an architectural style and not connect it to the HTTP implementation - which is what  a lot of REST proponents are doing.

By the way, as I pointed out before, there are a few other important architectural styles that are related to distributed systems like Event driven architecture, Spaced based architecture, peer-to-peer etc.

As for "Business Service Architecture" - I personally like to think about that as "SOA initiative" as in the strategic decision to try to implement an SOA in an organization while trying to achieve the more nebulous traits like business and IT alignment etc. (which is why it is nether architecture nor architecture style)

 

In a recent post Steve Vinoski said:

"Frankly, if I were an enterprise architect today, and I were genuinely concerned about development costs, agility, and extensibility, I’d be looking to solve everything I possibly could with dynamic languages and REST, and specifically the HTTP variety of REST. I’d avoid ESBs and the typical enterprise middleware frameworks unless I had a problem that really required them (see below). I’d also try to totally avoid SOAP and WS-*."

It is easy to dismiss this as just another yahoo who goes against conventional wisdom until you remember that Steve spent more than a decade working in Iona in leading roles like Chief Engineer of product innovations and helped develop some of the middleware standards for OMG and W3C.

Well, I guess that's becoming an epidemic  now :)  just recently we had Michael Stonebreaker talking about the RDBMS demise, Pat Helland talking about life beyond distributed transactions.  and now Steve on ESBs.

That trend aside, I think Steve is doing throwing the baby out with the bath water. The dream of a single infrastructure for an enterprise is ludicrous enough (Remeber Peter Deutsch and the "The network is homogeneous" fallacy). but if you drop the "E" from the ESB moniker you get a valuable middleware which is very usable in many situations and not just legacy system integration. For instance one thing that is missing form "HTTP variety of REST" implementation is reliable messaging. location transparency is  harder to solve with HTTP etc.

Another problem I have with the current approach of Steve is that he is replacing one dogma (EBSs are good) with another (ESBs are bad use Ruby, REST) - this is not a healthy approach. The solution should match the problem, that's probably the primary reason why we need architects after all

 

It seems that even the smartest people can get the difference between architcture, architecture styles and technology wrong
For instance Anne Thomas Manes points out the Roy Fielding makes this mistake in his REST and Relaxation presentation by mixing an architectural style with technology:

 "Roy is equating SOA with web services. Although a lot of folks use web services to implement services, that's simply an implementation decision"

But then procede to make the exact same mistake 

"So when watching Roy's presentation, replace the term "SOA" with "WS-*", and the discussion will make a lot more sense."

REST is an architectural style you can implement it with WS-* which is a technology. It is not the most natural way to use WS-* standards but it is doable.

Looking at the same context (i.e. Roy Fielding's presenation) Steve Jones makes a similar mistake confusing Architecture and Architecture style.

My definition for software architecture is

Software architecture is the collection of the fundamental decisions about a software product/solution designed to meet the project's quality attribute requirements. The architecture includes the main components, their main attributes, and their collaboration (i.e. interactions and behavior) to meet the quality attributes. Architecture can and usually should be expressed in several levels of abstraction (depending on the project's size).

An Architectural style is a blue print that can be used when you desing an architecture. An architectural style defines some of the components and thier attributes as weel as place constraints on how they can interact.

For instance, the REST constraints (taken from Anne's post mentioned above) are:

"Uniform Interface:

• Resources are identified by only one resource identifier mechanism
• Access methods (actions) mean the same for all resources (universal semantics)
• Manipulation of resources occurs through the exchange of representations
• Actions and representations are exchanged in self-describing messages

Hypertext as the engine of state:

• Each response contains a partial representation of server-side state
• Some representations contain directions on how to transition to the next state
• Each steady-state (page) embodies the current application state"

Architecutre Styles can be combined to create new architectural styles. Roy Fielding demonstrates this in his famous dissertation  where he demonstrate how REST is a composition of several styles such as  Client/Server, Layered system, Stateless etc. As another example (which a lesser degree of precision) I take about enhacing SOA with EDA in "bridging the gap between BI and SOA"

The last piece of the puzzle is technology. Technology (in the software context) are set of tools provided by a vendor to enable and support building software solutions. As I've said here numerous times, technologies has their own internal architectures (as they are software solutions themselves) which is why different technologies support different architectural styles and why the alignment of the technology with the architecture chosen for your solution is important.

Yes this post is all about semantics - but clear meanings are important to prevent confusion, at least in my opinion anyway

 

Another REST related post - this time I want to share a couple of observations I had after reading (Roy's presentation from RailsConf 2 days ago via Pete Lacey) and listening to Roy T. Fielding's presentations.


The first point has to do with a question which is sometimes raised whether you can do REST without HTTP. i.e. can you have a RESTful architecture  if you don't use the http protocol and further more not using the http verbs (GET/PUT/HEADER etc.) or  as the unifier interface. I talked about it a while ago and I think you can. listening to Roy's talk  it seems that, at least in http architect's opinion the answer is yes as well.

Another point that occurred to me, watching Roy's talk, which is related to the "REST magic" post I wrote a little over a week ago. The use of a uniform interface is tauted by REST proponents (and Roy himself) as coupling reducing formula. After all if you use a uniform interface you are not coupled to the particular semantics of any resource/service you already know the capabilities (actually the maximal capabilities) they offer. What ensues is that instead of using a lot of verbs (ReserveRoom, UpdateOrder etc.) you use a lot of nouns (/rooms/, /orders/order1 etc.)

This works extremely well on the "human" web where my browser can navigate to any-ol'-site without any prior knowledge of what's the site about. When I navigate to Amazon I can buy stuff, when I navigate to New York Times I can read stuff etc. The problem here is the browser is really dumb about what's going on. I, as a human using the browser, understand the context from the content (well, most of the time anyway;) ) so the browser can remain decoupled.  However when you translate it to the "programmable" web you usually don't have some mighty AI engine examining the response to understand the context - instead what you do is trade the verb coupling, which with WS-* web-services would be defined in a contract, you are now coupled to the nouns ( this is not to say the nouns aren't discoverable - since they are due to the hypertext or document orientation communication REST encourages). The end result is pretty similar to what you get when you use verb based contracts your software still needs to understand (where "understand" means some level of coupling) what it is doing with the "other" services. not to mention that you still need to understand the content of the message (sorry- response) to do anything useful with it.

In any event, while loose coupling is very desirable, we also need to remember that the only way to truly achieve complete decoupling is to not connect components. So some coupling is always needed if we want to produce meaningful systems.

What do you think?

 

If there's one reason to go to ApacheCon 07 in Atlanta, then it's probably Roy T. Fielding's "a little REST and Relaxation"

Here is the abstract:

"Representational State Transfer (REST) is an architectural style that I developed while improving the core Web protocols (URI, HTTP, and HTML) and leading them through the IETF standardization process. I later described REST as the primary example in my dissertation. Since then, REST has been used (and sometimes abused) by many people throughout the world as a source of guidance for Web application design. But is the REST that we hear about today the same as what I defined in my dissertation, or has it taken on the baggage that comes with an industry buzzword? This talk will provide a real introduction to REST and the design goals behind its evolution as the Web's arhitectural style. This is not about XML-over-HTTP as an alternative to SOAP, nor about "resource-oriented" frameworks that help simplify CRUD operations, but rather about the design goals and trade-offs that influence the development of network-based applications. I will also describe what happens when we relax some of the REST constraints, and how such relaxation is impacting the design of the waka protocol as a replacement for HTTP."

Now all I have to do is find an excuse for my boss... :)

There isn't a whole lot of information available  on WAKA  (that replacement for HTTP Roy mentions in the end of the abstract). Belwo are a few links I managed to find

• Roy's 2002 presentation on REST and WAKA (ppt)
• A blog post from Feb. 2007 by Robert O'Brien that talks a little about the new semantics offered by WAKA (such as MONITOR to get notifications if the resource changes)
• A podcast by Jon Udell interviewing Roy T. Fielding (from Aug 2006)
• A blog post by Mark Nottingham briefly contrasting WS-Transfer and WAKA (March 2006)

And there's a few others but not as interesting (to me anyway). Well, as we see this WAKA thing is in the works for a long time now. Also replacing something as ubiquiteus as HTTP is not a small feat. But I guess if anyone can pull this off it would be Roy... As always, only time will tell

Edited (18/9): it seems that a recent version of Roy Fielding’s presentation  is available online on parleys.com (via Stefan Tilkov)

 

From time to time I read about the magic that is RESTful services and how they solve everything and anything like scalability, idempotency, simplicity etc. for instance in "RESTful Web Services" by Sam Ruby and Leonard Richardson they say
 "PUT and DELETE operations are idempotent. if you DELETE a resource, it's gone. If you DELETE it again, it's still gone..." (p.103)
or
"the safe methods, GET and HEAD, are automatically idempotent as well" (p.219)

Another example comes from Anne Thomas Manes who said

"The REST architectural style defines a number of basic rules (constraints), and if you adhere to these rules, your applications will exhibit a number of desirable characteristics, such as simplicity, scalability, performance, evolvability, visibility, portability, and reliability.

The basic rules are:

• Everything that's interesting is named via a URI and becomes an addressable resource
  
• Every resource exposes a uniform interface (e.g., GET, PUT, POST, DELETE)
  
• You interact with the resource by exchanging representations of the resource's state using the standard methods in the uniform interface

"

I think such claims  are plainly wrong and misleading.
 
Don't get me wrong, I like the REST approach, since it encourages better service design - e.g. document oriented message exchange vs. the RPC like message exchange which the so called "WS-death-*s" (or actually the tools that support them) encourages.

It also encourages the above mentioned traits - however that's exactly the  point - REST encourages this thinking not solves scalability or other problems out of the box- you still need to design your services properly.

For instance if you follow Anne's rules you can still end up with a service which is stateful, that performs heavy distributed transactions against multiple databases and systems - i.e. a service that is neither simple, scalable or perfromant

DELETE will only be idempotent if the resource is idempotent (e.g. a specific version of a resource)  or the message is idempotent (e.g. requesting a deletion of a specific version) if you are deleting the "recent version" then it might have been recreated between your calls you are now deleting something completely different. heck, even a GET (read) message with a single reader can be made to be non-idempotent  if you decide to code something that alters the state of a resource significantly whenever it is read. When you have multiple readers and writers GET will not be idempotent "automatically" as two consecutive reads can give you two different representations as the resource might have changed (again unless the resources are idempotent)

REST is not different from other styles in this respect - for instance you can do Object orientation in C but working in an OO language encourages object orientation (the opposite is also true - using an Object Oriented language does not guarantee that you get an Object Oriented design)

At the end of the day, architects should still think about the design if they want to ensure the results matches the quality attributes they want to achieve - some environments/styles/tools will make some quality attributes more easy to achieve but nothing will solve the problems for you.