You raise an event when something interesting happens to you, you think it is important, but you don't care enough to know who is interested. you are even less interested in to personally going and to each an every interested party and letting them know. So - instead, you raise and event, and let the poor buggers take care of any implications by themselves. We raise the event "now" when the change happened - it is only important now anyway...


Looking from the "poor buggers" -the event consumer point of view things are more complicated. There are events which are cyclic in nature like stock price updates, the blips from a sonar etc. if you missed one, then it isn't really important you'd get the right information in the next update (actually, that isn't entirely true - see later in this post). Then, there are the events which only occur once. sometime it isn't important for you to listen to them if you are not up and running in the same time. Other times you can't afford to lose an event for instance if your ordering service (or component for that matter) communicates with the invoicing one using events you don't want to miss the event of a new order else you would loose money.

This basically means that the event producer and the event consumer are coupled in time - one way to solve it is to make sure both of these services are available at the same time i.e. if the invoices crashed, then processing orders should be suspended (note that this doesn't mean that you don't accept orders just that you don't process them).
Ok - maybe we can just raise the event "transactionally" - this would probably work, but we need to remember that the event producer doesn't really care about the event consumers, why would it want to fail because of them?!
Maybe a better way would be to "raise" the event over some reliable transport  - this has a few problems. one is that we've passed the problem to the connection between the event producer and the transport. It might be acceptable to have a transaction between the event producer and the transport. However, as I've already said the producer doesn't care much about the consumers..
We can have persistent subscriptions for existing consumers to prevent events from getting lost which make both creates a er minor problem that new consumers can't see past events but also has the risk of existing subscribers disappearing and their queue can then grow endlessly (or until an administrator would remove the subscription)

Ok, let's try to look at the problem from a different angle. looking at the events, what we can really see is that an event has a time-to-live (TTL) as far as the event consumer is concerned. For instance in the case of the cyclic events the TTL is the interval until the next  event. Actually, even with cyclic events the TTL might be larger - if we are also interested in analyzing trends or  ab normal occurrences (which is why I said it isn't entirely true we don't care about old events). In case of one-time events the TTL might be indefinite or maybe even then it might be some definite value (one day, week, year etc.). Since we can't know about the TTL of consumers it can be a good idea to make past events available somehow.

Thus, when you design an event centric architecture like  EDA (whether on top of SOA or not)  it is important to think about event consumers - we don't want to think about  specific  consumers since it negates the benefits of thinking in events, but I would say that you want to think about event consumers in general, after all your component is also an event consumer (do unto others as you would have them do unto you)

One option, which I already talked about, is to make past events available as a feed. Event consumers can then come at their own leisure and consume past event  (this can be in addition  to  to raising the events in real-time). This provides a partial solution as the maximal TTL is determined by the event producer (after which the event is deleted from the feed). This may be acceptable but you must be aware of that.
The other option is to to log all the events and provide an API to retrieve past events. In a sense the max TTL is still at the hands of the event producer only if you use a database it would probably be a large time compared with a feed. Alternatively the events can be logged on by a central "always present" event aggregator (in a manner similar to the aggregated reporting pattern I described for SOA).

To sum all this - events they seem only to matter in the instance in time they are created, we are used to that thinking from building OO systems where all the components are co-located in the same address-space and time (even there I can think of scenarios where we would want past events) - in a distributed world events need to have a TTL, the TTLs can vary and are determined by the events consumers. Lastly, as I demonstrated in the paragraph above, there are several strategies we can use to help solve the event TTL dilemma (and there are probably a few others).

 

Few months ago I wrote here about solving the mismatch between Service Oriented Architecture (SOA)  and Business Intelligence (BI) (see papers and articles section). Recently I got the following question from Ben:

One major question I have is around large data sets. As an experienced BI/DW architect and developer I have worked on a number of large scale data warehouses. Retrieving large data sets (i.e. millions of records) doesn't seem to fit well into SOA. As you state in your article, we could have another point-to-point interface, where the service which houses data we need gets a request and writes out a batch file (xml or plain ascii text). Then using typical ETL, we grab the file and load it. The underlying source system (service) can use optimization in generating a large data set (vs. record by record) and
the data warehouse can correspondingly load in bulk.

Like most architectural questions - the answer is "it depends"
For instance, if you do a run-of-the-mill ETL as a on-time setup then it is just that- a one time setup and I, personally, don't see any contradiction between SOA goals or tenets and that.

I do think that iit is better to enhance SOA with EDA interactions to provide a long term solution to the BI problem. You can also have a dedicated component that aggregated the information that flows in in these events and builds batch files that are suited for the ETL you've used during the setup phase (mentioned above).
It is true though that moving an SOA which is already in-place to EDA is not a small feat, but adding EDA layers does not have to mean that the old interfaces go away - especially not immediately (remember to treat services as products)

If you have a business that generated millions of records on a daily basis - then the situation is more complicated. Now you have to think about the trade-offs between "compromising" SOA and adding a dedicated interface (or a backdoor to the database) for the ETL vs. the implications of performance, bandwidth, transition costs, ROI  etc. of pushing that information with EDA.
 I, personally believe in pragmatism and the "no-silver-bullet" approach so I can't say that EDA is always the best solution (As an aside, this is part of the reason I write my book as patterns not as "best-practices guidance"). You may find that ETL is the best trade off in your situation. Yes I know that it isn't a definitive answer - but real life is (usually) a little more complicated than black and white solutions. As architects we need to find the best trade off for the situation at hand.

 

I've been working in Rafael for a total of 5 years now and I guess my tenure here is nearing its end.  The latest company-wide reorg has brought with it the end of the biometric product line (or maybe just its evolution into something I don't really like - it isn't completely clear yet, but since I don't like both option it doesn't rally matters).

I am checking several possibilities of joining one of the consulting companies here in Israel - none of the options has fully matured as of yet but I believe that in the next month or so something will be finalized - meanwhile I am also open for other options (my CV is on the about page - hint, hint :) ).

In any event, between the book, DDJ, InfoQ and looking for a new job (not to mention my family) I sure have have my hands full - I guess now I understand why they say "may you live in interesting times" is an old Chinese curse :)

 

I thought I has this  RESTful web services thing figured out, but following one of the threads on the Yahoo group on Service-Oriented-Architecture I came to the conclusion that maybe I don't.

Steve Jones tried to see if he understands REST by giving an example and that example was corrected by Anne Thomas Manes (who is a research director with the Burton Group which recently stated that the future of SOA is REST).
Here are the examples from the above mentioned thread:

POST http://example.org/customer
HTTP message body contains a representation of "anne"
server creates a subordinate resource called http://example.org/customer/anne

GET http://example.org/customer/anne
returns a representation of "anne"

GET http://example.org/customer/personByName?name=anne
returns a representation of "anne"
or perhaps returns the URI of the "anne" resource
or perhaps returns a list of URIs of all people named "anne"
might also be specified more simply as
GET http://example.org/customer?name=anne

GET http://example.org/customer/personByAge?age=27
returns a list of URIs of people whose age is 27
or perhaps returns a collection of representations of all people aged 27
might also be specified more simply as
GET http://example.org/customer?age=27

PUT http://example.org/customer/anne
HTTP message body contains a representation of "anne"
either creates a new resource called "anne" (if none exists)
or replaces the existing "anne" resource

PUT http://example.org/company/newco
HTTP message body contains a representation of "newco"
either creates a new resource called "newco" (if none exists)
or replaces the existing "newco" resource

If you prefer the server to assign the URI you would instead say

POST http://example.org/company
HTTP message body contains a representation of "newco"
server creates a subordinate resource called http://example.org/company/newco

POST http://example.org/customer/anne?addCompany=http://example.org/company/newco
this would append the newco company reference to the "anne" resource

You can see another example for what I am talking about here on Jon Udell's blog giving an example from RESTful Web Services, by Leonard Richardson and Sam Rubycovering  of doing a transaction in RESTful style

If all these are indeed "legal" or "correct" RESTful interactions I have 2 observations to make
First, I guess Pat Helland is right when he said "Every noun can be verbed" since I don't see the real difference between having a contract with a PersonsByAge request which returns a document* of Persons and a REST request like " GET http://example.org/customer/personByAge?age=27" or even " GET http://example.org/customer?age=27".

The second observation has to do with the so called "uniform interface". I would argue that the resources and their attributes (age=27, name="anne") are the interface. the POST, GET etc. uniform interface does not mean much more than the "uniform" SEND, BROADCAST  interface of messaging.
Further more if resources and their attributes are indeed "the interface" - than not only does REST not have a uniform contract - it actually has a dynamic one which changes in run-time as new resources are created - such as the "POST http://example.org/company"  which creates a new resource "http://example.org/company/newco" in the example above

---

* I think it is very important for SOA to have document oriented messages and not RPC one I\ll blog in a separate post about the differences. for now it is suffice to say that the REST hypermedia notion of returning the URIs of all the relevant persons should also be present (one way or another) in a good document oriented message even if you are using WS-* or plain messaging as transport
 

In addition to the drafts of selected patterns I publish on my site, you can now purchase my book via the Manning Early Access Program (MEAP).
MEAP means you can get chapter drafts as I write them and the complete book when its done (ebook or printed). Here is Manning's explanation:

"Buy now through MEAP (Manning Early Access Program) and get early access to the book, chapter by chapter, as soon as they become available. You choose the format - PDF or ThoutReader - or both. By subscribing to MEAP chapters, you get an opportunity to participate in the most sensitive, final piece of the publishing cycle by offering feedback to the author. Reader feedback to the author is welcome in the Author Online forum. As new chapters are released, announcements are made in the MEAP Announcement Forum. After all chapters are released, you will be able to download the complete edited ebook. If you order the print edition, we will ship it to you upon release, direct from the bindery, weeks before it is widely available elsewhere.

By the way, this is probably also a good time to mention that I'll be speaking about quite a few of the patterns in Architecture & Design World 2007 which will take place this July.

There is still a lot of work, but I already like to thank all the people in manning that helped me get this far. especially to Cynthia Kane my editor (hey, maybe now she'll give me more slack :) )
Ok, 'nuff blubbering, back to completing chapter 5...

 

Agile and documentation? what gives? First things first documentation is not something that is prohibited by Agile Manifesto. Working software is definitely preferred over "comprehensive documentation" but there can also be some value in documentation

The first question is why would we want to document anything if we have a working software. I think there are several stakeholders like project newcomers, maintainers etc. who will be interested in something that will let them get up to speed and provide them an overview of what's going on before they delve into code. You can read more on that in  a post I wrote almost a year ago called "Who needs a software architecture document" but in essence the main motivation for documentation is that assuming that the software is successful it will outlive the team - i.e. the people that built the software will not be the ones that will have to develop, maintain and support it for most of the software's life.

If we agree that we need a software architecture document the question is what to document and when.

There are two main "things" you can document in regard to architecture, the first is the obvious one: the architecture itself. In my experience the most value can be derived from documenting recipes i.e. how  to do stuff that is common in the architecture. These recipes can be a short description of the context and then a
pointer to a test (or tests) and an implementation that exercises this. You can think of the recipes as a type of a tutorial to the architecture.

Other documentation worthy elements related to the architecture are an overview and technology mapping (including what a developer needs to install to start working). The overview allows a newcomer to understand where to find what, the technology mapping allows for understanding what she needs to learn and install to be productive. Note that to be useful the overview should be at a higher level of abstraction than the code - otherwise you run the risk of missing the forest for the trees or at least not saving any time.

It is obvious that documenting any of this before your architecture is stable more or less is useless, as a rule of thumb I would say this can be around the 5th-6th iteration - assuming the team has to grow during the project. If the team stays stable for the duration of the project, this documentation can take place towards the end of the project (though I would probably add recipes to a wiki or something similar during the project as development patterns emerge).

The second "thing" you can document in regard to architecture are the decisions you decided against, in my opinion this is more important than all of the other items mentioned above together.  The reason for that, while it might take a while to understand a well written software and infer its architecture it can be done, but it is virtually impossible to understand the options that were disqualified from looking at the chosen solution.
Understanding the options that weren't used can save time for the person reading that description. both in understanding why things are the way they are. Furthermore it can save time trying things that didn't work or provide clues to options when the circumstances change (since, as we all know, requirements change..)
The best time to document decisions you decided not to take is when you opt not to use them - this is when you remember best "why". for instance, in my current project we use x.509 certificates to authenticate clients and we use decided to use Kerberos tickets to authenticate components within the service. There's a reason for making that translation*,   there's also a reason for making the transition by replacing the client certificate with the edge component's credentials instead of mapping the client's certificate to a Kerberos ticket using an Identity provider*. we had two developers spike different options for two weeks until we came to the current solution instead of the more obvious choice of passing the x.509 certificate from the edge into the service and using the client's credentials. This question is likely to come up when/if someone else would take over the project, when the technology will be updated etc. Again, if we know why we didn't make that choice we can better decide what to do when the circumstances change.

To sum up, there are few architecture related issues that are worth documenting even in agile projects. some of them can be postponed some of them are worth documenting a little earlier. In any event it is better to document after the fact and to keep the documentation light.

---

* It all has to do to to limitations of WCF in regard to  the transports we use (HTTP, MSMQ and TCP) and the request/reaction pattern (asynchronous communication) we use.

 

While I am on the topic of REST, it is probably a good time to comment on my (first) post on InfoQ "Debate: Does REST need a Description Language"

Personally, I think there's merit in Services publishing their message structures in a machine readable format. When a Service has a machine readable contact. generated stubs allows you to make the interaction with less bugs vs. hand crafted interactions. It also makes it easier to test the service itself.

I do agree with Stefan's views on runtime interface dependency where he said that if a service consumer needs just 20% of the information in a service it shouldn't be forced to deserialize (i.e. know or care about) the whole message.However, I think this is a weakness of tooling not the concept. What if you had a tool that reads the machine readable contract, allow you to pick the 20% you need and generate for you a stub that ignores all the other 80% and "hand pick" the 20% you need. This is what you would personally do yourself anyway, and since the code is generated from the Service's definition it would be more resilient and error-free This is effectively designing a personalized mini-contract from the published general one. It does mean that when that 20% changes you will be affected, but this is something you'd have anyway.

I also agree that that the WS-* standards and resulting contract are (and getting more) complicated. Much of this can probably be attributed to the "design by committee" effect. However, there are also some real challenged that the SOA and ROA architectural styles do not address and we still need to solve those. Trying to solve these challenges is, by the way, what prompted me to write my SOA patterns book...

 

DevHawk (Harry Pierson) raised today a question I was toying with myself for a while now - if REST is an architectural style can it exist without the specific technologies that define it today. Or as Harry put it :

1. REST is a an "architectural style for distributed hypermedia systems".
2. REST "has been used to guide the design and development" of HTTP and URI.
3. Therefore REST as an architectural style is independent of HTTP and URI.
4. Yet, I get the feeling that the REST community would consider a solution that uses the REST architectural style but not HTTP and/or URI as "not RESTful".

What I had in mind for example is to use messaging where the equivalent of the URI would be a topic hierarchy.
Topic hierarchy allows you to have a unique "URI" for each resource.
The next thing we need to take care of are the PUT, GET, POST and DELETE verbs - we can do that by making the verbs part of the message headers.
As an aside I'll also say that  if we try to think about it as an architectural constraint then we don't necessarily have to use these verbs, a more general rule would say that the verbs are uniform and well known rather than specific ones
The rest (no pun intended) of the concerns, like specifying related states etc. can be dealt with making conventions on the message formats
Is that still REST?! I wonder...

In any event, what worries me the most in regard to REST is the religious manner that some people seem to treat it. By the way that is the same phenomena we see with some of the Agile folks. As for me? - Well, I don't really care if I fit that label or the other. I am just payed to deliver working  and viable software :), but hey, that's another discussion.

 

Yesterday I attended an SOA governance presentation by Brent Carlson. The presentation was basically an updated version of an article he authored in 2006 "SOA Governance Best Practices - Architectural, Organizational and SDLC implications" As a  tool vendor Brent has a lot of focus on the governance processes which I don't completely agree with (I prefer Jim Coplien's organizational patterns approach - see my post from last week). I also think the reuse figures he cites (registration required) are a little optimistic common place for what I consider the right granularity for services.
He also made a few points I  strongly agree with

• Brent talked about difference between the needs of run-time service repository (e.g. UDDI or an ESB) and a development time one. You need to address the services and their interactions during the development and you need to do that in a way that would be easy for the development teams. For example, one thing you want to log is usage, who is using the services since that will let you perform impact analysis when you have to make a change
  
• Building an SOA for an organization is an iterative process not a "big-bang" effort. This means you can't do just top-down design. you need to be pragmatic and also roll out working services.

The reason for this post however is the insight Brent gave regarding treating services as products rather than applications

Treating Services as products is  important because even if you don't believe that the SOA initiative should be  an iterative process once the move is  finished you would have quite a few services deployed in your organization. These services would integrate and interact with other services - some of which outside of your organization. You would also want to capitalize on flexibility claim that SOA makes and adapt your services to the changing business needs.
The challenges you face regarding updating and upgrading  functionality , anticipating consumer's needs, allowing consumers to get used to changes etc. are exactly the challenges product management techniques and principles come to answer

Treating services as products means a lot of things. let's look at a few examples: For one, it means predictable release cycles services like products get updated over time you want service users to be able to cope with this changes. Predictable release cycles means they can get organized in advance. Another aspect is the emphasis on backward compatibility e.g. orderly deprecation of features and version management,.One other thing  is introducing a "product manager". someone whose responsibility is to interact with customers, and potential customers, understand their needs and build a release road map for the services.

You might be used to doing some of that with applications   but thinking about services an products makes all this more explicit and that in itself is also important.