One of the most interesting presentations in Architecture & Design
world was the eBay Architecture presentation by Randy Shoup and Dan Pritchett.
The presentation was only one hour long, so Randy and Dan didn't cover
all the topics in the slide. Here are some of the insight I took from
this presentation. Architecture evolution - eBay actually went through several architecture revolutions.
Their initial architecture cannot even begin to scale to their current
loads. It was, however, a very good fit for their initial quality
attributes - specifically, the emphasis on time to market and costs.
This shows the importance of balancing quality attributes. Sure an
architectural change is painful but if they'd future proofed too much I
doubt they would ever get something working. V2 demonstrated
that traditional 3-tier architecture would only scale so far. It was
nice to see how it evolved though. Also with the move from version 2.4
to 2.5 and later to 3 we see eBay learning about CAP -
the hard way. In its final (current) incarnation eBay's data
architecture prefers partitioning and availability over consistency.
This doesn't mean they forgo consistency altogether - just that they
trade the comfort zone of ACID transactions with the BASE approach.
Where BASE - stands for Basically Available, Scalable/Soft state &
Eventually Consistent. . eBay partitions thier data in two levels
one is a SOA like division by business areas (users, items etc.) and
the second level is an horizental partitioning based on access
paths.This BASE approach to data was dubbed by Dathan Pattishall (from
Flicker and Friendster) as sharding (via HighScalability).
This approach means things like high partitioning, no distributed
transactions (also see below), denormalization etc. (you might also
want to read the item I wrote on denormalizaiton in InfoQ yesterday). The more major implication here is that when it comes to internet scale, the database looses its importnace - or as Bill de Hora nicely puts it: The
use of RDBMSes as data backbones have to be rethought under these
volumes; as a result system designs and programming toolchains will be
altered. When the likes of Adam Bosworth, Mike Stonebraker, Pat Helland and Werner Vogels are saying as much, it behooves us to listen. As
I said the data architecture of eBay is SOAish - partitioned their
components and data along business lines, and they apply many of SOA
principles. They don't however unite data and components to create a
service and they don't (seem) to have the same contract boundaries
that SOA promotes (Randy told me that they are currently contemplating
SOA). Returning to the eBay do not use transactions.
"no transactions" which seems very controversial - but if we just
consider some of the points I made on transactions between services in
previous posts - it is the only logical way to ensure scaling. By the
way, as can be expected they do use transactions - when they are local
e.g. if the users table is spread over a couple of table both will be
updated together). The application layers also follow the
segmentation by business areas. eBay cacse metadata/immutable data as
much as possible. keep the application stateless (i.e. state comes from
client/db) e.g. they don't use sessions. The DAL virtualized the
horizontal partitioning mentioned above for the rest of the code. It
was also interesting to that eBay developed its own messaging
infrastructure - though Randy and Dan did not provide alot of details
on that Development process - It seems that eBay is using some
hybrid of feature driven development with waterfall (i.e. the
development is feature by feature - but the development of a feature is
waterfallish). The do have a constant delivery rate which they
synchronize using the concept of a train. if you have a features that
is it will be added to the train which is scheduled to arrive around
the time your feature will be ready. Several features are delivered as
a package which gives a predictable (weekly). I guess it also gives
them some nice metaphors to use such as a feature that doesn't make it
- misses the train or the train leaves on time etc. The slides of the presentation can be downloaded from Dan Pritchett's site (They not from the same event but they are pretty much the same slides. Also you can read Elliotte Rusty Harold's account of the presentation.
I won't say anything about my presentations (that's for others to say :) ). The point of this post is just to let you download them. So here they are: - SOA Patterns (2.14mb) - Takes a look at different strategies (patterns) to solve common SOA pitfalls
- Getting SPAMMED for architecture (4.56mb) - Takes a look at the activities architects can/should do when they think about software architectures. The presentation also covers architecture in agile projects.
Following the previous post I had a chance to exchange a few email with Mark Little (the
director of engineering in the JBoss division of Red Hat). Mark thinks
the topic of transactions and SOA has been beaten to death already and
wonder's why does it need to resurface (see his post "Is anyone out there?")
- I don't see a problem with discussions resurfacing when new people
are faced with situations others already solved (but that's a matter
for another post) Anyway, the reasons we're here is that I
think that during this conversation mark made a few interesting
observations and I think the end result is pretty interesting. I
decided (with his permission) to post it here ( It is only minimally
edited: no deletions, few additions (in []) and a few time shifting to
make it more coherent as a single conversation)
Mark :
From what I can see it's [the arguments on transaction and services -
are] the same old arguments that have gone round and round, ignoring
the important fundamental issues and not doing enough background
research. Sagas are transactional - it's just an Extended
Transaction model and not an ACID transaction model. Don't get hung up
on the word "transaction", which is way to overloaded in our industry
to actually mean anything by itself. Plus, 2PC is a consensus protocol
too; it does not impose any other aspect of ACID than the A. Even the D
is optional until/unless you want to tolerate failures. Arnon I know this is an old argument - but that doesn't mean it isn't worthwhile Mark It
isn't worthwhile if people aren't going to listen  I've been
involved in these debates so many times over the past 7 years (for Web
Services transactions) and longer for extended transactions, that it
gets a bit old after a while. Maybe we should create a wiki page and
point people at that ? Arnon I guess, but you should
keep in mind that people who are solely in the .NET camp only got WS-AT
recently with Windows Communications Foundation so you can expect the
issues to resurface. By the way a wiki might not be a bad idea [regarding
2PC.] 2PC is a distributed consensus protocol and in principle doesn’t
have to be related to ACID transaction. But I think the common view and
use of it is for ensuring distributed ACID behavior. Looking back at my
experience with XA and COM+ transactions it seems it does a good job at
achieving this ACIDness Mark This is an education
issue. The literature is clear on this. People who know and understand
transactional protocols don't make the mistake of equating 2PC to ACID
properties. Arnon Yes it is an educational issue . But
I am not sure that it is that common knowledge. It is expected that
middleware vendors who build the tools to support these protocols to
understand it better - I don’t think it is that widely known outside
these circles. Most of the architects I’ve don’t (maybe It time to look
for new friends ) By the way as 2PC is not resilient to
failures of the coordinator so in a highly distributed environment like
SOA it might have been a better idea to go with paxos commits if at all
you go down that path. Mark The reason WS-AT and WS-ACID
chose 2PC is: interoperability. All TP monitors support it. Try getting
IBM, MSFT, Oracle, BEA etc. to change to Paxos, 3PC, flat-commit, or
anything else and you'll be waiting for the heat death of the universe. Arnon Can’t argue with that Mark [also]
2PC is resilient to failures if the coordinator eventually recovers.
Paxos has its own failure assumptions too: Jim never disputed this.
Same as 3PC and other consensus protocols. As with *any* fault
tolerance approach (transactions, recovery blocks, replication, etc)
it's always probabilistic. All we're doing is making it highly unlikely
that the system cannot complete, but we can never make it entirely
safe. Even in the airline industry they can "only" go to a probability
that failures happen .000000001 Arnon You are
probably right that in an SOA situations the chances of not getting an
ACID transaction are worse than in a controlled environment - which
actually make the situation even worse since people using WS-AT
perceive it as allowing them ACID interaction (e.g. Juvals podcast) . Mark WS-AT
is *all* about ACID, in the same way WS-ACID is about ACID
transactions. It is *nothing* to do with SOA though. Web Services are
not purely the domain of SOA implementations! Arnon I
totally agree that Web-services and SOA are not directly related and
can each exist independently of each other. Again this is an
educational issue but, SOA==Web-services is a very common
misconception (I guess the word “service” in web-service doesn’t help
) [in any event] I think distributed transactions in general should be used carefully period. Mark Absolutely. They are not a global panacea and people who push them as such do more harm than good. Arnon WS-AT
is more problematic than regular distributed transactions as by
definition in an SOA you do not know who and how many other services
will participate in your transactions so you are much more likely to
run into problems. Sagas which embrace the temporal shift don't
give an illusion of ACIDness and allow to focus on achieving
distributed consensus while keeping all parties involved consistent. I
think that it is a much better option if you need transaction-like
behavior Mark For SOA, yes. Although Sagas are only good
for a certain type of use case. That's why we've always tried to
develop "live documents" that allow people to add new models when/if
needed. With a couple of exceptions during the BTP days, there has
always been consensus that one size does not fit all
(http://www.webservices.org/weblog/mark_little/blackadder_and_the_micro_kernel_approach_to_web_services_transactions). Web
Services *anything*, whether it's WS-AT, WS-Sec, or WS-Addressing all
have their non-SOA aspects because Web Services aren't developed purely
with SOA in mind. If that were to happen then Web Services as a
technology would lose some of their important benefits immediately. Arnon This
whole discussion is in the context of SOA (at least from my side ) –
naturally there’s a place for ACID transactions for other uses. Regarding
Sagas - calling them "Extended transactions that are not ACID" is just
semantics - my point was that they are not ACID transaction. I think
most people equate transactions with ACID transactions as well (but I
may be wrong) Mark Many people do and that again is an
education problem. The term Extended Transactions (don't need to say
"that are not ACID") has a well defined meaning in the R&D
community. There have been many good models and implementations around
Extended Transactions. They really took off in the vendor community
through the Additional Structuring Mechanisms for the OTS, back in the
1990's. If you check that out you'll see that it formed the basis of
WS-TX and WS-CAF. Even in Jim's original technical report he discussed
relaxing all of the ACID properties in a controller manner to get more
flexibility. That was the first extended transaction. In fact, ACID
transactions are just one type of extended transaction. There are many
many others, including nested transactions, coloured actions, epsilon
transactions, sagas etc. Unless I qualify it beforehand, I try
to never use the term "transaction" in isolation because it has
different meanings to different people. For example, when talking to
developers working in trading infrastructures, a "transaction" isn't an
ACID transaction at all. In telecos it's different again.
Evan H asked a question about distributed transactions and services in the MSDN architecture forum:
Are distributed transactions (ie.. WS-Transaction)
a violation of the "Autonomous" tenant of service orientation? Yes or
No and Why? Kudos if you can address concurrency and scalability (in
an enterprise with multiple interacting services).
I
answerd this questions back in april when I wrote a couple of posts
that explained why cross-service transactions are a bad idea: cross service transactions and some more thoughts on cross service transactions. Roger Sessions also agrees with this view (well, it seems actually, he wrote about it well before I did :) ): When
the WS-Transaction specification was first proposed, back in 2002, I
wrote an article explaining why I thought the idea of allowing true
transactions to span services was a bad idea. I published the article
in The ObjectWatch Newsletter, #41: http://www.objectwatch.com/newsletters/issue_41.htm.
Nothing since then has changed my mind. Atomic transactions require
holding locks, and spanning transactions across services requires
allowing a foreign, untrusted service to determine how long you will
hold your very precious database locks. Bad idea. Just because IBM and
Microsoft agreed on something doesn't make it good!
The reason I am bringing this issue back is that Juval Lowy (who wrote the article that triggered my first post on the subject) has recorded an Arcast with Ron Jabobs.
Where he re-iterated the idea that "Transactions is categoricaly the
only viable programming model" and you should strive to use it whenever
you can. It seems Juval admits you sometimes need to use Sagas
(which he called "long running transactions" - you can see in my link
why I think that's a wrong name). He also agrees that you can also use
a transactionable transport and then only do internal transactions from
each service to the transport (a pattern I call "Transactional
Service"). However, at the end of the day, he still thinks you should
use WS-AtomicTransactions whenever you can. I agree that
transactional programming is important. I think it is the simplest
programming model (from the developers side). I would probably never
write an interaction with a database that is not transactional; I look
very favorably at initiatives for in-memory ACI (no Durability)
transactions such as the one Ralf talks about. Until we get to Distributed Transactions... First, we should note that transactions are not "the only viable" option.As Martin Fowler notes
Ebay seems to be doing fine without distributed transactions. Not only
that, they abandoned distributed transaction and went
"transactionless"because they needed one simple thing... Scalable
performance .
In most COM+ scenarios you have a single server or a few internal
servers where the distributed transaction happen - and even there you
should plan your transactions carefully if you want to get any kind of
decent performance. In SOA scenarios the situation is more complicated
as the distribution level is expected to be higher (even if you don't
involve services from other companies). More distribution means longer
times to complete transactions (especially if a participant can flow
the transaction and extend it). It also means increasing the chances of
failure (see Steve Jones series of posts on five nines for SOA).
In my opinion, the more distributed components you have the more you
want their interaction to be decoupled in time - i.e. the opposite of
transactions. Juval also said he doesn't buy the denial of
service problem I mentioned (supporting a transaction means you allow
locks - if an external party doesn't commit you retain the lock..).
Juval said he assumes that a solutions has both authentication and
authorization so this shouldn't be an issue. For one, I have seen too
many projects where security was something that was neglected or
quickly patched in at the latest moment - so I would hardly assume
security. Even with security on - you increase your attack surface. But
that's just the half of it. Even if all your service consumers have
good intentions - you still don't know anything about their code. SOA
is not like the "good old days" where you owned the whole application
- this means you cannot trust their security to be ample. Also you
don't know anything about their code quality. Services are likely (in
the general case) to be deployed on different machines, even if they
start co-located. I think that a Service boundary should be treated as
a trust boundary just like a tier boundary.
I strongly believe you should have reduced assumptions on what's on the
other side of the service's boundary - transactions are not reduced
assumptions
SOA and distributed transactions do not go hand in hand - it isn't just
autonomy at stake here. It is a problem for performance and scalability
and even security period.
To finish this post - I would also highly recommend looking at Pat Helland's paper "Life Beyond Distributed Transactions: an Apostate's Opinion" and a post he recently made called " SOA and Newton's Universe", where he explains more eloquently than I ever could why SOA is not a good fit for distributed transactions.
Steve Jones has (yet another) great post called " Le Tour SOA - why support services are critical, but not important".
You should go read the article - but in a nutshell, Steve explains that
important services are the ones that bring business values and critical
services are the supporting ones that help keep the light on for the
important services to function properly.
While the post has SOA in the title. I think it is more general and is
also applicable to applications or any other IT generated components.
In fact it can also be applicable to IT itself as Nicholas Carr noted
in 2003 when he published his paper " IT doesn't matter".
Nicholas argues that IT will become akin to electricity and as such be
critical for the business to continue operating but not important. As a
side note I'd say I think this is might be true for traditional
businesses but not for businesses where the IT is the business (such as
banks, insurance companies, etc.)
Back to critical vs. important - I think this is an important for
architects to make this distinction to be able to prioritize work and
not confuse business value with semblance of business value due to
criticality for operations. This doesn't mean you can neglect critical
tasks (after all they are critical...). It is the important stuff that
will bring your business the competitive edge.
A couple of quick observations following the Events and temporal coupling post
Events, Current data and aggregated data all have Time-to-live aspects.
- Events value usually diminishes over time until the TTL reaches
- Current data usually have a constant value while their TTL lasts
(until a new value is the current data) - unless we are talking about
version data which is a component of or a step in the direction of
aggregated data.
- Aggregated data has the longest TTL, it is interesting to note that its value increases over time
Also while the Current data TTL is determined by the producer both Events and Aggregated data TTLs are determined by consumers
Yeah, I know these are not not earth shattering observations but I still think they are interesting
You raise an event when something interesting happens to you, you think
it is important, but you don't care enough to know who is interested.
you are even less interested in to personally going and to each an
every interested party and letting them know. So - instead, you raise
and event, and let the poor buggers take care of any implications by
themselves. We raise the event "now" when the change happened - it is
only important now anyway... Looking from the "poor buggers"
-the event consumer point of view things are more complicated. There
are events which are cyclic in nature like stock price updates, the
blips from a sonar etc. if you missed one, then it isn't really
important you'd get the right information in the next update (actually,
that isn't entirely true - see later in this post). Then, there are the
events which only occur once. sometime it isn't important for you to
listen to them if you are not up and running in the same time. Other
times you can't afford to lose an event for instance if your ordering
service (or component for that matter) communicates with the invoicing
one using events you don't want to miss the event of a new order else
you would loose money. This basically means that the event
producer and the event consumer are coupled in time - one way to solve
it is to make sure both of these services are available at the same
time i.e. if the invoices crashed, then processing orders should be
suspended (note that this doesn't mean that you don't accept orders
just that you don't process them). Ok - maybe we can just raise the
event "transactionally" - this would probably work, but we need to
remember that the event producer doesn't really care about the event
consumers, why would it want to fail because of them?! Maybe a
better way would be to "raise" the event over some reliable transport
- this has a few problems. one is that we've passed the problem to the
connection between the event producer and the transport. It might be
acceptable to have a transaction between the event producer and the
transport. However, as I've already said the producer doesn't care much
about the consumers.. We can have persistent subscriptions for
existing consumers to prevent events from getting lost which make both
creates a er minor problem that new consumers can't see past events but
also has the risk of existing subscribers disappearing and their queue
can then grow endlessly (or until an administrator would remove the
subscription) Ok, let's try to look at the problem from a
different angle. looking at the events, what we can really see is that
an event has a time-to-live (TTL) as far as the event consumer is
concerned. For instance in the case of the cyclic events the TTL is the
interval until the next event. Actually, even with cyclic events the
TTL might be larger - if we are also interested in analyzing trends or
ab normal occurrences (which is why I said it isn't entirely true we
don't care about old events). In case of one-time events the TTL might
be indefinite or maybe even then it might be some definite value (one
day, week, year etc.). Since we can't know about the TTL of consumers
it can be a good idea to make past events available somehow. Thus,
when you design an event centric architecture like EDA (whether on top
of SOA or not) it is important to think about event consumers - we
don't want to think about specific consumers since it negates the
benefits of thinking in events, but I would say that you want to think
about event consumers in general, after all your component is also an
event consumer (do unto others as you would have them do unto you) One option, which I already talked about,
is to make past events available as a feed. Event consumers can then
come at their own leisure and consume past event (this can be in
addition to to raising the events in real-time). This provides a
partial solution as the maximal TTL is determined by the event producer
(after which the event is deleted from the feed). This may be
acceptable but you must be aware of that. The other option is to to
log all the events and provide an API to retrieve past events. In a
sense the max TTL is still at the hands of the event producer only if
you use a database it would probably be a large time compared with a
feed. Alternatively the events can be logged on by a central "always
present" event aggregator (in a manner similar to the aggregated reporting pattern I described for SOA).
To sum all this - events they seem only to matter in the instance in
time they are created, we are used to that thinking from building OO
systems where all the components are co-located in the same
address-space and time (even there I can think of scenarios where we
would want past events) - in a distributed world events need to have a
TTL, the TTLs can vary and are determined by the events consumers.
Lastly, as I demonstrated in the paragraph above, there are several
strategies we can use to help solve the event TTL dilemma (and there
are probably a few others).
Few months ago I wrote here about solving the mismatch between Service
Oriented Architecture (SOA) and Business Intelligence (BI) (see papers and articles section). Recently I got the following question from Ben: One
major question I have is around large data sets. As an experienced
BI/DW architect and developer I have worked on a number of large scale
data warehouses. Retrieving large data sets (i.e. millions of records)
doesn't seem to fit well into SOA. As you state in your article, we
could have another point-to-point interface, where the service which
houses data we need gets a request and writes out a batch file (xml or
plain ascii text). Then using typical ETL, we grab the file and load
it. The underlying source system (service) can use optimization in
generating a large data set (vs. record by record) and
the data warehouse can correspondingly load in bulk. Like most architectural questions - the answer is "it depends" For
instance, if you do a run-of-the-mill ETL as a on-time setup then it is
just that- a one time setup and I, personally, don't see any
contradiction between SOA goals or tenets and that. I do think
that iit is better to enhance SOA with EDA interactions to provide a
long term solution to the BI problem. You can also have a dedicated
component that aggregated the information that flows in in these events
and builds batch files that are suited for the ETL you've used during
the setup phase (mentioned above). It is true though that moving an
SOA which is already in-place to EDA is not a small feat, but adding
EDA layers does not have to mean that the old interfaces go away -
especially not immediately (remember to treat services as products) If
you have a business that generated millions of records on a daily basis
- then the situation is more complicated. Now you have to think about
the trade-offs between "compromising" SOA and adding a dedicated
interface (or a backdoor to the database) for the ETL vs. the
implications of performance, bandwidth, transition costs, ROI etc. of
pushing that information with EDA. I, personally believe in pragmatism and the " no-silver-bullet" approach so I can't say that EDA is always the best solution (As an aside, this is part of the reason I write my book
as patterns not as "best-practices guidance"). You may find that ETL is
the best trade off in your situation. Yes I know that it isn't a
definitive answer - but real life is (usually) a little more
complicated than black and white solutions. As architects we need to
find the best trade off for the situation at hand.
In addition to the drafts of selected patterns I publish on my site, you can now purchase my book via the Manning Early Access Program (MEAP). MEAP
means you can get chapter drafts as I write them and the complete book
when its done (ebook or printed). Here is Manning's explanation: "Buy
now through MEAP (Manning Early Access Program) and get early access to
the book, chapter by chapter, as soon as they become available. You
choose the format - PDF or ThoutReader - or both. By subscribing to
MEAP chapters, you get an opportunity to participate in the most
sensitive, final piece of the publishing cycle by offering feedback to
the author. Reader feedback to the author is welcome in the Author
Online forum. As new chapters are released, announcements are made in
the MEAP Announcement Forum. After all chapters are released, you will
be able to download the complete edited ebook. If you order the print
edition, we will ship it to you upon release, direct from the bindery,
weeks before it is widely available elsewhere.
By the way, this is probably also a good time to mention that I'll be speaking about quite a few of the patterns in Architecture & Design World 2007 which will take place this July.
There is
still a lot of work, but I already like to thank all the people in
manning that helped me get this far. especially to Cynthia Kane my
editor (hey, maybe now she'll give me more slack :) )
Ok, 'nuff blubbering, back to completing chapter 5...
Agile and documentation? what gives? First things first documentation is not something that is prohibited by Agile Manifesto. Working software is definitely preferred over "comprehensive documentation" but there can also be some value in documentation The
first question is why would we want to document anything if we have a
working software. I think there are several stakeholders like project
newcomers, maintainers etc. who will be interested in something that
will let them get up to speed and provide them an overview of what's
going on before they delve into code. You can read more on that in a
post I wrote almost a year ago called " Who needs a software architecture document"
but in essence the main motivation for documentation is that assuming
that the software is successful it will outlive the team - i.e. the
people that built the software will not be the ones that will have to
develop, maintain and support it for most of the software's life. If we agree that we need a software architecture document the question is what to document and when. There
are two main "things" you can document in regard to architecture, the
first is the obvious one: the architecture itself. In my experience the
most value can be derived from documenting recipes i.e. how to do
stuff that is common in the architecture. These recipes can be a short
description of the context and then a pointer to a test (or tests)
and an implementation that exercises this. You can think of the recipes
as a type of a tutorial to the architecture. Other documentation
worthy elements related to the architecture are an overview and
technology mapping (including what a developer needs to install to
start working). The overview allows a newcomer to understand where to
find what, the technology mapping allows for understanding what she
needs to learn and install to be productive. Note that to be useful the
overview should be at a higher level of abstraction than the code -
otherwise you run the risk of missing the forest for the trees or at
least not saving any time. It is obvious that documenting any of
this before your architecture is stable more or less is useless, as a
rule of thumb I would say this can be around the 5th-6th iteration -
assuming the team has to grow during the project. If the team stays
stable for the duration of the project, this documentation can take
place towards the end of the project (though I would probably add
recipes to a wiki or something similar during the project as
development patterns emerge). The second "thing" you can
document in regard to architecture are the decisions you decided
against, in my opinion this is more important than all of the other
items mentioned above together. The reason for that, while it might
take a while to understand a well written software and infer its
architecture it can be done, but it is virtually impossible to
understand the options that were disqualified from looking at the
chosen solution. Understanding the options that weren't used can
save time for the person reading that description. both in
understanding why things are the way they are. Furthermore it can save
time trying things that didn't work or provide clues to options when
the circumstances change (since, as we all know, requirements change..) The
best time to document decisions you decided not to take is when you opt
not to use them - this is when you remember best "why". for instance,
in my current project we use x.509 certificates to authenticate clients
and we use decided to use Kerberos tickets to authenticate components
within the service. There's a reason for making that translation*,
there's also a reason for making the transition by replacing the client
certificate with the edge component's
credentials instead of mapping the client's certificate to a Kerberos
ticket using an Identity provider*. we had two developers spike
different options for two weeks until we came to the current solution
instead of the more obvious choice of passing the x.509 certificate
from the edge into the service and using the client's credentials. This
question is likely to come up when/if someone else would take over the
project, when the technology will be updated etc. Again, if we know why
we didn't make that choice we can better decide what to do when the
circumstances change.
To sum up, there are few architecture related issues that are worth
documenting even in agile projects. some of them can be postponed some
of them are worth documenting a little earlier. In any event it is
better to document after the fact and to keep the documentation light.
*
It all has to do to to limitations of WCF in regard to the transports
we use (HTTP, MSMQ and TCP) and the request/reaction pattern
(asynchronous communication) we use.
While I am on the topic of REST, it is probably a good time to comment on my (first) post on InfoQ " Debate: Does REST need a Description Language" Personally,
I think there's merit in Services publishing their message structures
in a machine readable format. When a Service has a machine readable
contact. generated stubs allows you to make the interaction with less
bugs vs. hand crafted interactions. It also makes it easier to test the
service itself. I do agree with Stefan's views on runtime interface dependency
where he said that if a service consumer needs just 20% of the
information in a service it shouldn't be forced to deserialize (i.e.
know or care about) the whole message.However, I think this is a
weakness of tooling not the concept. What if you had a tool that reads
the machine readable contract, allow you to pick the 20% you need and
generate for you a stub that ignores all the other 80% and "hand pick"
the 20% you need. This is what you would personally do yourself anyway,
and since the code is generated from the Service's definition it would
be more resilient and error-free This is effectively designing a
personalized mini-contract from the published general one. It does mean
that when that 20% changes you will be affected, but this is something
you'd have anyway. I also agree that that the WS-* standards and
resulting contract are (and getting more) complicated. Much of this can
probably be attributed to the "design by committee"
effect. However, there are also some real challenged that the SOA and
ROA architectural styles do not address and we still need to solve
those. Trying to solve these challenges is, by the way, what prompted
me to write my SOA patterns book...
DevHawk (Harry Pierson) r aised today a question
I was toying with myself for a while now - if REST is an architectural
style can it exist without the specific technologies that define it
today. Or as Harry put it :
- REST is a an "architectural style for distributed hypermedia systems".
- REST "has been used to guide the design and development" of HTTP and URI.
- Therefore REST as an architectural style is independent of HTTP and URI.
- Yet,
I get the feeling that the REST community would consider a solution
that uses the REST architectural style but not HTTP and/or URI as "not
RESTful".
What I had in mind for example is to use messaging where the equivalent of the URI would be a topic hierarchy.
Topic hierarchy allows you to have a unique "URI" for each resource.
The next thing we need to take care of are the PUT, GET, POST and DELETE verbs - we can do that by making the verbs part of the message headers.
As an aside I'll also say that if we try to think about it as an
architectural constraint then we don't necessarily have to use these
verbs, a more general rule would say that the verbs are uniform and
well known rather than specific ones
The rest (no pun intended) of the concerns, like specifying related states etc. can be dealt with making conventions on the message formats
Is that still REST?! I wonder...
In any event, what worries me the most in regard to REST is
the religious manner that some people seem to treat it. By the way that
is the same phenomena we see with some of the Agile folks. As for me? - Well, I don't really care if I fit that label or the other. I am just payed
to deliver working and viable software :), but hey, that's another discussion.
Last month we saw Ajax was making a move to the desktop with the aid of Adobe Apollo. Earlier this month I talked about it again when Sun announced JavaFX and after Microsoft officially announced Silverlight.
I guess it was inevitable that Google would also join the fun of
blurring the lines between desktop applications and web applications.
And indeed I just read that Google released a beta of Google Gears.
What does Google Gears do? well, in essence it lets you develop
occasionally disconnected web applications by providing an API to store
application resources and data locally as well as Workerpools that
allow you to run tasks in the background (sort of like a collection of
processes that host javascripts and can communicate by messages).
What all these means is that it wouldn't be too long before the the
architectural choice between web clients or desktop clients will not be
that important it will be more about making a technology choice on the
preferred development environment and tooling as well as designing
robust occasionally disconnected applications regardless of that
technology. Just look at the architectural guidance Google provides
for utilizing Google gear. You'd see that the components that they
identify are very similar to the high-level components you are likely
to design with any occasionally disconnected desktop application...
Yesterday I attended Jim Coplien's presentation on "Organizational
Patterns - a Key for Agile Systems Development". Overall I think It was
a very good presentation. Jim makes a few interesting claims, some of
which are controversial within both in the traditional and the agile
spaces Few examples - Process guidance (ISOs etc.) doesn't work - roles are stabler than processes, processes always change.
- Jim
says that in order to make a change you need to make it at the
organizational structure level. The processes will then support these
changes
- TDD is evil - it is just an re-incarnation of bottom-up procedural design. It is better to follow "Design by contract"
- He says XP is not a good methodology (He thinks SCRUM is good)
- etc.
Additionally
he talked about some of the organizational patterns he and Neil
Harrison discovered studying organizations for more than a decade. You
can read the Top ten patterns on his site.Jim covered 2 patterns that are related to software architecture: Architect controls product and Architect also Implements
Architect controls Product basically says that you should have an
architect and that she should oversee that the direction of the project
is flowing in the right direction.
Architect also implements - this pattern says that in order for the
architect to broaden her leadership without sacrificing depth and
pragmatics she must also participate in the implementation (beyond
advising and communicating). Jim gave the example of the development of
Borland's Quatro pro for windows in 1993 where the team's architect had
a daily meeting (akin to scrum stand-ups) for synchronization and would
then go and code with the developers. The Quatro pro team had 4
architects out of 12 persons that made the team.If a third of the
development team is architect I'd say he is right - My experience,
however with most organizations I see is that you hardly have one
architect per project (sometimes you only have one for several
projects). In these cases I hardly see the architect writing production
code as part of the team since she would not have time to fulfill her
architectural responsibilities. She must know how to code though and
she must be able to prove her designs in code or be able to offer a
candidate implementation if needed (I also wrote about that in the past
see "Should architect's code" part 1, part 2, part 3By
the way if you are located in Israel, Jim will be here for a couple of
weeks and he is giving a few courses like Agile Architecture,
Patterns of Agile Project Management etc. You can find more information on pacificsoft's site
Udi Dahan writes that " .NET/Java Interop is not a reason for SOA". Udi writes that companies that need to integrate two technologies turn to web-services and that
"The only problem is that in order for things to work right, they really must
have a chatty interface, and flow transaction context between these
“services”, and all the other things I describe as anti-patterns"
Udi is right that if you don't rethink and remodel your systems you
will (probably) not have an SOA as you are likely to find your self
implementing anti-patterns such as the ones he mentions.
However, using Web-services does not automatically mean that you are
doing an SOA. If you don't think about moving to SOA you can still opt
to use web-services as a remoting or RPC technology to connect two
systems. The advantage over the other proprietary products Udi mentions
is that web-services are a standard technology. This will work well or
fail is orthogonal to the technology choice. It depends on the
architectures of the systems you integrate. If you need to flow
transaction between the systems you'd also need that even if you
cross-compile one of the applications in the other environment.
Another thing I don't agree with is the word must Udi uses.
First, while it is likely that older systems has chatty interfaces it
is not a must. The designers of the legacy system may have thought
about the consequences of distribution without regard to SOA. Also you
can still wrap an existing system with a service contract (using
web-services or any other technology) and not get to chatty interfaces
etc. However that means that the wrapper should have some substance or
business logic inside it to mask the old system's behavior this is
especially important if you are thinking about moving to SOA and you
take into consideration that the business will not just halt and wait
there until you are done. You have to think about interim solutions,
such interim solutions can include wrapping a legacy system with an
Edge Component and a SOA facade (a pattern I call Legacy Bridge) while
you move in the grader direction of a full blown SOA.
It has been awhile since I last published a pattern draft - but I guess it is better late than never. The saga pattern deals with manginc complex interactions between services without the use of atomic transactions, which as I mentioned in the past are not a good idea (see " Transactions between services? No, No, No!" and " Some more thoughts on Cross-service transactions" ) You can download the draft for the Saga pattern from here. I'll also add a link to it from the SOA Patterns book section (where you can also download the other pattern drafts I published) By the way, I am not happy with the current sketch (the pattern illustration) in this pattern, so it will probably change in later drafts. I would be happy to hear any suggestions you have for improving it.
I just read Shy Cohen's (via Nick Malik) article in Microsoft's Architecture Journal entitled "Ontology and Taxonomy of Services in a Service-Oriented Architecture"
Shy provides a list of what he calls service types. He identifies two major types bus services and application. He then continues to sub-divide them, the Bus Services are divided into communication and utility services and the Application services are divided into entity, capability, activity and process services.
I have to say it was quite alarming to see this coming from someone who had deep involvement in defining Windows Communication Foundation Indigo.
Where do I start?
well, for one, it seems completely fails to make the distinction between Services as in Service Oriented Architecture and Services as in capabilities or features an infrastructure provide. The "Communication services" are for the most part capabilities that a service infrastructure (such as an ESB) provides. Not Services you would define in an SOA initiative
And then there's the matter of service granularity and the difference between Remote objects and SOA for instance, the example Shy gives for a "method" (his word) on a Customer service (entity service):
"An example of a domain-specific operation is a customers service that exposes a method called FindCustomerByLocation that can locate a customer's ID given the customer's address"
why would a service return a customer ID? This is the kind of call you would make on an object you hold a reference to not some remote "Something" that also want to authorize your call and may reside in a different company. This kind of thinking is what made remote objects fail. Gregor Hohpe explained that nicely in a paper called Developing software in a Service Oriented World
The Transparency Illusion. Distributed components promised to hide remote communication from the developer by making the remoteness "transparent". While the basic syntactic interaction between remote components can be wrapped inside a proxy object, it turned out that dealing with partial failures, latency, and remote exceptions could not be hidden from the developer. It turned out that 90% transparency was actually worse than no transparency because it gave developers a false sense of comfort.
As a side note, Gregor recently gave a presentation that covers this paper at JavaZone which you can watch online at InfoQReturning to Shy's article let's take a look at another quote:
Capability Service may flow an atomic transaction in which it is included to the Entity Services that it uses. Capability Services are also used to implement a Reservation Pattern over Entity Services that do not support that pattern, and to a much lesser extent over other Capability Services that do not support that pattern.
I already explained why cross-service transactions and especially flowing transactions is not a good idea in SOA so I won't do it again here - but you can read about it both here (" Transactions between Services? No, No, No!") and here (" Cross-Service Transactions").
Also I truly hope Shy didn't mean .NET data sets when he said "In some cases, typically for convenience reasons, Entity Service implementers choose to expose the underlying data as data sets rather than strongly-schematized XML data. Even though data sets are not entities in the strict sense, those services are still considered Entity Services for classification purposes."
In any event the whole decomposition of Services into fine grained "capability", "Activity" and "process" takes no account of the fact the SOA is a distributed architecture...maybe Microsoft is not affected by the fallacies of distributed computing ?
*ad nauseam (latin)- to the point of disgust
Yesterday, we had a meeting of a few architects and Memi Lavi gave a nice presentation on Architecture usability. The notions of architecture and frameworks/software infrastructure are a little mixed in this presentation, but in essence Memi is right to say that among the quality attributes the architecture has to address you should have modifiability and architecture usage scenarios like "adding a complex form will take less than 2 weeks" or "wiring a new component into the workflow will take less than one day" etc. Reducing friction* of frameworks is very important. I don't think it is the most important though. since if you could only have one of the quality attributes you need to support (performance, correctness, architecture usability, framework friction, system usability, availability, budget etc.) I am not sure you would always pick architecture usability. Driving home, I thought that essentially software architecture is justThe set of compromises, trying to maximize the effect of a limited set of prioritized quality attributes
we can't have it all - so we need to prioritize our list of quality attributes and focus on the most important ones.
*Friction (from Ryan via Udi): "friction" is a (subjective) measure of how much the tooling gets in
your way when trying to solve a specific-case problem. I've come to
evaluate frameworks based on two rough metrics: how far the framework
goes in solving the general case problem out of the box and how little
friction the framework creates when you have to solve the specific-case
problem yourself. When a framework finds a balance between these two
areas, we call it "well designed."
Pat Helland is back in Microsoft (after a two years vacation in Amazon) and more importantly he also restarted blogging. I only met him in person a few times - but he is definitely one of the few persons really worth listening to - especially when it comes to distributed computing. Not only does he make interesting observations he is also capable of explaining them in a crisp and interesting manner. Indeed, it didn't take too long (his second post) before he blogged some valuable content. The post is called Memories, guesses and apologies (go read it). Pat talks about how the notion of time in a distributed environment is subjective and you can really know what happened before what and what we can do about it (I really think you should just go read it :) ). Another related aspect of the phenomena Pat mentioned is that taking a snapshot in time, the chances of having a single unified truth in a distributed system degrade in a proportional manner to the system's load. I had a chance to work on a few systems where some of the sites had either occasionally connected or connected over low bandwidth networks. This situation makes the whole notion of guessing the state and compensating and/or apologizing for wrong conclusions much more explicit than in always connected high bandwidth system. Nevertheless, latency still exists even in connected systems and and you should really be weary of assuming a universal truth - unless you can stop the businesses long enough to allow complete synchronization. As I mentioned a few days ago, we can't afford to have cross-service transactions (I also think we can't afford too many distributed transaction in non-SOA architectures, but this is a especially true for SOA) which makes things even worse in this sense. One thing we can do in an SOA to achieve distributed consensus is to run a Saga. Saga, which is a long running conversation between services, is probably one of the most important interaction patterns for SOA. You know what? instead of trying to explain it here in a haste i'll just publish the pattern draft - I'll try to do that before the end of the week.
Johanna Rothman writed about " letting go of BDUF"
(Big Design Up Front). One statement she makes is that you can't
predict what the architecture needs to be. I can't say I agree with
that, since many times you do know something about the project and you
do have prior expereice that can give you enough confidence to lay some
ground rules. I called this Just Enough Design Up Front or JEDUF for short. Another
statement Johanna made, which I wholeheartedly agree with, is that you
should let the architecture evolve. Evolving an architecture sounds
very compelling but it is not a simple feat. Architectural decisions
tend to have system wide implications which means that changing one too
late in the game you'd get a lot of rewrite and/or refactoring to do
. My strategy to solve that conflict is to:
- Set the first one or two iterations as architectural ones. Some
of the work in these iterations is to spike technological and
architectural risk. Nevertheless most of architectural iterations are
still about delivering business value and user stories. The difference
is that the prioritization of the requirements is also done based on
technical risks and not just business ones. By the way, when you write
quality attribute requirements as scenarios makes them usable as user stories helps customers understand their business value.
- Try to think about prior experience to produce the baseline architecture
- One of the quality attributes that you should bring into the
table is flexibility - but be weary of putting too much effort into
building this flexibility in
- Don't try to implement architectural components thoroughly - it
is enough to run a thin thread through them and expand then when the
need arise. Sometimes it is even enough just to identify them as
possible future extensions.
- Try to postpone architectural decisions to the last responsible
moment. However, when that moment comes -make the decision. try to
validate the architectural decisions by spiking them out before you
introduce them into the project
These steps don't promise that the initial architecture sticks, but in
my experience it makes it possible to minimize the number of
architectural decisions but still have a relatively solid foundation to
base your project on
An article I wrote on Business Intelligence (BI) and Service Oriented Architecture (SOA) has just been published on MSDN. You can find it here http://msdn2.microsoft.com/en-us/library/bb419307.aspx. The article explains the SOA & BI mismatch and how to bridge it by adding EDA to SOA. (I bloged about it here before, but the article is more ordered and complete)
Back in January, I took part in an architect panel that Microsoft
Israel organized. The panel was led by Ron Jacobs and it featured Udi Dahan, Assaf Jacoby, Coby Cohen, Dudu Benabou and myself. A few days ago Ron edited this recoding and turned it into a
podcast in his Arcast series.
The panel's focus was on lessons learned from mistakes made in past project. Ego
maniac as I may be :) -- even though you don't get to hear me much in
the final edited version -- I think the podcast is worth listening to,
as the panel raised some interesting points. You can download the
podcast here (don't worry it is in English even though it was recorded in Israel)
PS I am the first speaker after the introduction, in case you are wondering.
I have seen the following question on one of the forums I follow "I have studied up on the SOA approach and it all sounds good. But most articles stop at the theory.
Lets
say I sell things. I have a CustomerProfileService. The application
does CRUD through this service to a back end database. Its autonomous
and isolated.
I have anther service, InventoryItemProfileService.
Again, the application does CRUD through this service to a back end
database. It is autonomous from the CustomerProfileService. Not only
may it live on a different DB from the CustomerProfileService, it might
exist on a different platform.
Now lets get to the InvoiceService.
Lets say from the client side, I would guess that i would have a
CreateInvoice(custID,itemID[] ) method. The InvoiceService would then
call out to the CustomerProfileService for profile that meets the needs
of the invoice, then another call out to the
InventoryItemProfileService for the item descriptions and such.
Here
is the question. It would seem like in the back end (the db) of the
InvoiceService there would be tables to support the customer info and
the item info from the invoice. Where prior to SOA, when everything
was in the same db, these requirements would be largely satisfied by
joins. Now a logical join across services just seems radically
expensive (everytime you touch the invoice). hence the need for the
customer and item tables local to the invoice service.
Does this sound right? Just how often does the InvoiceService have to go back to these other supporting services?"
I also got a comment with a similar theme on my Cross Service Transactions post. I
see a few problems with the way the services in the question are
modeled (like CRUDy interface) but in the end it all boils down to the
root cause -and the real problem: granularity of the services.
Sure
when "a service" is too small it doesn't make sense to separate its
tables from those of other services. it doesn't make sense to have
transactions that span only what's internal to the service. It doesn't
make sense to pay the price to make a service autonomous (like caching
reference data from other services). When the granularity is too small
you will often find that you need to make a loot of interactions with
other so called services. you are more likely to have CRUDy interfaces.
You are also more likely to have slow performing solution and suffer from low-availability. Using
services in a granularity mentioned above is, in my opinion, a
nightmare that would probably make you work very hard to maintain the
SOA principles in place - or the more likely option, that you would
circumvent the principles so that you can get something maintainable,
usable and performing (and flip the bozo bit on this all SOA thing) So
what is the right granularity. Well, it is not a one-size-fits-all kind
of thing, but as a rule of thumb I would say anything just shy of a
sub-system and up. A service has to have enough meat so that it would
make sense having it autonomous; that the transactions would fit nicely
inside its boundaries; that it would be worthwile making it
highly-available; that you can pass a complete task/document to it and
it won't have to talk to a gazillion other services to complete
processing it; etc. If your application's idea of invoices is a
2 tables one with a header and one with invoice details - then don't
make that a service. if invoicing is a sub-system with complex business
rules a lot of options and what-not - then it can be a good candidate Think about it next time you design a service :)
After seeing Juval Lowy's article on WCF transaction propagation in the May issue of MSDN magazine. I posted " Transactions
Between Services? No, No, No! " in my DDJ blog. I've got a few comments which I thought warrant a post in their own-right.
The previous post was triggered by an article that promoted flowing
transactions (i.e. you perform a transaction against one or two services and
then one of the services calls an additional service and it joins the
transaction). It is important to say that I think transactions between services
should be discouraged regardless of automating extension of transactions.
Transaction propaqgation just makes the matters worse.
There might still be some edge case where you have to have an atomic
transaction from a service consumer to the service. I think that in the vast
majority of SOA implementations you shouldn't do that and I would think real
hard about the other options before allowing it in my architecture.In general I think cross-service transactions are an antipattern (and that's the way you'd find them documented in my SOA patterns book :) )
One of
the comments I received began with:
"Cross service transactions are a sure way to introduce coupling and
performance problems into your SOA." I'm not sure I agree with that thought.
Logically speaking, cross service transactions are a must. The question is how
to implement them. There are two mechanisms we can use for implementing TXs: (1)
ACID TXs; (2) Long-running TXs. The latter is preferable for the cases Arnon is
talking about (large geographical distances, multiple trust authorities, and
distinct execution environments). ACID TXs are more suitable for what Guy has
mentioned (DeleteCustomer service invokes the DeleteCustomerOrder service
internally). I agree with Arnon the a-synchronicity is preferable, but we all
have encountered use-cases where ACID-ness is required from a business
requirement level... [snipped]
One minor point in regard to this comment is that I don't like the term
long running transaction - there is a long running interactions between services
and I think the term SAGA describes them better. Sagas are made of a series of
business activities that flow back and forth between services to realize a
larger business process. Note that these interactions doesn't necessarily have
transaction-like behavior.
which brings me to the more important point of looking at the statement
"Logically speaking, cross service transactions are a must". I don't think so.
For instance, if a service that manages the inventory in a warehouse receives a
request for some items and later a cancelation of that request. The first
request can trigger the inventory service to order some more items from a
supplier. Whether or not the cancellation would cause a cancellation of the
order of the supplier depends on the business rules of the inventory service for
inventory levels for the items ordered. it might also depend on whether or not
the items have already been received etc. The cancellation (the "abort") of the
original request does not have to translate to an abort (or compensation) on the
request receiver. Furthermore if the service communications model is based on
the push model (e.g. using EDA with SOA) the cancellation notice would just be
propagated without regard to the inventory service -. It is the inventory
service's responsibility to understand the ramifications of this event and act
accordingly. Even the example given in the comment 'DeleteCustomer service
invokes the DeleteCustomerOrder service internally" is not a good candidate from
ACID transactions (there's also a problem of service granularity here - I'll
talk about it later). Since when the customer service decides to delete a
comment and request the Orders service to delete orders - there's a reasonable
chance that some of the orders are already paid for but not delivered. In this
case the customer cannot really delete the customer until all the paid orders
are resolved. Or maybe the order service is a facade to a night batch that does
the actual deletion. - I know I am just fantasizing with these examples but the
point is that the customer service has no knowledge on the order service or the
inventory service above except the messages supported in their contract. To
assume something about the internal behavior is problematic. Even if you know
about the internal structure on the onset, the whole idea of SOA is that the
services can evolve independently from each other...
Another thought triggered by the example in the comment originated by the
granularity of the services (DeleteCustomer service vs. a Customer Service that
also supports deleting customers) is that we should be really conscious to the
difference between other architectures like 3-tier client/server and SOA. SOA is
actually more distributed than 3-tier - we cross a distribution boundary every
time we pass a message from a service to a service and not just when we move a
massage from a client-tier to an application server. We add this distribution to
gain advantages in flexibility and agility. However, we should note that this is
a weakness of SOA (considering for example, that Martin Fowler's first law
of distributed object design is" Don't distribute your objects!") means we
should really pay attention to the way services interact with each other.
- The granularity of services - having a lot of fine grained services means
there will be a lot of interactions over the wire (even if you don't go out to
the network you still have to serialize/deserialize, follow the security policy
etc.) rather than internal interactions that much faster
- The Granularity of messages - The same considerations should also guide us
to try to create larger and fewer messages. for the example above . Instead of a
DeleteCustomerOrder message maybe something like an UpdateCustomersOrders
message that can hold a list of customers and orders and the status changes or .
by the way this would also support off-line clients better since they can
accumulate changes.
- The assumptions we can make on the other service's availability,
performance, internal structure, the trust we have for it etc. - We should try
to minimize the assumptions we make and concentrate on what can be inferred from
the contract. Remember that policies can change externally so the business logic
within a service cannot count on them being constant. this brings us back to the
issue of transaction. every cross-wire interaction increases the chances of
failure - in transactions one failure invalidates all the transaction is
invalidate. every cross-wire interaction within a transaction increases the
length of time we lock internal resources (even if we do trust all the involved
parties) - especially if that transaction can extend itself automatically. Also
as I've mentioned in the previous post the transactions also open the door for
denial of service attacks.
If we want to reap the benefits that are sold under the SOA moniker, like
flexibility and agility, we really have to pay attention to this extra
distribution and design our services differently than we would components in a
3-tier architecture - but hey, that's why they pay us the big bucks, right ? :)
I should probably also add that building SOAs is not a goal in itself.
We can build perfectly good solutions using other architectures - but if we find
that we do need SOA (or any other architecture for that matter) we have to pay
attention to the way we implement it to both keep its benefits and not harm
other quality attributes like performance, security etc..
Back in January I opined that moving to web applications was not the optimal solution to the real problem we have/had with desktop applications which was installation woes. What we got was a poor UI without installation problems so we (software industry) started to resolve problems like we had when we moved from terminals to graphical UIs etc. - So now we have Rich Internet Applications (RIA) - using technologies like AJAX - but they suffer from other problems which again we've already been through Well that was the topic of the post in January. Now I've stumbled upon an interesting/amusing twist - called Adobe ApolloApollo let's you, yes you've guessed it, take your RIA applications and deploy them as desktop applications. you can now take your HTML, CSSs , AJAX scripts pack them up as a single file (AIR) and lo and behold deploy them on the desktop. You even get these nifty start menu and desktop shortcuts :) The reason not to dismiss this a complete waste of time - is that what we actually see here is another example of a trend to convergence web and desktop UI architectures and programming models. I say "another" because coming from the desktop direction Microsoft is doing pretty much the same thing. WPF brings the web-programming model with its markup (XAML) and "code-behind" concepts to the desktop as well as pushing the same model to the browser with WPF/E . The difference between Microsoft's and Adobe's solutions is that, Adobe is coming from the web-side and, as I said, Microsoft is coming from the Desktop side - both companies are striding toward the same goal - and what we are left with, is yet another technology war
Follwoing several months of hard work, IASA - The International Association of Software Architect, has finally published on-line the Architect Skill set library. What is the skill set library you ask ? Well, it is what you get when a few dozens of architects (your humble servant included) author papers relating their experience on the different aspects of the architect’s role- everything from design, infrastructure, quality attributes to human dynamics (soft skills)
I should probably mention mention both Nicole Tedesco and Dana Gerow who orchestrated all this effort, and whiteout whom, I personally doubt if the project was now online
By the way, what is even more interesting is that IASA is organizing training material based on this body of collective knowledge - and if this works well we’ll finally have a comprehensive course to train new architects. Meanwhile, you can find the all the PDFs online at http://www.iasahome.org/web/home/skillset
P.S.
As I mentioned earlier I also wrote a paper for this project – on Views and Viewpoints which you can download directly from here
I was going to try to explain why it took me so long since I've posted the last pattern draft on-line when I saw that a couple of my fellow Manning authors already did that. See Roy Osherov's "Writing a book is like developing Software" and Fabrice Marguerie's "My Writing Experience". I have similar experience here -there are a few commonalities for software writing and it seems that the counter measures of shorter iterations, refactorings (which I guess writers know as rephrasing) and increased inspections seem to work here as well.
Finally, I am back to writing new stuff and I am completing Chapter 4 now. Chapter 4 deals with SOA security pattern, and I've decided to release the "Service Firewall" pattern as free draft. Note that it is a draft and it can change by the time it gets to publication for example the Edge Component, which I published a few months ago already went through some extensive rewrite (maybe I'll post the updated draft..)
The Service Firewall helps deal with malicious "service consumers" and protect the services from several types of attack including for example XDoS (XML Denial of Service), malicious content, preventing leaking private information from the service etc.
You can download the draft for Service Firewall pattern from here .
Following my post on SOA definition, Alex left the following comment
“One question - how can an organization achieve "agility" through an SOA, if not through "re-use"? Isn't re-use really the ROI for implementing a Service?”
The way I see it, Agility means the ability to change rapidly and it doesn’t have to mean reuse –for instance, it can come from the ability to replace a component without disturbing other dependent components – though you can say that this is reuse as you are reusing the interface (contract).
When you replace or update a service you may reuse some or maybe even all of the previous version of a service – as long as the context for that service didn’t change significantly – if it did the granularity of the reusable components will be much smaller than a “service”.
I would also note that I think there’s a difference between reuse and use. If you take the same ordering capabilities and you include it in two business processes that just using it. I’ve seen reuse of services in product companies where services were reused with few modifications between two or more solutions but this isn’t very common.
Regarding the ROI of SOA –That doesn’t have to be reuse or just reuse it is also things like easier connectivity so that you can integrate faster with partners or new components that are developed . Another way to measure ROI is measure the gains in easier replacability and adaptability so you can faster respond to changing business requirements (e.g., changing what counts as a VIP customer without letting any of the service’s consumers that something changed).
Udi has some comments on my SOA definition. Udi says that the definition I provided does not support the notion of publish/subscribe using topics for services. My answer to this is yes and no :)
First thing first, I never said (or at least I never meant to say) that contracts are limited to only incoming messages. Contracts contain incoming and outgoing messages. I probably should have stated it more clearly though.
Udi says “Contract: Who owns the message type being published? The publisher or the subscriber? Common SOA knowledge would say that the message belongs to the contract of the service that receives it”
I don’t know who is “Common SOA knowledge”. In my opinion, this thinking is a wrong “even” for request/reply. The reply message belongs to the service the sends the reply
Regarding Endpoints – if the subscribers go to a topic as in “ServiceName\TopicName “ then yes I would call that an Endpoint since this is a well known address consumers (subscribers) go to find messages published by a service
Regarding consumers Udi says “ Is the publishing service “using” the subscriber when it publishes a message? I don’t think so, and the subscriber definitely isn’t using the publisher at that point either. So, we’ve got some inter-service message-based communication going on and it isn’t clear if we even have a service consumer. In fact, if all a service ever did was subscribe to some topics, and publish messages on other topics, it looks like we’d have very loose-coupling but be straying from the common SOA wisdom.”
Maybe that’s just semantics but I don’t see why the subscriber isn’t using the publisher- The publisher publishes a message on a topic this is part of its offering. The subscriber chooses to consume that information and maybe do some stuff with that – possibly publishing some other messages. That’s a “using” relationship to me.
Nevertheless - SOA is not a synonym for "Distributed system" so there are cases when distributed components that communicates through messages aren’t SOA. For example publish/Subscribe using topics where the topics are common and shared between components so that multiple services can publish on the same topic does not, in my opinion, fall under the definition of SOA . This doesn’t say that this is a bad architecture in any way – but it isn’t SOA either.
As I said in the “What is SOA posts” for an architecture to be SOA you need autonomous components , that publish and accepts messages defined in contracts, delivered at an endpoint and governed by policies to service consumers – no more, but no less either.
I've been talking about SOA for a while now it's finally time to (try to) properly define it
I've publised this as a 5 posts on my DDJ blog and I thought it was good enough to be publised as a single whitepaper:
"Service Oriented Architecture or SOA for short has been with us for quite a while. Yefim V. Natiz, a Gartner’s analyst, first talked about SOA back in 1996. However it seems that only in the recent year or so SOA has matured enough for real systems based on the SOA concepts to start to appear – or has it? There is so much hype and misconceptions surrounding SOA that we first have to clear them all up before we can explain what SOA is – let alone identify who really uses it...." (Download full PDF (670K))
You can see additional presentations and papers I wrote here
[originally published in my DDJ blog]
You may have read my BI and SOA post where I suggested using EDA within SOA to solve the BI/SOA impedance mismatch. Jack Van Hoof made the following comment on that post:
Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0. Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other….
I think that "Synchronous RPC" is not a very good (or useful) definition for SOA (see my series on what is SOA anyway). Nevertheless, I also think that even if all you have is synchronous request/reply you can still implement both asynchronous messaging and EDA How can we implement Asynchronous Messaging?
Option 1 Duplex Channel
Let’s say you are a service consumer. You send me your request. Instead of a reply I just acknowledge you that I got the message. I put the message into a queue and process it on my "spare" time. I then call you with the answer.
Option 2 One way Channel
Again you send the request. Instead of a reply, I give you a token or a ticket for the answer. When you think it is time, for example when the time promised in the SLA elapse (or whenever), you call me again, give me the ticket, and I look up the answer and give you your reply. If we hide all this protocol inside some software infrastructure the applications can see asynchronous messaging even though we have synchronous request/reply on the lower levels.
Okay, so what about Events? How can we publish events just using request/reply. The previous example would not work since we can miss out on important events?
If you are reading this blog -- chances are you already have the answer working on your computer -- yes, it is RSS. Think about it using an RSS Reader that pulls the server that publishes this blog you reach out using synchronous request/reply and get all the posts (events) that were added since the last time you asked.
There are a few additional architectural benefits for working this way. For one the service does not have to manage subscribers. Secondly, the consumer doesn’t have to be there the moment the event occurred to be able to consume it -- and the management and set up is easier and simpler than using queuing engines
Jack Van Hoof left the following comment on my post on BI & SOA:
"Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0.
Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other. In my article 'How EDA extends SOA and why it is important' I explained the differences between the two patterns and when to use the one or the other.
Because of the completely different nature and use of the two patterns, it is necessary to be able to distinguish between the both and to name them. You might say making such a distinction is a universal architectural principle. Combining both of the patterns into an increment of the version number of one of them is - IMHO - not a very clever act. I believe it is appropriate and desirable to use the acronyms SOA and EDA to make this distinction, because SOA and EDA are both positioned in the same architectural domain; SOA focusing on (the decomposition of) business functions and EDA focusing on business events."
I agree with some of the things Jack says but not all of them. The way I see it EDA and SOA are two different architectural styles- but I guess that I see it a little different than Jack does
EDA is an evolution of the publish-subscribe style - and can exist independent of SOA i.e. you can implement it with other architectural styles SOA is an evolution of the component based development style which puts an emphasis on interoperability and adaptability.
However I don't agree that SOA is "Synchronous RPC". That's just the initial "wave" of SOA implementations since synchronous interactions are easier to grasp and implement. I think that adhering to SOA principles you can also implement additional interaction patterns including, asynchronous messages, publish/subscribe and EDA (and combining SOA with EDA is what I suggested for solving BI in an SOA)
I don't like the SOA 2.0 term as well - but that's just because I don’t see a need for defining a new term :)
I'll post more about this once I finish the "What is SOA anway" series on DDJ where I explain the way I see SOA
[based on a few posts from my DDJ blog]
Implementing Business Intelligence (BI) solution on top of Service Oriented Architecture (SOA) is not a simple feat. A recent survey by Ventana Research shows that "...only one-third of respondents reported they believe their internal IT personnel have the knowledge and skills to implement BI services.". There's a good reason for that since there an inherent impedance mismatch between BI and SOA which takes some effort to overcome. The purpose of this paper is to look to explain the problem as well as look at the possible solutions.
Service-Oriented Architecture is about autonomous loosely coupled components. These traits gives you lots of benefits such as greater flexibility and agility but it also means that services have private data. Data that you don't want to expose to the outside as exposing it will decrease autonomy and increase coupling. This is why services only expose data and processes via contracts rather then exposing their internal structure.
That is all fine until you start to think about business intelligence. The cornerstone of any business intelligence initiative is gathering, collecting and consolidating data from all over the place. Once you have the data, you can use tools to analyze it, data mine it, slice, splice, aggregate, and whatnot. Traditionally BI builds on ETL (Extract, Transfer, Load) which goes directly to the database of the involved sources.
And here lies the problem: On the one hand we have services that want to keep their data private, and on the other we have a datamart or warehouse that wants that data badly.
What are our options?
- If you go with traditional ETL, you introduce coupling into your service.
- If you only rely on contracts that were constructed for business processes you may be missing out on important data.
- If you build a specific contract that exposes "all" the data you are back at the point-to-point integration -- solving point-to-point integration is one of the reason we want SOA in the first place.
The second option seems to be the most reasonable choice of the three -- but it also has several problems. One problem is that the BI needs to know about all the contracts. The second was already mentioned -- important data might be missing. The third problem is that the BI system need to fetch data from the services which means it may miss out on data in the intervals between request. On the other hand, too frequent requests and you can congest your network easily as well as cause DOS on your own services.
Clearly we need a fourth option
In my opinion, the best way to tackle BI in SOA is to add publication messages into the contract. By "publication messages", I mean that the service will publish its state either in a periodic manner or per event to anyone who listening. This is a service communication pattern which I call "Inversion of Communications" since it reverse the request/reply communication style which is common for SOA.
To make the solution complete, you can add additional requests/reply or request/reaction messages to allow consumers to retrieve initial snapshots. Following this approach, you get an event stream of the changes within the service in a manner that is not specific for the BI. In fact, having other services react on the event stream can increase the overall loose coupling in the system - for instance by caching results of other services
Why is this better than the other three approaches? For one , you can get a good picture of what happens within the service. However the contract is not specific for the BI and can be used by other services to cache the service state (thus increasing their own autonomy), for reporting (you can see an early draft of the aggregated reporting pattern), and for BI purposes. By working against a steady stream of events, the BI platforms can Analise treands, keep history and get the complete picture they need.
The approach above is sometimes referred to as "Event Driven Architecture" (EDA) and while I (and others) see EDA as another facet of SOA, not everyone agrees. Gartner, for instance, sees EDA as another paradigm and SOA just for request/reply, or client/server. Recently, however, they published a paper that calls the approach described here as "Advanced SOA". I tend to agree more with the "advanced SOA" definition and don't see a contradiction with EDA and the SOA definitions. We are still using the same components and the same relations only adding an additional message exchange pattern into our toolbox.
A note on implementation: If you are implementing SOA over an ESB that is rather easy to implement as most ESBs support publishing events out of the box. Using the WS* stack of protocols, you have WS-BaseNotification, WS-BrokeredNotification and WS-Topic set of standards. If you are on the REST camp, then I guess you will need to implement publish/subscribe by yourself.
Once you have event streams on the network, The BI components grab that data scrub it as much as they like and push it to their datamarts and data warehouses. However, event steams can also enable much more complex and interesting analysis of real time events and real time trend data using complex event processing (CEP) tools to get real-time business activity monitoring (BAM)
You can also get post as as a presentation down loadable from the papers section on my site or directly from here. (The download is about 3MB.)
I've stumbled upon a presentation by Ron Jacobs on the Software architect's role (via Shahid Sah's blog) called Architects and the Architecture of Software. In this presentation Ron compares the architect's role for that of an explorer, an advocate and a designer.
While I can go for the designer bit - although I don't like the heavy analogy to building architects (I know, I know I have that as well in my software architecture presentation - but at least it is no longer there in the next version)
However I would personally replace advocate with a mentor and explorer with a polymath or renaissance man and add a leader and visionary as well (although Ron mentiones that as part of the discussion on explorer)
Advocate is someone who observes, listens and gives advice - but a mentor is someone who helps others reach the right decisions and help them learn and evolve. I think that has much more value. I want a Socrates not an Alan Dershowitz on my team
An explorer looks for new grounds and is a bit of a visionary - but a renaissance man is both knowledgable and inventive. As a development manager I rather have someone who knows what he is doing, understand the wider perspective and can find solutions to my problems - and not someone who would take me on a road to uncharted territories. I'd take Leonardo Da Vinchi over Columbos ( who accidently gave the competitive edge to spain and didn't even know it) any day.
A visionary and leader is also important - you want someone who is able to look far and that can help your team get there- I guess that is somewhat akin to an explorer (in the sense Ron mentions) but again I'd rather have a Martin Luther King than a Columbos (though lucky wouldn't heart).
But hey, that's just my opinion 
One unique aspect of SOA vs. other architecture styles like Object Orientation , Client/Server or even 3-Tier architecture is that it is built for highly distributed systems. Each and every service is a sub-system in itself it can run on its own machine and be located everywhere in the world . Many times, the service itself needs to be distributed in its own right. One reason to use distributed computing inside the service is computational intensive tasks.
One of my recent projects was the development of a biometric platform. The platform can be used for many usage scenarios. A simple scenario is an access control systems - e.g. authorize entrance into a secure building or area. This is a relatively simple scenario as you usually only have to deal with few thousands of people and as a person requests entry she also declares who she is (e.g. using an RFID card with her ID). In these cases you can go to the database, lookup the appropriate record , run the biometric algorithm or algorithms and verify the person is who she says she is. However the same platform also has to work for other, much more demanding and computing intensive scenarios. For example consider a forensics scenario where you have a fingerprint collected at a crime scene, in this case you don’t know who the person you are looking for is, and you have to run your search on basically all the database which can contain millions of records. Keep in mind that when you match a biometric template1] you calculate the probability of a match (based on the internal structure of the template) and that each template weights about a one kilobyte you quickly realize that this can be quite a CPU intensive task.
Sometimes when you develop you SOAs you will have algorithmic tasks or other computational heavy tasks such as the one mentioned above and the question is
How can a Service handle computational heavy tasks in a scalable manner?
You can get the full pattern from here
[This is an early draft of one of the Performance, Scalability and availability Patterns from my SOA Patterns book]
I've added a section called SOA Patterns on the site while holds the current draft for the table of contents of the SOA Patterns book I am writing. The section lists the problem each pattern addresses as well as links to published patterns. Also, you can use this to monitor my progress (patterns that already have their problem written down already have drafts; the others are in-progress or not started).
I am currently working on chapter 4: Security & Manageability patterns (not counting delays mentioned in the previous post).
Also, as I think I've already mentioned, I'll make public at least one pattern per month, if you are interested in a specific pattern in particular (from those which are ready - now chapters 2&3) drop me a note and I'll publish the one that gets the most votes
My editors at manning think that my chapter 1 of the SOA patterns book is not good enough.
They basically say that the chapter talks about too much theory vs. the other chapters which contain much more down-to-earth stuff (e.g. Edge Pattern, Aggregated Reporting Pattern, Decoupled Invocation Pattern ). Also they’ve said that I spend too many pages explaining what architecture is or taking about distributed system before I get to SOA – which is the topic of the book.
The way I see it, understanding architecture and distributed systems is essential to understanding SOA (from the development side i.e. when you want to design and build services). For example the discussion on quality attributes explains how you can use scenarios to find architectural requirements (and each pattern then has a section on relevant scenarios to help you find if the pattern is applicable to your needs)
I would be very interested in hearing what you have to say (either as comments here or emails to me) about the Chapter’s structure and content (considering most of the books will be patterns like the Edge pattern)
Thanks in advance
The business rationale behind going on the SOA road is increasing the alignment of the business and IT, so we divide the business into a bunch of business services and everything is just fine. However the minute we start diving into the SOA implementation details we are swamped by a horde of technologies, cross-cutting concerns (auditing, security, etc.) and whatnot.
For example, in one project I was involved with, we implemented an SOA over a messaging middleware (Tibco's Rendezvous). Just when everything was fine and dandy - along came another project which could potentially use few of the services. Well, almost, it needed a slightly different contract and it also used completely different wire protocol - WSE 3.0 (Microsoft interim solution for the WS-* stack before Windows Communication Foundation). And that's just one simple example - cross cutting concerns and implementation details are everywhere. The question is then:
How can you handle cross cutting concerns like multiple technologies, protocols, changing policies etc. while keeping the service's focuses on its core concerns - i.e. the business logic.
You can get the full pattern from here
[This is an early draft of one of the Service Structural Patterns from my SOA Patterns book]
I've posted a new paper on the site. Designing a good architecture is not enough. The paper explains how to make sure the architecture is both relevant ands followed throughout the project. You can download the paper directly from here.
I am going to present SOA in one of our internal forums next week - so I thought it would be a good opportunity to dust-off my SOA presentation and give it a little face lift. You can download a copy from the papers and articles section (or get it directly from here).
As always, any comments are welcome
The draft for the first chapter of my SOA Patterns book is available on-line from Manning Publications Co.
The first chapter talks about software architecture and the inputs the architect can/should use to design one (emphasizing Quality Attributes); Explains the challenges of distributed systems and takes a look at the SOA from an architectural perspective.
You can download the chapter from here
Any comments are welcome (you can also leave your comments at soa@rgoarchitects.com)
Roy Osherov recommended this site today - but he also urged me to write more frequently.
This is probably a good opportunity to explain how posts are divided between my 3 blogs
-
First theres the blog on Dr. Dobb's Journal. This blog is published on the "Architecture & Design" section of DDJ portal. I blog there about 3 times a week. Jon (my Editor @ DDJ) prefers a steady stream of blogs over longer posts which means that I break down large subjects (like OO principles, fallacies of distributed computing and the currently running series on the Architect's soft skills) into many parts.
-
The second blog is a new one on Microsoft's Israel blogs site. The aim of this site is to bring Architecture content in Hebrew to the Israeli architects (As can be imagined, most of the technical content available is in English, I thought it was important to generate some content in Hebrew as well)
-
The last blog is this one. My current plan for this blog is as follows
-
- Cross posting selected posts from the DDJ site
- I am posting here complete articles made by editing and aggregating multi-part blogs posts (again such as the fallacies etc.)
- Pointers to presentations and articles I publish
- In the near future, I'll start posting bits of my upcoming SOA patterns book (I am currently writing chapter 3). I've already documented 8 patterns (of more than 50 patterns and about 30 anti-patters). I plan to publish here at least some of the patterns here for review (I am still crossing the t's and dotting the i's with my publisher but I expect this to be finalized soon)
so Roy, does seven (7) posts in seven days (including 3 on Ms Israel site, 3 on DDJ and this one) qualify as posting often enough? :)
[Will also be cross-posted on my DDJ blog]
Working on my SOA patterns book, I thought of this rule for contract versioning which my shameless ego wanted to dub Arnon's Contract Versioning Principle. I was happy playing with this thought, until I realized that there isn't some profound new understanding here, this is just an application of LSP for service contracts.
Liskov Substitution Principle (LSP) which I recently blogged about here as part of a series of blogs on Object Oriented Principles, basically states that a subclass should be usable instead of it parent class. To put this in other words you could say that a subclass should meet the expectations that users of the parent class have come to expect from the parent class's observable behavior.
So LSP applied to SOA would state that:
When changing the internal behavior of a service, you don't need to create a new version of the contract if for each operation defined in the contract the preconditions are the same or weaker and the postconditions (i.e. the outcome of the request) are the same or stronger or in other words the to retain the same contract version, the new version of the service should meet the expectations that consumers of the service have come to expect from the old version's observable behavior
For example, let's say you have a customer service and the contract lets you get a customer's VIP status. If you changed the way the VIP status is calculated (e.g. in the old version the customer had to have 1 million dollars in her account, but now she must have 10 million dollars) there's no need to create a new contract version. However if you introduced a new level of VIP status (e.g. 1 Million = Gold, 10 Million = Platinum) you do need a new version for the contract
I've added a new section on the site www.rgoarchitects.com/Papers to allow easy access to all the papers, presentations and articles I published (and will be publishing e.g. I'll add a paper on architect soft skills in a month or so etc.)
Over the last few months I've posted a series of blogs on DDJ that cover the basic Object Oriented principles (e.g. Single Responsibility Principle, Don't Repeat Yourself, Inversion of Control etc.).
I've assembled all the posts into a single whitepaper which you can get here.
Also you can download the same (plus a little more) material as a powerpoint presentation.
[Crosspost from my DDJ blog]
When talking about multi-tiered architectures, we need to remember that the tier boundary is significant. The tier boundary is where distribution happens and if you remember the "fallacies of distributed computing", you know not to take that lightly.
A tier is a physical boundary (versus an Edge in an SOA which is a logical boundary, for example) and the implications are numerous. For instance, you need to consider:
- Trust--who do you let in?
- Security--what do you send out?
- Performance--you need to serialize to pass the boundary, and remote data is expensive to fetch.
- Availability--what happens if you crash?
- Manageability--can anyone see what's your state? Help you recover?
- Temporal coupling--can you afford to make synchronous calls?
- and many similar questions.
Yet many times people think passing a tier is as simple as passing a logical layer. I should know. I made this stupid mistake more than 15 years ago in one of the first distributed systems I designed. I planned this beautiful separation of the UI controls from the business logic (I didn't know it was called "MVC" and that someone else had figured it out ages ago, so I was pretty proud of myself). When you clicked on a button you just used metadata to say that BL should catch it. I had all this wonderful "infrastructure"that handled passing the call to its destination.
But then we wanted to take this n-layer application and put the BL in an "application server" which will handle multiple clients. Oh--now we need to move events over the wire , handle calls from multiple unrelated clients, pass a lot of data back and forth, and what about security... you can imagine the fiasco.
Thus, as Niels Bohr once said, "An expert is a person who has made all the mistakes which can be made in a very narrow field." But you don't have to make the same mistakes. Just remember that a tier is a natural boundary. You know what? You should probably even want to consider it the edge of a cliff at the end of your application--and be careful not to fall down.
[crosspost from my DDJ blog]
In a comment to my previous post on Architecture vs. Design, Yoni said:
It seems you are categorizing technical issues as architecture and logical issues as design. I think Martin Fowler's definition of "Making sure important things remain decoupled and easy to change" transverses both categories and is easier to follow.
I have a few things to say about this.
First, I don't categorize technical things as "architectural" and logical ones as "design." What I do say is both "architecture" and "design" are types of design where with one you focus on the wider aspects of the solution and quality attributes, while with the other you focus on local and functional aspects.
I don't see how the definition Yoni brings up is a better way to distinguish between the two? Who is to say what is important and what is not? Isn't decoupling an important trait at all level including so called "detailed design" level (e.g. utilizing dependency injection at the class level will give you better testability). Moreover, decoupling is important but sometimes you need to trade that to be able to satisfy a more prioritized quality attribute (if you want to meet a projec's quality, budget and schedule targets); see my definition of what software architecture is.
Another thing is that it doesn't matter that much where the line between architecture and design passes. The distinction between architecture and design is a semantic one that reminds us that the design of a system needs to be done in several levels of abstraction (provided the system is not too trivial). We need to abstract certain aspects of the system in order to be able to grasp the big picture. You cannot (well, I can't anyway) think at a 100 man-years project. You can only think about it at the class level and understand how everything will work together. Again, architecture is there to remind us to focus on a level of abstraction that lets you deal with non-local decisions and to make sure quality attributes are met if we cross the line to design - No biggie.
[edited version of post I made on Dr. Dobbs Portal]
Back in April I provided a definition for "architecture" as one of my first posts on DDJ. I also promised I'll talk about the distinction betwen Architecture and Design. Well this time is now.
When I try to think about it. I see two base criteria to distinguish between the architecture and design:
- Design deals with local decisions, where architecture is broader. For instance, you "design" the interfaces for your classes, but you "architect" the division into tiers.
- Design is mostly about the functional requirements, while architecture is mostly about quality attributes. You design how a specific workflow will fulfill a certain use case, but you architect the solution to the system's availability.
It is probably quite evident that this distinction only provides blurry borders between architecture and design; for example, when you have a multi-tier solution and you "architect" the UI and say it will implement MVP pattern. Can this be considered local decision and thus design or is this the overall decision (for the UI) and thus architecture?
The way I see it the exact cross-point from architecture to design is not that important. The point in talking about two distinct activities in the development process is to maintain separation of concern. you need to handle both to make sure a solution will actually work whether you do a little design while architecting or do a little architecture while designing really doesn't matter. Also architects should be involved in both activities anyway...
Udi Dahan, which is one of the best architects I know has recently created an excellent course on SOA - you can find the details of the syllabus on Udi's site .
I had a chance to work with Udi in the past and the solution we implemented utilized many of the patterns and techniques Udi covers in his course - so these are not just nice theories but rather real stuff that works
[crossposted from DDJ]
Yesterday I attended an interesting presentation on SOA by Dr. Donald F. Ferguson, chief architect for IBM's software group.
I was happy to hear him validate some of my thoughts on SOA (e.g., workflows are better kept inside services rather then outside, transaction boundaries should be inside a service, and so on), and introduced a couple of things I didn't know much about (for example, OSGi, a SOA platform for networked devices that's not based on web services. He also presented some nice insights (for instance, looking at the middleware as an infrastructure service and thus nicely unifying SOA and EDA)
On of the insights Donald presented was the use of heuristics as an aid to modeling and validating architectures. Some of the heuristics he mentioned include:
- Occam's Razor -- avoid needless repetition
- Don't create something new if you can compose existing stuff to get the same result
- externalize volatility -- don't put in the code things that are likely to change
- Focus on "name,value" programming not "offset programming" -- make things easy to understand
- Different is hard
If you look at heuristics as an abstraction of experience, they can provide as a good tool for keeping yourself on the right track. Some heuristics are universal (maybe the ones mentioned above and a few others like "simplify, simplify, simplify" or "the original statement of a problem is probably not the best one or it may even not be the right one"), but the problem is, as always, deciding (in advance) which heuristics to apply to a problem.
If you interested in using heuristics as an architect tool you may want to look at " The Art of Sysytem Architecting", by Mark Maier and Eberhardt Rechtin. The book discusses the architectures of different system types (collaborative, IT, Manufacacturing, etc.) and provides heuristics for each of these systems.
Heuristics are a good tool to use when you design an architecture and in a way the different design principles (e.g., the single responsibility principle) can also be considered heuristics. Nevertheless it is very important to verify designs by additional methods like code and formal evaluation and not rely on heurisitcs as the only tool.
Last week I published a 3 part article on O/R mapping on my blog @ Dr. Dobb's Portal. The paper describes the benefits and costs of using O/R mapping as well as recommend when O/R mapping should be used.
Here it is as a single whitepaper: Architecture Dilemmas - OR Mappin.pdf (228.78 KB)
[Crossposted from DDJ]
Following the series of posts that discussed whether architects should code (see Part I, Part II and Part III), I thought I'd expand on how I see the software architect's role.
To be a good architect, a person needs to have experience taking a project from inception to production and maintenance, they should have project management experience, and have spent time working directly with customers. On top of that, know-how and experience is needed in design and technology. Having a diverse experience is what enables the architect to have a wide view on a project, understand the need for pragmatism, better understand stakeholders concerns as well understand the implications of design and technology decisions (on scheduling and budgeting, for instance). Having a wide experience also puts architects in a unique position where they can really help make projects meet their goals.
Software Architecture deals with the system's quality attributes. This makes the architect the ultimate person responsible for making sure the solution meets these quality goals (and I don't mean quality as in low bug count, rather the soundness of the solution and it ability to address the various stakeholders concerns).
To make this quality claim a reality I tend to take a holistic view in regard to the architect's role, which, simply put, means that the architect needs to do whatever it is needed to make the project go forward and succeed. Taking this direction toward quality is only possible if the architect has the wide range of experiences mentioned above.
How does that translate to real life? Here are few examples for "non-architect tasks" from my experience:
- On one project the system engineering team was working in the wrong direction in regard to analyzing requirements (this cost us six months of delay). I helped introduce the concept of Use Cases and created the project's initial use case model. (You can read the insights I have from my experience in my paper Use Case Methodology for large systems (pdf) )
- On another occasion one of the project managers was trying to evaluate how a certain technology will help advance the project. I helped her construct the WBS of the activities needed to complete the needed functionality (assuming the technology is in place) and the helped her create the estimates.
- On another project we had performance bottlenecks related to the technology used (ESRI, displaying vector maps with high refresh rates for displayed objects). I got together with another architect to pin-point the problem, diagnose it, and came up with a solutions (mapping ESRI temp files to a RAM-disk).
- I worked on a project where time was running out and a milestone was looming fast. I helped introduce the project to some SCRUM-like techniques (why only "SCRUM-like"? Remember that Rome was not built in one day either) and by working closely with the PM helped the team reach the milestone successfully (indeed not all the feature were completed--but the ones that did work were the features important to the end-user which made the milestone a success).
And the list goes on and on. And yes, coding can also be a part of this list (though for me it usually only translates to writing short proof of concepts--and for others it may mean coding some of the tasks with the team).
To sum up, a good architect is not just a lead developer on steroids. An architect should have a much wider range of capabilities and experience. Architects are "enablers". They should use their capabilities to help advance the solution and ensure its overall quality.
I have amassed more than 30 patterns related to SOA (e.g. SOA Patterns - Decoupled Invocation and SOA Patterns - Aggregated Reporting which I previously published here). I have patterns around security, availability, scalability, composition, adding a UI etc. Some of the patterns are original (I think) and some are based on other peoples work.
I am trying to decide whether or not it would be worthwhile putting all these patterns into a book. Writing a book is a very time consuming task (or so I am told) - so I thought I'd run a quick poll between the readers of this blog to see how many of you would be interested in reading (and buying) this book if it will get published.
I know this is not a representative crowd - but it can give me a (very) rough idea on the interest in such a book.
Please send any comments (comments like "forget it, no one would ever want to read anything you write" are also ok) to soa@rgoarchitects.com (or leave a comment here)
Thanks in advance - Arnon.
[Edited version of post in DDJ]
Since I have been blogging for about a year now on this blog and now also on the DDJ blog. I think it is time to try making something with more two-way communications.
Consequently, I am going to run a little experiment for a few weeks and see how it goes.
The idea is as follows: If you have an interesting architectural or design dilemma, drop me an email at ask@rgoarchitects.com I'll pick one issue per week and post (on the DDJ blog) the dilemma (anonymously) plus voice my opinion (and/or suggested solution)--and then everyone else can chime in with their comments and insight which hopefully will shed some light on the subject.
I'd be interested to hear both your opinions on this initiative and, of course, interesting dilemmas you are facing. Again, send your dilemmas to ask@rgoarchitects.com)
[crosspost from DDJ]
Reading the comments on my previous two posts on whether architects should code (here and here) as well as the comments on Johanna Rothman's posts (here, here and here) leads me to a few observations:
The first apparent thing is that the issue is a very loaded. Some people believe it is essential for architects to code, while others (like me) believe that their time is better spent on other issues. (That said, it seems that a small majority of the commenters think architects should code as part of the development team--at least for feedback purposes if nothing else.)
There is a wide consensus (me included)that architects should know how to code and have extensive experience in coding. It is also agreed that architects should be involved in the project--that is, not just drop off the architecture, then disengage.
I still believe that when the project is big enough (that is, big enough to warrent more than one team working on it) the project is better served by the architect getting involved in all the teams, rather than participating as a developer in one of them. If you are an architect and develop as part of the development team you are (or should be anyway) committed--meaning you need to deliver the piece of code under your responsibility at an acceptable quality level as other developers. Which is exactly why you would be less likely to deliver on your responsibilities for the total quality of the project. I assume some of the differences in opinion can be attributed to disagreement on what software architecture is , at least when compared to design).
I also think those who think architects must code see the architect as some sort of a lead developer again. I don't buy that. The architect's role is much broader than that (see also this post by Kevin Seal, which also discusses this issue). I see a holistic view of the architect role, which is making sure the product is delivererable. This may translate to the architect coding a module or two, but it can also translate to a lot of other things. Examples from my experience as an architect include preparing initial cost estimates, iteration planning, helping debug and testing, solving installation problems, analyzing requirements, conducting design and code review, design, and prototyping (yes, that's coding but as I said in the previous posts, that's not writing the production code and this is not having to meet deadlines etc.).
I also liked a comment by Graham Oakses on one of Johanna's posts :
My experience is that an architect is pulled between three poles--the product, the team and the client. The product pole pulls you towards managing the "conceptual integrity" of the design. The team pole pulls you towards mentoring people, helping them build skills, etc (which may mean consciously letting someone write code that you could do much better yourself). The client pole pulls you towards translating between the technical and the client domains (which is often where you get pulled into powerpoint). You need to trade these poles off differently on every project...
To sum up, the answer to "should architects code? " is like so many things in life is--it depends.
[Crossposted from my DDJ blog]
About the same time I wrote the post on whether architects should code, saying that architects should be able to prototype but shouldn't be part of the dev team (in the sense that the architect shouldn't get coding tasks that results in production code), Johanna Rothman wrote a blogpost that claimed architects must code .
Two days ago she posted a more detailed explanation of her view. I agree with most of the points she made:
- Architects need to participate in the project; that is, not be some outsider who just drops her architecture on the team and leaves).
- The best way to test a design is to code and run it.
- It is beneficial for architects to know to code.
- It is important that architects understand the implications of their decisions on the code and developers.
I don't see how architects taking coding tasks serves the greater good, versus their monitoring teams that code and making sure all aspects of the architecture actually fit the problem and work. Again, this may work on smaller projects, but probably not on larger ones.
You may also want to look at two related posts I made in the past
SAF Architecture Evaluation: Evaluation in Code talks about some of the ways architecture can be validated in code.
SAF Deployment: What to do when the architecture seems stable? talks about the architect's involvement in the project when they think the architecture is "finished".
A couple of points regarding the analogy Rothman uses--that is, architects who design bathrooms for hotels. Building architects are seldom a good analogy for software architects (I once used it as well). However, there are far too many differences (maybe I'll blog about that sometime in the future).
This brings me to the second point. This analogy doesn't serve Rothman's point well since building architects never actually participate in laying down brick or installing bathrooms. The fact that hotel bathrooms are not comfortable means that this quality was low on their priorities. In any event, verifying if a bathroom is usable--you don't have to install it just use it. (If you do take the analogy, you don't have to code it just stick around to see what's going on
Here is another SOA pattern from the list of patterns I am publishing.
One of the core goals of going with SOA is to enable loose coupling. The request-reply communication pattern, which is very prevalent, inhibits this decoupling. The problem is for the caller or consumer of the service - the consumer service is dependant on the timely response of the called service for its normal operations. To help elevate the consequences of this dependency the service that is consumed should maintain QOS (Quality of Service) as part of its contract (it doesn't have to be part of the machine-readable contract but it needs to be defined and adhered to). Consider for example an on-line music store. On a normal business day that can have few thousands of purchases nicely distributed around the clock. And then when a new <name your favorite band here> album debuts they can have much higher peaks than their usual requests load. They still need to be able to handle all coming requests or the (potential) buyers will take their business elsewhere.
How can I maintain a level of QOS, handle peaks and high-loads without my service failing?
One option is to estimate the peak loads and get enough computation power to ensure you can handle them – this causes problems. One is a problem of waste you can have machines just sitting there twiddling their thumbs so to speak. However the idle computers have purchase, maintenance and operational costs. The other problem is unexpected loads (e.g. Harry Potter craze for an Amazon-like site) – the estimated load might not be enough.
Ensuring QOS gets even more problematic when some of the actions performed in the service access resources/services that are not in the under the service control (- e.g. taking to a credit card clearing in the e-commerce example mentioned earlier).
Another issue that needs to be take care of is prioritizing requests – a Service most likely handles several types of requests – not all of them need the same level of QOS. You can set the QOS for according to the most demanding request type – but then you may need more resources.
Decouple the invocation- separate the reply from the request: Acknowledge receipt in the edge, pass incoming request to a queue, load-balance and prioritize behind the queue.
Making the Edge acknowledge the receipt of the request (for our e-commerce example this can translate to "Your order has been received and is being processed, you would get confirmation email when the transaction completes") allows hiding operations that take long time from the service consumers (be that other services or end-users).
Writing requests to the Queue is a relatively low-cost operation that can be performed fast thus allowing handling request peaks. The actual handling of the incoming requests can be performed more slowly according to the available resources of the service. The load balancing can be done by setting different number of readers working against the queue.
Making the Queue a Priority Queue (or having several queues according to priority) allows for maintaining different levels of QOS for different message types.
Decoupled Invocation can be combined with the Gateway pattern to allow scaling out the service.
Decoupled Invocation is enhanced by the use of Correlated Messages pattern which helps relate the request and the reactions.
Acknowledge in the Service
Sometimes the initial response needs to involve some business logic and is not just an acknowledgment. In this case the Edge doesn't respond, it just passes the request to the service, the service sends both the initial reaction and the reaction.
Michael Platt talks about SOA vs. Web 2.0 He provides several links to blogs and article that basically claim that SOA is dead and long live the new king Web 2.0.
One thing I have to say is that if indeed the hype around SOA is starting to calm - this is a very good sign. Finally we can go about adding SOA to our toolset and use it when it is appropriate (not just because management has got to have an SOA). Also it can be a good sign that SOA is maturing.
Another point I'd like to make is that SOA and Web 2.0 are not really related - there is no reason why one should compete with the other. Why using an AJAX front-end makes it impossible to have Services in the backend (it may be appropriate to have a Client/Server/Service Scenario - where the front-ends don't hit the services directly (the other option is Peer/Service) - I may talk about these 2 mini-patterns in my SOA pattern series). Another example where SOA and Web 2.0 can work together is RSS. A service can expose its list of recent changes as an RSS feed (as well as providing the more "traditional" web-services API). Exposing an RSS feed can be an option to implement the Inversion of Communication pattern I mentioned in a previous post).
To sum things up - Web 2.0 may be more hyped today than SOA. Web 2.0 and SOA can co-exist and actually complement each other.
In any event I think we (as an industry) should focus more on delivering great applications and solutions rather than fight about whose trend-du-jour is fancier or sexier.
[Edited]
After writing about the example of using RSS for Service communication I stumbled today on RSSBus which is an effort to create an ESB on top of RSS protocol ...
[crosspost from Dr. Dobb's Portal]
Test Driven Development (TDD) is, in a nutshell, writing a unit test up front--making it fail, making it work , refactor, and repeat until the product is finished. (If this is new to you, read more at testdriven.com )
So with TDD you get a bunch of unit tests that are also proven as regression tests. That's pretty cool.
TDD also lets you work in small increments while maintaining the working code. That's even cooler.
And lastly TDD has a very good influence on design:
- It encourage loose-coupling. When you want make something testable you want to remove the dependencies it has so you can test it by itself.
- It makes you think about the interface of the unit under test--how is the interface going to look?
- It makes you think about how the unit under test would be used--for example, the behavior of what you are writing (or designing).
Sounds great to me. I think TDD is a great way to do the detailed design. You specify the results (interface + behavior), then implement that design. One thing I don't buy though is that TDD alone will produce an "emergent design" for the whole system. The way I see it is that you have to do some design up-front (assuming your system is not a trivial one) since TDD, being a coding technique, keeps you working at sea-level.
There's also a fundamental matter of scale--it might be possible in theory to start that 100 man-year project as a single object, then refactor it in baby-steps until you'd get the perfect system. I believe that if you don't work at a higher level of abstraction (vs. code), you will not be able to partition the system in a reasonable time. This was true when we moved from assembly code to higher level languages which enabled us to write much more complex software--and it is true today as we need to answer the ever-changing requirements of modern enterprises.
To sum up, TDD is good for testing and it is a good design methodology for the detailed design level. It can be used to drive the overall design on smaller project--but on larger systems we need additional methods and tools to cope with the overall design and architecture.
There
are 113 domain models over at eclipse.org site of all sorts of things.
I guess most models don't have any practical value (what will I do with a
metamodel of COBOL) but there are several interesting ones of things
like RSS, WSDL, and SDM
It is also interesting to note that the
models are expressed in several forms including MS DSL models
, UML
and images . The transformation from format to format was done by code.
(Found
via Steve Cook )
As promised, here is the first pattern. If you like this pattern but
you think there is something missing to gain better understanding
please drop me an email: arnon at rgoarchitects.com . Naturally any
other comments are also welcome :)
Getting
an SOA right is very hard, not so much because of the technical problems (we
know how to deal with those, don't we?), but rather it is very hard to figure
where to put the borders and keep the right business alignment. Assuming you somehow managed that, the real
fun begins - you now have to produce reports, dozens and dozens of reports.
Many reports will fall within the boundaries of single services (if you have a
good partition), however many reports will also require adding data from
several services. For example, in a Telco scenario, you may have a Customer,
Billing and Provisioning Service (a real-life example would have dozens of
additional services) now a customer is calling customer care and you want the
CRM to show everything about the customer what outstanding invoices does she
have, what equipments and services (GPRS, UMTS, friends and family etc.) she
got, what her status as a customer (loyal , VIP, senior citizen …) open service requests etc. Things get much
more complicated when you need to summarize or group data from multiple
services
How do you get a decent cross business entities report with
the data scattered about in all those services?
One
possible solution would be to create the report at the consuming end (e.g. UI)
visit all of the services involved then do all the grouping, cross-cuts etc.
This solution is not very good from the performance perspective (you need to
get more data then needed and you have to post-process it). It is also
problematic from the flexibility perspective each service involved has to
expose interfaces to get the data for the specific query (otherwise you
mobilize even more data).
Another
option is to go straight to the data, you may still need to hit multiple
database servers to get to the data but the performance will be better. The
problem is this is throwing your service boundaries down the drain and
introducing a lot of dependency.
A
third is to create interim Services ("Entity Aggregation") - this
works fine as long as you have real business reasons to do the aggregations
(there is an overhead with adding business logic to handle the aggregated data)
and as long as you only have few of those
(or you might end up with a single "service" with all the
business).
Create an Aggregated Reporting Service by building an Operational Data Store (ODS) to enable
creating sophisticated reports on otherwise dispersed data
The
ODS is similar in concept to a data mart e.g. data is subject based,
integrated, scrubbed etc. However, the
main differences are that the data is up-to-date and that there is little or no
history.
For
incoming data the Aggregated Reporting Edge performs the data transformations
from contract data into reporting data. The service updates the ODS by
scrubbing the data (can be limited unless the data has to go to a data mart /
data warehouse) and then integrating it and De-normalize into subjects. Incoming report request fill parameters for
the pre-prepared reports.
One
problem with Aggregated Reporting is that it is not a Business Service (i.e. it
is a technical solution rather than a business oriented one) - however since
unlike Entity Aggregation the data in Aggregated Reporting is Read-Only this doesn't affect
the business.
Aggregated
Reporting is easier to implement when combined with Inversion of Communication
Aggregated reporting with Data Mart/Data Warehouse
Instead
of just storing recent operational data, this version enhances the depth and
complexity of queries that can be executed against the service. The downside is
the increased complexity in setting up the data mart - both from the
operational costs perspective (e.g. additional storage) and from the design and
development perspective (you need to think about long term aspects, indexing
etc.) as you also need to scrub data and consider the structure of your schemas
much more carefully.
Sidebar:
Operational Data Store (ODS)
The
ODS is probably the best kept secret of data warehousing technology. It has
been around almost as long but it isn't as famous.
The
data in the ODS is operational - live data and not static data. The ODS can be
thought of the as the cache memory of the data mart / data warehouse.
It
is important to note that while it doesn't need the same amount of planning and
set-up as a data mart, an ODS still requires careful planning in order to bring
real business value.
The
figure below shows the classical usage of an ODS in an OLTP/Data Mart
environment.
Originally
it was thought there would be 4 types of ODS
Class
I - Near Real-Time synchronization of the ODS with operational data from the
OLTP databases. an implementation of
Class I is the preferred type for the Aggregated Reporting pattern
Class
II - Update the ODS every four hours or so
Class
III - Overnight updates of the ODS
Class
IV - the ODS is updated from the data
mart / data warehouse
In
reality there are more variants - for example a powerful (and complex to build)
option is to merge a Class IV ODS with one of the other Classes and get.
I
decide to write a short series of blog post on SOA patterns. These are not
patterns that are only usable for SOA, however, I have found them particularly
useful in implementing SOAs.
This
isn’t an exhaustive list of pattern - on the contrary I'll try not repeat
patterns which are well known (like
Entity Aggregation http://patternshare.org/default.aspx/Home.PP.EntityAggregation
)
I
am a little busy these days (e.g. I have to complete an architecture document
for one of my projects) - so this post will only introduce the (first batch of)
patterns . And the following posts (in the series) will expand on each one
(i.e. explain What to do, usage context, consequences etc.). Then, if I'll get good feedback maybe
I'll publish some more.
So,
what patterns are we talking about here?
Well:
- Gateway - How
do you scale a service without exposing too many endpoints?
- Inversion of
Communication - How do I get the data from other services without too much
coupling?
- Biztalkize -
How do I control volatile behavior inside the service ?
- Aggregated
Reporting - How do you get a decent cross business entities report with
the data scattered about in all those services?
- Transparent
Emergence - How do I know where to find a service?
- Decoupled
invocation - How can I handle peaks and high-loads without my service
failing?
- Orchestrated
Choreography - How do I expand the
behavior of hard-to-change service (e.g. legacy systems exposed as
services) ?
Well,
I hope this sparkle enough interest to make you follow the rest of the posts on
this subject :)
Uncle
Bob (Apparently Robert C. Martin?)
writes about Architecture as a secondary effect .
The
article postulate that : <Quote> - The main goal of architecture is flexibility, maintainability, and
scalability.
- But we have learned that the kind of unit tests and acceptance
tests produced by the discipline of Test Driven Development are much more
important to flexibility, maintainability, and scalability.
- Therefore architecture is a second order effect and tests are the
primary effect
I had not thought about this before this round table. Here we
were, a bunch of architects and designers, strongly debating the role and
procedure of architecture, and the conclusion we come up with is that all the
effort and struggle we go through results in a secondary improvement in
flexibility, maintainability, and scalability. Writing tests (writing them
first) has the primary effect
</Quote>
I
think Bob is missing/downplaying one very important aspect - and that is level
of abstraction.
- Many
agree (me included) that code is the final design artifact.
- Many
agree (again me included) that TDD is a powerful design technique (you may want
to check out TDD Misconceptions or Rocky Lhotka vs. the world as summed by Jeremy Miler)
However
since code is (obviously) detailed design you just can't go straight to code
(test code or otherwise). In order to cope with a large/complex problem you
need to tackle the problem at higher levels of abstractions first. Even more so
there may be a need to go through several abstractions levels before you start
coding anything. Unfortunately there aren't any real options to test models*.
In
my opinion you cannot escape designing architecture in other (non-code) models
for any, but the most trivial, system.
The
way I see it the correct approach is to
- Work
iteratively
- Test early -
i.e. make sure that the architecture designed really works as soon as
possible (see, for example, my post on evaluating architecture in code )
So
- Is Architecture a secondary effect?
No,
sorry, but I really, really don't think so.
*
There are some options to allow simulation and validation (i.e. tests ) of
models in the embedded world (e.g. http://www.embeddedplus.com/EmbPlusSMST.php
or http://www.ilogix.com/sublevel.aspx?id=286)
- however I find that these approaches don't scale well to IT problems (which
have a lot more variables and are usually much larger than embedded systems) or
even to complex embedded system. You just have to specify too much before you
get a usable simulation rendering the whole effort useless.
One
of the roles of the software architect is to act as a mentor/coach. Reviewing
some of the designs in one of my projects' teams it seems the time was ripe for
doing just that. Thus, last week I gave them a presentation on the basics of good OO design - which I thought might also be of interest for other people (you can download a copy
here - 312KB).
The
presentation starts with the 7 deadly
sins of software design:
- Rigidity – make
it hard to change
- Fragility –
make it easy to break
- Immobility –
make it hard to reuse
- Viscosity –
make it hard to do the right thing
- Needless
Complexity – over design
- Needless
Repetition – error prone
- Not doing any
It
is interesting to note that just yesterday I read an interesting piece on what
makes good design (i.e. looking from the positive side) by James Shore (found via Sam Gentile)
The
main part of the presentation demonstrates the 5 basic design principles
(drafted by people like Robert C. Martin , and Barbara Liskov
):
- OCP open-closed
principle - a class should be open for extension but closed for
modifications
- SRP single
responsibility principle - a class should have a single responsibility
- ISP interface
segregation principle - there should be separate interfaces for different
consumer types
- LSP Liskov
substitution principle - basically design by contract - a sub-class should
fulfill the same expectations its suparclass set
- DIP dependency
inversion principle - classes should depends on abstractions, class
consumers should depend on abstractions and abstractions shouldn't depend
on details.
These
principles are the basis for some of the
techniques widely used today - few examples include:
Inversion
Of Control - builds on OCP
Dependency
Injection - a mechanism to allow DIP
Contract
First - building on LSP,DIP
At
the end of the day following these principles helps managing classes
dependencies, increase overall loose coupling and cohesion thus increasing the
overall quality of design. It sometimes amazes me how using just a few simple rules can improve maintainability,
flexibility and usefulness of designs so much.
For those of you who are following my writing on SAF (or the Milestone 1 post for more details) - here is
an introductory presentation to SAF (1.2 Mb). I delivered a slightly
modified version of this 2 days ago and someone commented he would like
to see more information on strategies to deal with quality attributes.
I'd be happy to hear any other comments you may have
Last
week Beat
Schwegler and Ingo Rammer visited Israel with the Software
Factories Seminar (The link is to the same seminar presented in Finland - videos are in English)
Software
Factories is not a new idea - see for
example "Software reuse: From Library to Factory" by M. L. Griss
(published in 1993(!)) which talks about "Software Factories" and
"Domain Specific kits": components, frameworks, glue languages
etc. The current Microsoft incarnation of Software Factories takes a
similar approach focusing on Domain Specific Languages, Frameworks but also
adding important aspects like multiple viewpoints, patterns and designers. The idea is
that building on modern technologies, as
well as learning from the mistakes from sister approaches to code generation
(OMG's MDA, in case you are wondering) will enable us to build something that is
useable.
Microsoft
seems to be taking some steps in the right direction (GAT is probably the best example). Nevertheless
there is still a long way to go before we can realize the dream of
"factories" for vertical applications
This
is evident if you take a look at the
current crop of DSL examples. These are either some Horizontal languages and tools
(i.e. not aimed at a specific business domain) like UIP designer or the
GAT4WS beat talked about @ the event or even worse UML designers... ('nuff said).
The
other thing is, that developing real factories, and getting them right is a really
complicated task, which requires a lot of domain knowledge and effort. In the recent Microsoft
Architect's Journal there's an article by Jack Greenfield and
Mauro Regio regarding a software factory for the Health Level 7 ( a standard
for Health organization collaboration) - As far as I know this project has been
under way for more than a year now (first time I heard about it was Feb. 2005)
and yet the article ends with :
"Our
experience in developing a factory for HL7 collaboration ports
has
shown that we need to define better frameworks, tools, and processes
to
specify the factory schema, to manage factory configuration
in
a flexible and extensible way, and to better understand how
and
when domain-specific languages should be used."
My expericen with MDA shows similar results. Nevertheless,
they continue to say that:
"At
the same time, initial implementations of extension mechanisms like GAT and
DSL
have proven their value, filling significant gaps in software factory
infrastructure,
and pointing to future innovation in that area.
So,
maybe there's hope after all :)
I
just read an excellent post by
Gregor Hohpe talking about the motivation for Event Driven semantics for
service communication.
Gregor
gives an example of a shipping service listening on order events and address
change events to produce shipments.
It
is nice to see how architectural approaches transcends business domains so well
- The Naval C4I project Udi Dahan and myself are working on,
we basically try to take the same approach. For example: A Sensors service
publishes its status every predefined time - The sensor knows if something is
wrong with its state. A sensor, however, doesn't know if the problem is important
or not. We designed an Alerts service
that listens in on status messages, based on (changing) business rules a
certain status may trigger an alert event (which a UI can then choose to
display); a severe alert may result in an SMS alerting a technician to come and
have a look.
However,
while this approach is very good for inter-service communication -things aren't
as rosy when it comes to interacting
with UIs. The point is UIs are based on
interaction so the request-reply idiom (should actually be implemented as
request-reaction) is much more prevalent. Users really want to know their
request is being taken care of
Another
lesson we learnt is that since services make go on-line and off-line
independently of each other, it is not enough just to support event listening
for event aggregators to be up-to-date. One option is to relay on reliable
messaging to any event posted will eventually get to the listener - there are
several problems with this approach for example:
- For one, you need a reliable message
transport which might be a problem e.g. you may not be able to use
JMS/MSMQ between enterprises and/or the protocols you use don't support it
(e.g. WS-RM is not durable see here and here )
- Even if you
have reliable communication, if one
service has been offline for a long period of time (where long is defined
by the communication load) - it may be a waste of time (or plainly wrong) to process old events that are no longer
valid
Another option to handle this situation is to
supply in the contract request for current state (the current state can be
published using the same message structure used by the matching event). The
advantage here is that a server coming on-line can quickly and efficiently get
up-to-speed on the current situation.
The
event thinking is relatively on par with Take-it-or-leave-it approach for
contracts construction, but as I said in the previous post on contracts, I think it is more beneficial to know about your consumer and take their input
into account
Apropos
EDA, I also learnt today that the
Micro-Services strategy Udi and me came up with had already been "invented"
several years ago. It is called SEDA (Staged Event Driven Architecture) there's
a nice presentation explaining it here
Yoni
Commented on my SAF - Deployment post :
"Somewhere in your blog you've mentioned Martin Fowler's "Who needs an Architect" article in a positive way. However, it
seems that in contrast to Fowler's notions of an architect who's job is to
"remove architecture by finding ways to eliminate irreversibility in
software designs" - you are advocating making long-term binding decisions
in the initial stages of the development cycle"
It is very hard to argue with Martin Fowler (as Gregor
Hohpe put it in a recent post "Now
who would want to argue with Martin Fowler? His opinions are facts " ) However, I think that
devising solutions for flexibility (such as the database schema example in the
article) are also architectural decisions. What makes things even
"worse" is that these decisions are usually made at the expense of
other quality attributes (e.g. performace) or other project constrains (e.g. time
to market) - meaning that a decision on flexibility is a making tradeoff or in other words an architectural decision
"par excelance".
Thus
- architectural decisions, by definition,
are early - whether we like that
or not. Furthermore, Flexibility is an important quality attribute, one the architect brings
into the table (in the same way that other stakeholders bring their concerns -
e.g. the Project manager who want the project to end on time and on budget). It
is the architect's role to balance quality attributes, understand the tradeoffs
and make the decisions that will enable the project to achieve its goals. Many
of these decisions have to be made early so that the project can move on, some
of these decisions have to be made early to prevent ad-hoc architecture
(un-balanced, spur of the moment decisions that will be hard to change). I
believe very few decisions can be postponed without doing anything (i.e.
without making some active decision to introduce flexibility).
I've just found out (via Gianpaolo's blog) that Roger Wolter (former PM of Service Broker) started blogging . He is going to focus on Data in a Service Oriented world. I had a chance to work with Roger for a short time, which was enough to notice that if anyone knows about data, it is him. I guess there is no surprise there, considering his past at Microsoft working on :SQL Server Service Broker, SQL Express, SQL XML Datatype, SOAP
Toolkit, SQLXML and COM Plus. His first post (after the obligatory "hello world" post) is about Service Broker positioning (vs. MSMQ, Biztalk and WCF) - Subscribed
It
took awhile but I finally managed to complete describing all the activities of
SAF at least at some level of detail.
What is SAF?
There is very little guidance on how one can go
about designing/developing an architecture for a software project. The SPAMMED
architecture framework (SAF)
aims to help fill this gap.
SAF is a set of activities that an architect can follow
when she sets out to design an architecture. These activities helps the
architect to keep abreast of the project's needs and the drivers that affect
the architecture.
Overview posts:
Getting SPAMMED for architecture
SPAMMED State chart
SAF Activities
The Activities of SAF include
- Stakeholders -identify the
stakeholders - Anyone with vested interest in the project (end
users, clients, project manager, developers etc.) These are the people you
will have to explain you architecture to. These are the people that have
concerns that the architecture will have to satisfy (or most likely
balance). Thus, the fist step is to identify and rank them.
- Principles – list Principles, Goals and
Constrains. These are the properties you wish your architecture to have
(or lack) based on your previous experience. Constrains can also come from
your stakeholders (e.g. management decided that all project should be
.NET, a deadline etc.)
- Attributes – discover quality
attributes, the non-functional reqeirements, that (once prioritized) serve
as the guide for the overall goodness of the system (Performace,
Availability, scalability etc.)
- Model – model (and document if needed)
the architecture as seen from different viewpoints (list of
viewpoints is stakeholder driven). Example
for viewpoints include package diagrams, deployment diagrams, DB
Schema etc. etc.
- Map – Technology mapping, buy vs. make
decisions etc.
- Evaluate – Since architecture is the set of
decision that are hardest to change it is worthwhile to spend some time
trying to evaluate if they are indeed correct before commencing on
- Deploy – Software architectures are not a
fire and forget thing. As an architect you still have to make sure that
the guidelines set are indeed followed and even more importantly that the
architecture chosen indeed match the project’s needs and doesn’t have to
be reworked.
Feedback & Future
I
would be very interested to hear any feedback on SAF - you can send feedback to saf@rgoarchitects.com or just leave a comment
On
the next SAF posts I am going to talk about SAF alignment with development
methodologies (RUP, MSF 4, SCRUM etc.) as well as strategies for following or
acting upon the activities.
The last activity of SAF is deployment of the architecture. This step can make the difference between an ivory-tower architect and one whose designs are actually used in real software projects.
Deployment of the architecture actually means two things
- Verification and feedback loop. - making sure the architecture is actually the right one.
- Governance - making sure that the architecture is followed by the designers and developers
Verification and Feedback loop
It is common practice to think of software development as an iterative process. Knowing that Software architecture is early and that it encompass the important decision which are hard to change, it is probably wise to think of the first few iterations as architectural ones. You would try to work with the team and form the major abstractions, hopefully come up with a blueprint for processes etc. However, the fact that you've decided only the (let say) first two iterations are architectural ones, doesn't mean that new non-functional requirements won't emerge during later iterations. Since practice shows us it is (almost always) futile to try to analyze "all" the requirements before we do any design - this is almost bound to happen.
Be ready to monitor the project's progress after the "architectural iterations" to see and deal with any emergent requirements. As the stakeholders (e.g. product owner, project manager and the architect) try to prioritize tasks/requirements from iteration to iteration, hopefully most of the architectural issues would both surface and be taken care of the first few iterations (remember - architectural decisions are the ones that are hard to change…) - but unfortunately this isn't always so. I recently audited the architecture of a project (which has been running for more than a year) where they identifies the need for transactions, but following some stupid 80/20 rule decided that most of the system does not need that (deadly sin #1) and that they'd be able to easily patch it in later down the road (deadly sin #2) - so not enough just to identify a need, you also need to deal with it early. The first few iterations also serve as a feedback loop on the suitability of the architecture, sure you've tried to evaluate the architecture both in code and on paper , but once it is deployed, or used for real - that is its real test. You will probably want to allocate some time to sit with developers and designers and actually see how the architecture is used and the implications of your decisions.
Governance
It isn't enough to design a software architecture, show it off at some architecture review and maybe at product demonstrations. Designers and developers are more than likely to bend your architecture to their convenience - this can happen for a multitude of reasons - for example:
- They flipped the bozo on you - they think they know better
- They don't see the big picture and they try to optimize locally
- They don't know any better, they just do what they always did
- They believe they are following your design but they didn't really understand what you've meant (or alternatively - you didn't explain yourself very well)
- They cut corners e.g. to meet dead-lines (just to appease the project manager breathing over their shoulder)
- Etc.
When the architecture is compromised, it can definitely have severe effects on later iterations and the overall stability/usability of the system. It is important to note that it may even result in harsh implications for the organization - imagine someone circumvents some auditing mechanisms you've put in place to comply with SOX or Basel II.
What this means for the architect is that she cannot disengage from the project completely even once the architecture is stabilized (i.e. after the first few iterations). Architect involvement is needed for design reviews, maybe key code reviews , on mile stones etc.
I think it is one of the architect rolls to act as a "comptroller" and oversee that the project stays on track as far as the architecture goes. Unfortunately there aren't too many automated tools to help with that. DSLs is one direction that may help (look for example at the Guidance and Automation Toolkit here and here ). I've also recently saw an application that claims it can enforce and monitor SOA policies on e.g. on contacts (for both Java and .NET). I hope we'll see more of these tools in the future - meanwhile it is up to us, architects, to do that as part of our overall responsibility.
In
the previous post I said "don't bubble exceptions out of your service"
- Ebenezer Ikonne asks "Well I wonder
what the verbiage of the exception should be? If a null pointer occurred
in the service, what message should I return back to the consumer of the
service?"
First
off, lets consider the meaning of bubbling the exception - what would a remote
consumer, sitting on some other company's server do with a "null
pointer" exception?! - the consumer doesn't have any control on the
resources or life cycle (or anything else for that matter) of the service
it is trying to consume. Also if it depends on the internal problems of the
service it consumes it makes it (the consumer) much less autonomous.
So,
what's the other option? Well, as I mentioned in my previous post it is best if
the service can "pretend" nothing really happened e.g. log the
incoming message before doing anything and then if there's an exception respond
(if the contract requires response by a deadline) with a "got your
message, working on it, you'd get a confirmation message soon" sort of
reaction. If the exception occurs before the incoming message is saved then it
is probably needed to respond with "out of service, try again soon"
if the edge is not up then you (as a consumer) should (finally) get an
exception (the protocol failed - the message you've sent did not arrive)
By
the way a I think that a somewhat similar principle is true for bubbling
exceptions across layers in a layered architecture
Take
a look at Jeff Schneider's blog - for some nice Lego illustrations of composite
applications concepts
In one of the discussions
on the MSDN Architecture Forum someone mentioned that when a service invocation
results in an exception his company
standard is to:
Create a new SOAP error
(exception) with minimal info ("The requested service failed")
and that error includes (innerException) the original error (This way, if
someone receives my error and they are not familiar with my errors - they
will get a simple soap error. If they are familiar with my error, they
will have the ability to inspect it further)
I
think this is this is the very distinct
anti-pattern of how to do SOA i
Lets
look at few reasons why:
Log the error - that is fine and
everything - maybe it'll help during the post mortem, but the operations
people should also be notified somehow. Otherwise you have a dead service
there and no one knows
"exiting the service" - Services shouldn't fail - a failed
service can mean a lost business opportunity. When
you can't service the requests, you should at least be able to
maintain the "well know end-point" up and running and let
everyone know that the service is, well, "out of service - be back in
XYZ". The preferable solution is to still be able to queue incoming requests and handle them later (this may
not be possible if part of the policy (or SLA) for the service is to react
within few seconds, but again, in many other circumstances it is very
plausible.
SOAP Exception - why through a SOAP
exception - the protocol/communication worked fine...
"...(innerException) the original error" - do not
expose internal implementation out side of the service - only what's in
the contract - in other
words don't, just don't bubble
exceptions out of your service.
It
is very easy to make, what seems like, a small concession on the purity of your
service, but your SOA concept and loose coupling specifically can deteriorate
very rapidly on even the smallest compromises
Being an Architect is a lonely job. You get to interact with all the stakeholder one can think of, and sure, everybody has an opinion, but in the end you get to make the important decisions by yourself. Even when an organization has several architects, many times only one is assigned to a project at a given time. Well maybe it is time to move to pair architecting (pair programming redux for architects)
Over the past few months I've had the chance to work with another architect (Udi Dahan ) on an the architecture for a new product line.
This actually proved to be a very positive experience:
-
You get informed feedback for ideas
-
You get to look at a problem from more angles
-
Working together helps refine the design (instant reviews and mutual feedback)
-
You can play good cop/bad cop (or bad cop/worse cop :)) vs. The different stakeholders (PM, Devs, SMEs etc.)
-
You can divide the work to get more things done (be less of a bottleneck)
- It is also easier to work at the different levels required with less context switching ( presenting to non-technical customers, working with programmers, convincing upper management etc.)
Now, few iterations into the project, the architecture is pretty stabilized, but, we're still working together only now we mostly divide the work between us. I get to do the "fun" stuff - working with the marketing guys; working on the schedule and iterations with the project manager; etc. While Udi plays the "Architect as a coach" with the developers of the team as well as redesigning the clients (After we gave too much slack to the client team during the previous 2 iterations)
Now I wouldn't recommend bringing too many architects into the fray as this can easily degrade to a "design by committee" sort of effort but it is definitely beneficial to have more than one architect working on a project.
About a month ago Microsoft launched two Forums aimed at architects on the MSDN forums
Architecture Forums: General Forum and Architecture Forums: Modeling and Tools
Here is the introduction as posted by Simon Guest :
"Welcome to the Architecture General Forum. This forum is for discussing general issues and experiences related to architecture and designing solutions on the Microsoft platform. This forum is moderated by several members of Microsoft's Architecture Strategy Team. We welcome all questions and comments related to architecture, although we recommend that product specific questions (for example, "I can't get x to install") are directed to more appropriate forums. "
This is actually a bit of old news - last month the IASA (International Association of Software Architects) renewed web-site was launched with few interesting articles by Scott Ambler, Simon Guest and others.
You can find the site here
On the previous post on Architecture evaluation I talked about evaluating a candidate architecture in code. This post is dedicated to evaluation on paper.
I remember one system I was working on, I was keen on making the architecture asynchronous and message oriented (it was all circa 2001 by the way) However, I was new on the team and my role (as the project's architect) wasn't well defined so I had to do many compromises and get wide acceptance in order to get anything going forward. We set a team to try to come up with a suitable architecture, since each of the team members had his/her own experience we came out of these meeting with more than 20 (!) different candidate architectures (actually there were fewer architecture variations but they were multiplied by the possible technology mappings). Trying to decide which was the best option to follow we trying to conduct some sort of a QFD process where several members where in charge of the weights and the rest where in charge of evaluating and scoring the different categories (per option). Like most "design by committee" efforts this also proved a doomed from the start - and the option everybody disliked got the highest score. If you are wondering what happened - we scraped this effort and started from scratch in a more sensible way (which included a detailed prototype) - what's important for the purpose of this post is that it got me thinking that there must be a better way to do evaluate architectures. Well, a lot of research and several projects later I think that there are few techniques that give much better results.
The first methodology I stumbled upon was ATAM (short for Architecture Tradeoffs Analysis Method), developed by SEI.
ATAM is a rather lengthy and formal method to evaluate architectures it requires a lot of preparation and commitment from the different stakeholders. You can get an overview of the process from the following (~130K) ATAM presentation I prepared few years ago (While this is probably not the best presentation in terms of presenting it to a crowd (I know better now :) ) it does provide a good overview of the 9 ATAM steps).
ATAM is explained in more details in "Evaluating Software Architectures", the book also details two more evaluation methods SAAM (which I'll let you read in the book) and ARID (Active Reviews for Intermediate Designs).
ARID, like ATAM, is a scenario based technique, meaning that as part of the evaluation process you need to identify scenarios where the system's quality attributes (see Quality attributes - Introduction ) occur or manifest themselves. The main idea in ARID is that for each (prioritized) scenario the participants try to draft code that solves that scenario utilizing/following the design under test. The results of the effort are then evaluated for ease of use, correctness etc.
There's a good introductory whitepaper on ARID in SEIs web site.
Note that ARID is more suited to agile/iterative development (compared with ATAM) since (as its name implies) it doesn't require the architecture to be completed and finalized up front.
While I was working for Microsoft, I stumbled upon another evaluation method called LAAAM (which is now a part of MSF 4.0 for CMMI Improvement). LAAAM which stands for Lightweight Architecture Alternative Analysis Method is also scenario based and like ARID is more agile alternative to ATAM.
In LAAAM you create a matrix which has scenarios on one dimension and architectural approach, decision or strategy on the other dimension. Each cell is evaluated on three criteria:
- Fit - how viable is the approach to solve the scenario (including risk, alignment to the organization's standards, etc.)
- Development Cost
- Operations Cost
LAAAM was developed by Jeromy Carriere while he was working for Microsoft (he is now working for Fidelity Investments in Boston).
SAF works well with all of these techniques, as one of the basic steps is to identify the quality attributes and write down scenarios where these attributes manifest themselves in the system (see Utility Trees - Hatching quality attributes )
To sum things up -
There are several ways to evaluate software architectures on paper - ATAM, ARID, LAAAM and few others (I didn't discuss here)
Scenarios based evaluations help verify quality attributes are being taken care of by the suggested architecture
Paper based evaluations can help reduce the number of options to few (hopefully one or two) leading solutions which can then be evaluated in code (as the previous post on this subject suggested)
Today I attended a presentation on Service Orientation (SO) by Clemens Vasters. The presentation itself was very interesting, Clemens mentioned that SO is about business alignment and loose coupling (two claims I all heartedly agree with). He also made some claims about SOA which I only partially agree with (but that's not the point of this post).
Clemens, used the business practices of the pre-computer era to build the case for service orientation - utilizing several metaphors such as Department = Service and folder+form = message etc.
From these metaphors we can basically infer 2 types of service communication patterns:
- one way messages - a service consumer can ask a service to perform some action
- Request/Reply - A consumer asks for a service and waits for the results (synchronously or asynchronously)
I guess there's a general limitation to metaphors - since a metaphor is (a kind of) a model of a problem, it doesn't catch/include all the properties or possibilities of the problem. Using a metaphor, we should be careful not to neglect options that are not covered by it.
In this particular case, it is hard to see other possible ways for services to communicate. Unlike business practices of the 1920s, things can be handles in parallel, a service may not care who deals with data it publishes e.g. in a publish subscribe scenario or if there's an external (to the service) workflow that takes care of routing (read: orchestration in MS's lingo or Choreography in IBM's lingo) data to interested services. For example:
Using a publish/subscribe communication An Ordering service may publish any approved order and both the invoicing service and a logistics service can trigger a JIT inventory process to ensure that supplies will be available to fulfill the order.
By the way, I think this point (regarding metaphors) is also valid for the "system metaphor" in extreme programming (but that's a point for another post)
Reading the November issue of Crosstalk I came across an interesting article on managing Architectural dependencies using a Dependency Structure Matrix (DSM) - rather than dependency graphs induced by UML. The article is called "Dependency Models to Manage Software Architecture" by Neeraj Sangal and Frank Waldman"
The article talks about the value of using DSM technique for managing architecture evolution as requirements change (analyzing ANT as an example) - Apropos "Architecture Evaluation", I think the DSM technique can also be useful in early stages of the architectural design as a way to infer dependencies and to help promote loose coupling
In the introduction to architecture evaluation I said there are two approaches to evaluating a software architecture. This post talks about the first approach - evaluating an architecture in code.
POCs
The first evaluation-by-code tool is the Proof of Concept (POC for short). Building a POC is about building a minimal amount of code implementing a focused area of the architecture or the architecture's technology mapping. The aim of the POC is to help weight alternatives (when you are contemplating which way to go), lower technical risks or lower stakeholders' anxiety over an architectural choice.
POCs map quite well into XPs spikes
Lets look at a few POC examples (examples from my past projects)
Example 1: Validate the feasibility of an architectural direction
On one project we inherited this ugly application incorporating its own proprietary cgi web-server, the architectural decision was to keep this as a black-box and develop the project using a better more scalable architecture (though we still needed to utilize functionality form the C++ server now and then). The challenge to make this happen was to be able to maintain and pass the session from the rest of the application (JSP, Servlets and J2EE) onto the C++. A (successful) POC that tackled this issue allowed us to advance in the chosen architectural direction, reducing the risk significantly.
Example 2: Validate a technology mapping
On another project I worked for (when I was with Microsoft) we analyzed the project quality attributes and found that there's a need for near-fault tolerance (fail-over in 5 seconds or less). The architectural solution that we decided on was to use an active server and a semi-active one (on-line, ready to take over server that constantly applies state from the active server)* - for technology mapping we considered several options (e.g. fault-tolerant hardware ). One of the option considered was using SQL Server 2005 Database mirroring to keep the two servers synchronized (DB mirroring gives you a failover of the DB in about 5 seconds or less). In order to verify this direction I set up a small Proof of concept to verify that this direction is viable. I was told that after I left Microsoft, further investigation of the issues found led to Microsoft's decision to postponed Mirroring for now.
Example 3: comparing alternatives
We wanted to compare MSMQ vs. using an existing distributed object middleware both in terms of performance and usability (is it developer-friendly). We crafted 2 POCs one for each technology which enabled us to compare the two approaches head-to-head.
POCs help evaluate alternatives and lower risk in specific areas of the architecture (and design for that matter). However, POCs will not give you a feel on how the overall architecture will play together - enter prototypes
Prototypes
A prototype is basically a working simplified model of the system. There are many characteristics to distinguish between different types of prototypes (hi-fidelity/low-fidelity, global/local etc.) - let focus on two:
- Horizontal prototype - which models wide aspects of a single layer, i.e. many features with little details. The most common example for Horizontal prototype is a user interface prototype which is used to test the overall interaction with the system.
- Vertical prototype -Implementing some sub-system or a limited set of features across all layers /modules.
The Vertical prototype is useful way evaluating, getting a feel and understanding of how the different components, that makes the architecture, work in unison without getting bogged down in all the fine details of the system's functional requirements.
Example: Using a prototype to evaluate an architecture alternative.
We were getting ready to embark on a rather large project (we did the prototype around the release of .Net 1.0 and the project is still going on…). We wanted to understand the capabilities and limitations of .NET. We chose a limited aspect of the system (which we considered the most risky), chose some of the designated team-leaders and took an architect from Microsoft Consulting Services to help us build the "by the book" architecture.
We did a very extensive prototype, total effort of 3-4 man-years including all the preliminary work and the post-mortem analysis. We gained a lot of insights on what .NET can and cannot give us out of the box, we understood the limitations of the components we integrated (e.g. ESRI's limitations in displaying near-realtime moving objects) and we also used the preliminary prototype (which was a performance hog) as a platform for running POCs for other architectural and technological directions. Additionally, once we solved the performance problems, we also used it as a demo for the client.
By the way, this experience also had some additional positive residual effects like, getting the team leaders up-to-speed on the (then) new technology, jelling the core team etc.
Taking all the information gathered during the prototype we were able to design a better, more robust architecture for the project itself (which the architect, that came after I left the project, managed to mangle - However that's another story altogether :) )
I've found that in most cases exploratory prototypes, or "Throwaway prototypes" are more useful as they really let you get to the crux of the matter quickly, i.e. getting all the components connected the way the architecture dictates to test their interactions and usage. Again, the idea here is to focus on evaluating the architecture, not on the implementation details of the overall solution. Nevertheless, once the architecture is more mature you may choose one of the prototypes and evolve it into the actual system (sort of turning it into an architectural skeleton).
Architectural Skeletons
Once you've decided on a candidate architecture (i.e. the architecture you want to use for the project) your first iteration or two (This might not be the first iteration as you may already done a couple or so prototype iterations) should be focused on creating the architecture skeleton.
Architecture skeleton is about implementing the minimal set (bare bones, so to speak) of the project's functionality that is needed to connect all the pieces in a meaningful, integrated way (for example it can includes an implementation of a single thread in a use case or a important story). It is somewhat similar to a prototype, with 2 differences :
- It has to implement real functionality of the system (though the functionality is, usually, very thin)
- You don't throw it away (hopefully anyway)
Most current methodologies (RUP, MSF for CMMI Improvement, XP etc.) support the notion of architectural skeleton (though not using this name). In RUP, for example, you would have the architectural skeleton up and running at the end of the elaboration phase - a running architecture which you can expand and functionality to in the construction phase.
It is important to implement a skeleton (vs. starting to implement the different components and try to integrate them later) as it gives you a relatively early opportunity to actually test if your architecture holds, and it is much better to find errors, especially architectural ones, as early as possible.
Summary
I demonstrated 3 "tools" to enable evaluation of architectural decisions in general and the overall architecture in particular:
- POC - focused on a specific area
- Prototype - overall architecture with "simulated" behavior
- Skeleton - "barely running" implementation of the chosen architecture.
The problem with these approaches, especially prototype and skeletons is that they require a relatively long time as well as resources to implement. We need some additional tools in our evaluation toolset to allow us to focus on architecture alternatives that are most likely to match our needs.
I think that there are such tools, and on the next post on architecture evaluation I will try give my view on what they are and how to use them.
* other options are active-active and active-passive (e.g. Windows clustering)
Ok, so you've designed this grand glimmering SOA, and you have a few dozens services each assigned to a development team.
As the iterations spans the contracts have to be amended (be that WSDLs, plain old messages, object APIs or whatever) - do not leave the task of changing, aging or maturing to your developers, team leads or anyone else - I strongly believe that the architect, having defined the boundaries between the major components (services in this case) is the one also responsible for the space between the services. Anything that travels this space is in the realm of the architect*
In my opinion, failure to be in charge of the contracts can result in major breakage of the architecture (chatty interfaces, security problems, services dependency to name a few of the possible and likely problems)
I've been a little busy in the last couple of weeks and unfortunately, didn't have time to blog - to get my self started again - here is a short introductory post on the subject of Architecture evaluation:
I said in "what's Software architecture" - architecture is both an early artifact and it also represents the significant decisions about the system - or to sum it up "Architecture is the decisions that you wish you could get right early in a project.” (Ralph Johnston*). That is exactly why I made evaluation one of the key steps in SAF. We want to raise the level of certainty that the direction(s) we are taking are indeed the right ones and we will not hit a wall later on. Especially considering most projects these days are iterative, which makes this even more challenging.
But, how do you evaluate an architecture? (on a give set of quality attributes) How can you tell if utilizing, say, SOA, will yield better results than distributed objects ? Is using fault-tolerant hardware better than using a database (on performance vs. robustness trade-off)? and so on and so forth. Even if we say that flexibility and the ability to cope and embrace change is the most important trait (read quality attribute)for our architecture we still need a way to evaluate which approach will give us the most flexibility
The way I see it there are basically two approaches for evaluating software architecture
- In code - as in proof of concepts, architectural prototypes and architectural skeletons
- On paper/discussion based -so called "formal" methods: ATAM, ARID, LAAAM to name a few
I will try to use the next few posts on SAF to elaborate on these issues - where, the next post on SAF - Evaluation will explain and demonstrate the "in code" methods. The post following that, will talk about the paper based methods, and the third post on the subject, will try to contrast the two approaches and talk about their respective pros and cons
*As quoted by Martin fowler in "Who needs an Architect" - which, by the way, provides for a very interesting reading on architecture and the architect's role.
The next step in SAF after Modeling is technology Mapping. While mapping is not a part of the architecture per se, it is, in my opinion, an important and sometime crucial step.
Before I rumble on explaining why I think this is an important step, let me try to define what exactly do I mean by "technology mapping"
Architecture in essence is technology neutral - it describes the major components (read objects/services/components etc.) and their interactions - but it doesn't specify what technologies and what technologies, products or existing assets will be used to implement it - that's where Technology Mapping steps in.
For example, in a previous post (Architectural Modeling - a First Step ) I presented the following block diagram as a possible view for an architecture (Layer diagram):
One possible technology mapping for this (view of an) architecture is depicted in the following diagram:
Before I continue any further, I should probably say that I think that technology mapping is basically a design activity and not an architectural one. But wait, if that is true, why am I mentioning this as a step in a framework for building architectures (SAF ..) ?
Well, besides the obvious answer, that it helped me create a nice acronym for the process - being the second M in SAPMMED, Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible.
For example, in one of the projects I am currently working on we made the decision to create an SOA (surprise, surprise*). We decided that the various services shall communicate using a service-bus and our intention was to map that to a messaging product (such as Microsoft's MSMQ or Tibco's Rendezvous ). As it happens, the powers that be (i.e. upper management) decided that there is no room for investing in a new inter-process communication solution and that the project will be forced to reuse the existing solution. The existing solution, in this case, is a proprietary distributed objects solution developed in house. While our intention was to promote message-oriented development and contracts. The existing middleware, being a distributed object framework induce chatty RPC-like contracts (you can read more on RPC vs. message-orientation in "Mort gets the message" and few of the other posts on John Cavnar-Johnson's blog ). The implications of this technology decision is that the architecture team has to closely pay attention** (on daily level for the initial iterations) to what the designers and developers do so as to ensure that the architectural decisions are kept.
On another project I worked for in the past we designed the solution to use an application server (multi-tiered hardware architecture) however the solution also had to incorporate ESRI's ArcGIS which (at the time) only worked in a "two-tier" client/server manner with its underlying data. In order to support the implementation of an application server (which we felt was really needed) we had to (decide and) develop an independent "entity layer" on top of ArcGIS. Failure to notice that the technology mapping implications would have resulted in the architecture not being implemented (and serious scalability issues for that particular project).
The second reason for including Technology mapping in the architectural process is to help promote reuse in projects by making this as an activity at the architectural level (i.e. both early in the process, and making the decision to reuse a component a global decision governing the solution).
Making reuse a first-class citizen in the architectural effort can greatly help promoting product-line approach (examples for additional factors that can help promote product-lines include domain-driven design or concepts like software factories)
To recap:
-
Technology mapping is about deciding which products, technologies and existing assets are going to be utilized for implementing the architecture.
-
Technology mapping can have a significant impact on the ability to adhere to the architecture to the point where under certain mapping constraints you may need to reevaluate the architecture (i.e. is it worth-while to go "against the stream" and implement the architecture using a technology/components whose architecture.
-
Deciding which existing assets can be reused at the architectural phase helps to create systematic reuse (vs. opportunistic reuse) which improves time-to-market and solution quality (assuming you carefully choose assets for reuse )
I recommend either making technology mapping a required step in the architectural process or alternatively revisiting the architecture once this step occur as part of the design.
*when you get over the incredible amount of hype around it (I believe) SOA does have several distinct business and technological advantages - I'll try to elaborate more on this in one of the next posts
**I'll talk more on what happens once the architecture is "released" for public consumption when I'll coved D- Deployment phase of SAF
I claimed in the past (on my "what's software architecture" post) that Architecture is a type of design - if that is true, an interesting question is do we also have architectural patterns ?
I think the answer is yes - there are architectural patterns they are also called architectural styles - I actually like this term better as it is helps differentiate them from design patterns; for example I agree with Harry Pierson's observation that many of the patterns in Martin Fowler's PoEAA are indeed technical patterns at an engineering or design level and not architectural patterns.
The difference between architecture styles and design patterns is similar to the distinction between architecture and design - architectural styles effect the solution globally or at least it affects major parts of the solution and not solve local issues. Although it is interesting to note that some architecture styles have been well known well before the notion of design patterns for software was introduced.
SEI's architecture glossary defines Architecture Style as
"A specialization of element and relation types, together with a set of constraints on how they can be used."
That's a good start but it might be a little hard to understand we can basically say that an architectural style defines a family of solutions in terms of a pattern of structural organization using a vocabulary of components and connector types (plus constraints on their use). It is also worth mentioning that different styles can be combined to create compound or derived styles.
I'll try to illustrate what a style is using an example:
One of the basic architecture styles is "Layered architecture":
The layered style is composed of layers (the components) which provides facilities and has a specific roles. The layers have communication paths / dependencies (the connectors).
In a layered style a layer has some limitations on how it can communicate with other layers (the constraints). Typically a layered is allowed to call only the layer below it and be called only by the layer above it (but there are variants e.g. a layer can call to any layer below it; vertical layers that can call multiple layers; etc. - as long as the layers communication paths are limited by some rules)
You can see the application of layered style all over the place for example: logical software layers (e.g. presentation component, UI Controllers, business processes, business components, data entities, data access layer) , SOA layers (fundamental , Intermediate, Process), physical tiers (e.g. database server, application server, web server and clients) etc.
There are many other architecture styles including for example pipe and filter, push based, peer-to-peer, blackboard, MVC, PAC, or the more recent Adaptive Object Models, REST and SOA (some will probably disagree that SOA is a style - but I'll try to explain why I think it is on another opportunity). I'll try to talk a little about some of them in the next posts.
Ok, so we've identifies stakeholders, set principles and guidelines, found out what are our architectural requirements and we want to start modeling already - especially considering architectural modeling is great fun (for techno-geeks like myself anyway). However, before we just start to create an endless flurry of blocks, boxes, arrows and whatnot, it is probably worthwhile to take a few minutes to think which Models we want to create and what views do we want to use to communicate them otherwise we may end up doing a very good job thinking about and describing something that doesn't interest anyone and gives no real value.
IEEE 1471 defines (an architectural) view as a representation of a whole system from the perspective of a set of concerns (where concerns are the key interests of the different stakeholders)
A view is comprised of parts of one or more models to demonstrate the how the concerns are covered
A view is (sort of) an instantiation of the pattern defined in a viewpoint (see more below)
So for an example if our concerns are concurrency and timing issues looking from this viewpoint can be looking at the threads and process model of the solution and we can use views like process diagrams and timing diagrams to visualize it.
One thing we can do (over time) is to create a viewpoint repository and then when faced with a set of concerns we can easily choose which views to create.
Rich Hilliard suggests that a viewpoint would hold the following information:
- Viewpoint name
- The stakeholders addressed by the viewpoint
- The stakeholder concerns to be addressed by the viewpoint
- The viewpoint language, modeling techniques, or analytical methods used
- The source, if any, of the viewpoint (e.g., author, literature citation)
A viewpoint may also include:
- Any consistency or completeness checks associated with the underlying method to be applied to models within the view
- Any evaluation or analysis techniques to be applied to models within the view
- Any heuristics, patterns, or other guidelines which aid in the synthesis of an associated view or its models
If you are just starting out you can turn to various frameworks to see which views they have and use that as the basis of your repository for example:
RUP suggest a 4+1 set of viewpoints:
Microsoft is currently moving from the MSF 3.0 model (4 spheres - contextual, conceptual, logical, physical crossed with 4 "views" -business view, applications view information view and technology view totaling in 16 viewpoints) to the following set of viewpoints:
Other examples are DODAF (3 main viewpoints with 26 sub-viewpoints), RM-ODP (5 viewpoints) and Zachman Framework (with 36 viewpoints)
Lastly the software architecture document template I've posted also contains a list of possible views
What about a minimal set of viewpoints? Well, I don't know about a minimal set - there are however few viewpoints that usually get used:
Some sort layer view (usually block diagram)
A logical view (main classes/packages)
A deployment diagram (tiers, zones etc.)
A view to show concurrency and timing issues.
On SOAs there's also a service view (services, policies)
To wrap-up -
-
Choosing which views to create (viewpoints) is an important step before embarking on modeling.
-
Viewpoints are chosen according to the stakeholder's concerns (to help make sure the concerns are addressed and to communicate the architecture better).
-
Growing a viewpoint repository is a way to reuse your knowledge of concerns to views mapping.
Harry Pierson writes in his blog a very interesting piece investigating the tenets of good models.
While I tend to agree that there is a place for precise formal models that can be transformed easily to lower levels.
I would also like to argue that
- I think that imprecise models are also very useful, since at different points in time during the development you cannot fully specify all the finest details (even if for the "current" level of abstraction) esp. since most projects these days are iterative.
- Which brings me to the next point - for imprecise models - I don't necessarily think that there's a need to keep (all of) them updated during the development life-cycle. The high-level designs can be replaces by detailed designs and they in turn can be replaces with the code itself - good code explains itself beautifully :) .
- You should carefully weight the ROI for creating such a precise model. For example I happened to work on a large (hundreds of man-year) project where the initial thought was to use a tool (Vitech's Core) to for requirements analysis. The benefit was that (if done right) the model created can be "run" using their built-in simulator. After spending more than half a year (of a rather large team) we finally decided to drop this precise model for a much less precise model of use-cases which allow for a varying level of abstraction. It should be noted though, that (cross-subsystems) use cases are later refined into a DSL which is actively used for generating cross-subsystem interfaces and simulate missing system during integrations.
- Another point from the former example is on the timing of requiring the precision. Modeling tools should allow several levels of precision, since in earlier stages you (usually) cannot determine all the bits and bytes that will allow for a "deterministic transformation"
Just my 2 cents
I've added a sample skeletal template for a Software Architecture Document on the SAF page.
Few notes on the template:
- It is based on formality level required for large safety critical projects - most projects do not need this level of formality and can (and should) do with fewer views
- When there isn't a specific requirement by the customer to create an "official" architecture document - you probably want to have it just barely good enough
- The view list in this template is not an exhaustive list - these just the views I've used most often. (more on views on the next post :) )
Any feedback (either as a comment or to my email) is more than welcome
Michael Platt talks about the basic questions related to architects (what is architecture?; why are architects needed?; what are architects?; what skills do architects have?; and what are the types of architect). I agree with some of the things he says - especially the discussion on the architect skills.
I think Michael's definition of architecture is too simplistic (see more below) and I don't agree with his classification of architects
(here is what I think on architecture types). For example he bundles solution architects under technical architects I believe solution architect also have a lot to do with the problem domain and not just the technical or technological sides of the solution.
He also defines a "product architect" :
"Product level architects have an in depth understanding of the use of a specific product in a technical architecture domain such as Lotus notes in the messaging domain. Typical product architects are Exchange, SQL Server (normally as a DBA), Windows, and Networking etc"
I don't think that people under these roles are actually architects - they are definitely specialist and they may very well be experts but the breadth of their designs is very local in the scope of a complete solution and their skills will never be used on their own - for example even if you do a data warehouse project designing the database is (a very important) part of the project but there's still a lot to do with getting the data into there and deciding what data you want to store.
Architecture talks about things in the global (in the context of a project/solution/product line/enterprise) and design deals with local issues (how to model the UI, optimize the DB, set up Lotus Notes etc.)
What do you think?
The next step in SAF (after "quality Attributes") is Modeling. Webster's dictionary defines "Model" as (among other things) : "3 : structural design; 7 : Archetype ; 10b : a type or design of product (as a car);11 : a description or analogy used to help visualize something (as an atom) that cannot be directly observed" - as I mentioned in my what's software architecture post there's no single structure that IS the architecture - this means that we'll have to look at the architecture from different angles (viewpoints) - for example a block diagram such as the diagram below (accompanied with some documentation that explains it) may tell us something about the layers that will be used to solve a problem it tells us nothing about the business that we are trying to solve.
We need to augment with more views on the system (such as the next diagram) so that we can better visualize and convey the architecture of the system.
IEEE 1471-2000 "Recommended Practice for Architectural Description of Software Intensive Systems" defines the relationship between the Stakeholders, their concerns, viewpoints, views and models:
(IEEE 1471 model adapted from a presentation by Rich Hilliard )
By the way, the fact that we need to look at the architecture from different viewpoints doesn't necessarily mean that the documentation isn't just POW("plain old whiteboard") - The formality of the documentation is driven by the project style (agile/formal) and other stakeholder constraints (company standards, customer requirements etc.)
Since models participate in views (which in turn conforms to viewpoints - which address the stakeholder's need/interested in) I consider choosing the views a prerequisite for modeling. Thus, on the next post on modeling I am going to talk a little about choosing views (suggested minimal set, viewpoint library etc.). Once I'll get that off the table I'll try to talk a little on architectural styles and patterns and follow that with some strategies for dealing with quality attributes before moving to the next SAF phase (Mapping).
Rich Turner writes about architecture & change. I agree with a lot of what he says.
I often find myself explaining to stakeholder (PMs, developers, etc.) that the only constant thing in the project is that it is that it is changing :)
One project I am currently working on, exhibits this inherent change - For one the customers don't really know yet what - they want (think) that they would be able to use the exact same solution for a several very different configurations (ranging from standalone computer to a server farm with dozens of clients) etc.
What do we, as architects, can do to handle these situations? Well, what I usually do is add a lot of modifiability related quality attributes (see utility trees) these can include requirements for interoperability, adaptability, changeability etc. etc. (An example scenario may be: provided with a verified algorithm, replacing an existing navigation algorithm shall take less than 3 man-weeks).
While I will (probably) talk a lot more on strategies for handling quality attributes in posts regarding modeling (as part of my SAF posts), SOA (sans hype…) is a good strategy to cope with flexibility (i.e. changing requirements). The explicit boundaries and contract first approach help localize changes also the resulted loose coupling help to replace, add and remove services easier, Lastly the basing communication on policy helps postponing issues that has to do with the network (QOS, security etc.)
A completely different angle on the issue of changes - is that sometimes it can be problematic to allow it, even at the expense of missing some of the customer's changing requirements. An example (maybe the only one) is for critical systems, where a defect can result in loss of lives (two polar examples are medical systems on the one hand and weapon systems on the other). In order to be able to ensure software safety (Identifying hazards, proper fault tree analysis etc.) there's a need to affix the requirements for longer periods compared with other types of projects.
In previous posts (here and here) it seems I downplayed the importance of functional requirements (vs. quality attributes) on the architecture. Nevertheless the functional requirements do have few important roles in shaping/looking at the architecture. One aspect of the functional requirements role was demonstrated in the scenarios that describe the instantiation of quality attributes within the system. Lets look at a couple of other aspects.
As I mentioned in a previous post no single structure is the architecture - this means that in order to present an architecture it is needed to describe it from several different angles or viewpoints (I'll spend some time taking about these in the next few posts on SAF). One of these has to do with the domain architecture for the solution. This can include identifying business areas trying to identify services (in an SOA), identifying major components for a component based architecture, or even identifying major use cases in a "use case view" (as part of a Unified Process 4+1 approach )
Listening to "Almost Cut My Hair" by Crosby, Stills & Nash, in the background kinds of brings me to the 3rd area where I see functional requirements meets architectural decisions. When there are extreme/hard/important requirements there are many times where you have to decide whether to cut your hair and bend the entire design to answer that requirement (i.e. make that a global decision - hence an architectural one) or satisfy it by a specialized local solution (i.e. Postpone it to the design phase). For example, I once worked on a system where we identified one area of the solution that needs a (relatively) high update rate (low latency, high throughput for updates coming from an external system, processing it within the system and all the way to the user's workstation window). While this was both predicted to be frequently used and an essential requirement for the success of the system the majority of the system's functionality did not have these characteristics. The decision I've made was to treat it as a local issue (to be given a specific solution @ design time). (Unfortunately ?) I moved to another company before the project ended, and the architect the followed me took the decision the opposite decision (i.e. to make the solution for that problem an architectural solution for all the system's "entities") - which resulted in making all the interactions within the system (even the simplest ones) asynchronous. I recently had a chance to see the effects of this decision on the system's schedule, robustness and complexity, well let me just say that the lesson here is that while cutting your hair (making an architectural decision) is not an irreversible decision, you cannot just undo it instantly and it can take quite a long time (=money) to correct things.
To sum things up
-
Functional requirements manifest themselves as part of the utility tree (as the scenarios),
-
It can also be important to view the architecture from the functional perspective
-
Significant/important functional requirement should be weighted for their influence on the system's architecture (should they get a local treatment (as part of the design) or affect the system globally)
Grady Booch (one of the "three amigos" - fathers of UML) added a gellery of Architecture views from the literature (jpg's) to his Architecture Handbook site
Note: registration needed
In the previous post about SAF I introduced the concept of quality attributes. I wrote that using a "utility tree" approach is a very good way to identify, document and prioritize quality attributes. The purpose of this post is to expand on this issue
As I mentioned before, MSF 4 for CMMI improvement make use of LAAAM (developed by Microsoft's Jeromy Carriere )
) for assessing the architecture (it is used there for assessing the architecture, which is also a good place to use it - but I'll talk about that when I get to E(valuation) of SAF.). LAAAM also builds on a "utility tree, below are the sub-activities mentioned in the MSF beta bits:
- Examine quality of service requirements and product requirements to determine the key drivers of quality and function in the application.
- Construct a utility tree that represents the overall quality of the application. The root node in the tree is labeled Utility.
- Subsequent nodes are typically labeled in standard quality terms such as modifiability, availability, security. The tree should represent the hierarchical nature of the qualities and provide a basis for prioritization.
- Each level in the tree is further refinement of the qualities. Ultimately the leaves of the tree become scenarios.
- For each leaf in the utility tree, write a scenario. The scenario is in the form of context, stimulus, and response. For example, "Under normal operation, perform a database transaction in fewer than 100 milliseconds."
- Open the assessment matrix template. Enter each scenario as a row in the assessment matrix.
ATAM (by SEI) - (another architecture evaluation methodology) talks about a similar process with the addition of prioritization:
- Select the general, important quality attributes to be the high-level node
- E.g. performance, modifiability, security and availability.
- Refine them to more specific categories
- All leaves of the utility tree are “scenarios”.
- Prioritize scenarios
- Present the quality attribute goals in detail
This post is going to cover writing the scenarios, their prioritization and what's missing from both these methods (since they are evaluation methods) - ways to help us identify which quality attributes to use in the first place.
First, before we delve too much into details, here is an example for what the end result might look like (taken from http://www.akqit.ch/w3/pdf/bosch_atam.pdf - I am trying to see what I can publicize from project's I've been involved with - but I guess this will have to be later, i.e. in a separate post)
It is hard to explain exactly how you would go about eliciting the quality attributes and their refinements (I think that the best way to do that would be through a workshop - but it's hard to do that over a blog :) - it does, however, include the same techniques you would use to elevate any other requirement -either by building on your past experience from similar systems but mostly by working closely with your stakeholders:
- Interviews - meeting with individuals stakeholders to discuss their view of the system
- Brainstorming - meetings with multiple stakeholders trying to come with attributes and scenarios
- Reading written requirements (if available) - e.g. RFPs, use cases , project risks document etc.
To help with the elicitation, I'll try to give you some list for the first two levels (Attributes and refinements) that can serve as a repository or checklist when you are working with the stakeholders.
I already provided a relatively long list of quality attributes to draw from to create level 1 of the tree (though the list is not an exhaustive one) in the previous post .
For the next level 2 of the tree (refinement) consider the following lists for the common quality attributes (most from A Method?< Analysis Tradeoff Architecture the to Scenarios General of Applicability)
- Performance
- latency
- deadline
- throughput
- jitter
- miss rate
- data loss
- Availability -
- time period when the system must be available
- availability time
- time period in which the system can be in degraded mode
- repair time
- boot time
- Modifiability / Replacability / Adaptability /Interoperability
- difficulty in terms of time
- cost/effort in terms of number of components affected
- effort
- money
- Efficiency
- Resource X (CPU/Memory/…) usage on average per unit of time
- Max usage of Resource
- Availability of resource over time
- Usability / Learnability / Understandability / Operability
- task time
- number of errors
- number of problems solved
- user satisfaction
- gain of user knowledge
- ratio of successful support requests to total requests
- amount of time/data lost
The scenarios are the most important part of the utility tree, the main reason is that the scenarios help us understand the quality attributes needed, and more importantly, by tying the attributes to real instances in the system the scenarios help make these goals both concrete and measurable.
A couple of things that are important to note about scenarios
- First and foremost - Scenarios should be as specific as possible.
- Scenarios should cover a range of
- Anticipated uses of the system (“use case” scenarios) - what happens under normal use
- Anticipated changes to (growth scenarios) - where you expect the system to go and develop
- Unanticipated stresses to the system ("Soap opera scenarios" or exploratory scenarios , pushing the envelop etc.)
Scenarios are basically statements that have a context a stimulus and a response and describe a situation in the systems where the quality attribute manifests itself.
Context - under what circumstances
Stimulus - trigger in Use case lingo
Response - what the system does.
let's look at few examples to try to clarify this:
- Under normal operation, perform a database transaction in under 100 milliseconds (Use case)
- Remote user requests a database report via the Web during peak period and receives it within 5 seconds (Use case).
- Add a new data server to reduce latency in scenario 1 to 2.5 seconds within 1 person-week. (Growth)
- An intrusion is detected, and the system cannot lock the doors. The system activates the electromagnetic fence so that the intruder cannot escape (Use Case)
- For a new release, integrate a new component implementation in three weeks. (Growth)
- Half of the servers go down during normal operation without affecting overall system availability (Soap opera)
- Under normal operations, queuing orders to a site which is down, system suspends within 10 minutes of first failed request and all resources are available while requests are suspended. Distribution to others is not impacted. (Use case)
- By adding hardware alone, increase the number of orders processed hourly by a factor of ten while keeping the worst-case response time below 2 seconds (Soap opera)
If we take one of these (e.g. "An intrusion is detected, and the system cannot lock the doors. The system activates the electromagnetic fence so that the intruder cannot escape ")
The stimulus - An intrusion is detected
Context - the system cannot lock the doors.
Response - the system activates…
Or another one (Half of the servers go down during normal operation without affecting overall system availability)
Stimulus - Half the servers go down
Context during normal operation
Response - without affecting overall ...
The last step is prioritizing the scenarios, it is common to use 2 criteria (though you can use more)
- Importance to system success
- Risk/difficulty in achieving
The interesting scenarios (where you would focus) are the ones with high priority (H,H);(H,M) and (M,H) - these will be used as input for the modeling step of SAF
I'll try to provide samples based on my experience in one of the future posts.
I just found this on the new CodeGallery on GotDotNet (via Brad Wilson's blog). The paper provides for an interesting reading and discusses some of the issues I was going to cover regarding Architecture Evaluation (I will probably blog about them anyway, but that's just because I like to write :) ).
Also note that the document is labeled "Microsoft Confidential." on the first page - I am guessing that the document status changed and they forgot to remove this notice - but it might also mean the document will not be there for long...
Harry Pierson talks about architecture and software architects. He quotes and adopts Alan Cooper 's* view of an architect : "...The architect is responsible for determining who the user is, what he or she is trying to accomplish, and what behavior the software must exhibit to satisfy these human goals..."
I agree that customer focus is a very important aspect of the architect's work - To quote from WWISA's definition of the architect's role:
"Client advocacy is the cornerstone of the architect’s role ... An architect ceases to be an advocate if tethered to a prescribed set of technologies, tools, or methodologies, only narrowing the solutions available to the client…". Marc Sewell in "The Software Architect's Profession: An Introduction" expands this view comparing software architects to construction architects, i.e. the architect role is to represent the client vs. the construction (development) organization.
Nevertheless - I believe that the only way for the architect to accomplish these feats is to do design - no - not "low-level" design of local issues, but yes the design of the overall system, be that partition into business-aligned services advocated by SOA or identifying strategies needed to cope with fault-tolerance (an availability issue).
Yes "Coming up with the lists of functional requirements and non-functional constraints is the architecture problem" (as Harry wrote in a previous post) is an important part of the architect's job - but it is far from being the only part, this is just the preface to the actual work of laying out a solution that can support these requirements and especially the quality attributes (non-functional reqs.). (To use Marc Sewel's analogy) Just as building architects design blueprints for buildings, Naval architects design blueprints for ship building - the software architect has to draw the blueprints which the development teams will use.
Lastly, Jack Greenfield et. al. pointed out in "software factories" the model of one level of abstraction are the specifications for the next level of abstraction thus the requirements are the specification of what the system does (without specifying what the solution is) and architecture is the specification of what the solution is (without specifying how it should be implemented)
It is also important to note that the customer is not the only stakeholder whose concerns has to be considered (and balanced) by the architect (see a previous post I made on stakeholders) - This aspect is even more intensified in "real-life" since the architect is more often than not a part (or hired by) the developing organization and not the client (i.e. you don't usually see a client that hires an architect to represent it vs. the development contractor).
Having said that, I would also like to comment on the quote "architecture is design but not all design is architecture" - what I meant to imply by that is not that architecture is some sort of "good design", but rather, it means that architecture is a certain level of design that takes into account global decisions which affects the whole system (and identifies key local decisions) and that there are other levels of design which are not architecture (what Harry calls engineering)
* Alan Cooper is considered as the father of VB. He is also the author of very enlightening books on interaction design: "About Face 2.0" and "The inmates are running the asylum"
If you are following this or Udi Dahan's blog, you've probably already read that we are currently working on a project together. The project we're working on has a lot (and I mean a lot) of flexibility requirements (adaptability, replacability, interoperability etc.) which pretty much pushed us to work service-orientations.
Naturally (?), we invested a lot in partitioning our services around logical boundaries, which will allow autonomy and loose coupling. we also tried to make sure this partitioning is aligned with the business goals of both the current project and the future road-map (the same architecture should grow to serve a product line).
The more uncommon (and probably more interesting) design decision we made is regarding the internal structure of the services. We have an additional level of services (i.e. Micro services) that partition the service internally. The big difference however is that the alignment of the services boundaries is technical/design driven and not business driven. For example we can have a micro service to deal with persistence and another to do number crunching (for important algorithms) etc.
What are the benefits of this architectural decision?
- Unified semantics both inside and outside the service
- Unified solutions for availability
- Fine grained control on scale-out strategies and bottleneck handling (scale-out in the service level or the micro-service level is relatively easy)
- Built in solution for increasing concurrency (remember, partitioning along technical decisions)
- Contract based negotiations allow for increasing flexibility in identified areas (e.g. set important algorithms as a micro service and it will be easier to upgrade them later to a more advanced version)
The main cons are additional complexity and increased latency inside the service. also there's the risk of partitioning the micro services to too fine-grained services and increase the latency further (watch out for chatty contracts)
Other things you to pay attention to - You probably don't want to have fine-grained micro services (just as much as you wouldn't want them in the services level). Also, it is probably not a very good idea to have too many levels of micro services (it seems that a single level below the service is just right , but I don't have enough data to validate that yet).
By the way, it was Udi that dubbed the term "Micro Service" - so if you decide to use it, sent the royalties to him :)
Udi Dahan and me are currently architecting an interesting little-big project (little as it has a very limited scope for a first deliverable on a very tight schedule and big as the stakeholders require the architecture being drafted now to fit/scale for some very aspiring plans for the future - past this deliverable). It is very interesting working together seeing how different people have different styles, approaches and even directions looking at the same set of requirements and problems (and that's when we basically do have a common approach).
Yesterday Udi posted on one such difference - in the area of documenting our architectural designs. Udi thinks it is more simple and understandable to use sequence diagrams while I opted for using UML 2.0 communication diagram
The main reason for me to choose this diagram )over sequences) was to show off my UML skills (?!) - well, not really - rather the reason was that we were trying to describe to overall context of the UI. Using a Communications diagram helps alleviate the responsibilities of the classes in the (UI) context by showing both the relations of the main classes (actually class instances) and the main flows of messages between them. Using a single diagram allows for understanding (once you get yourself familiar with the syntax) how all the object collaborate to achieve the mutual goal of making the UI tick (I guess a better name for this diagram is collaboration diagram -but that means something else in UML 1.x).
Also as Scott Ambler mentions in The Object Primer 3rd Edition: Agile Model Driven Development with UML 2 collaboration diagrams are useful when use cases aren't the primary requirements artifact (which is the case in the particular project we're working on).
As for "visual shock" - well I think the class diagram is less intuitive and straightforward compared with the communication diagram. what exactly is the difference between aggregation and composition and how is that different from association. Also when would you use association over dependency etc. Class diagrams seems to be "simple" because we are used them. Communication diagrams
Lastly talking about the "white-board" approach - compared with sequence diagrams, using communication diagrams it is much simpler to rearrange message and/or to insert a new messages in the middle of a sequence (just draw a new message and renumber the other messages)
I read today a post by David Ing called "An Overly Long Guide to Being a Software Architect" . David talks about different aspects of a software architect - among those things he mention two important soft skills for architects namely Organizational Politics and communications. Two additional soft skills (or competencies) that an architect needs are strategic thinking and Leadership (There may be some others but I think these 4 are the main ones).
Dana Bredemeyer measures competencies from 3 viewpoints - what you know, what you do and what you are.
For example looking at Organizational Politics
- What you know - Who the key stakeholders are, what they want from the business and personal perspectives
- What you do - Communicate, network, build relations, sell the vision/architecture
- What you are - comfortable with compromise and conflict, able to look at situations from several viewpoints, articulate, patient, resilient, sensitive to power flow within the organization.
Or if we look at leadership
- What you know - yourself
- What you do - mentor others, motivate others, build teams and set their vision, influence decision makers
- What you are - committed, passionate, credible.
Brdemeyer also supplies competency elaborations (levels for each competency and and guidelines on how to advance yourself between the levels) for Strategic alignment, Organizational Politics and Leadership .
Another interesting source on architecture competencies is a book called "Software Architecture - Organizational Principles and Patterns" by David M. Dikel, David Kane and James R. Wilson.
The authors detail a reference model of the organizational aspects of the software architecture process (vs. for example the SPAMMED Architecture Framework (SAF) which details the more technical aspects of the process).
The model takes about 5 principles :
- Vision - Deals with the mapping of future value to the constraints on the architecture and how well understood, flexible etc. are the architecture's structure goals
- Rhythm - the predictability and recurrence in the exchange of deliverables between the architecture group and the architecture consumers.
- Anticipation - The extent to which the architects predict, validate and adapt the architecture to changing requirements (as well as technologies, competition etc.)
- Partnering - the extent to which the architects interact with the architecture stakeholders to allow maximizing the value delivers or received by the different parties
- Simplification - achieving the "zen" (clarification and minimization) so to speak of both the architecture and the organizational environment where it lives.
An example for the patterns and antipatterns that relates to Stakeholders (first step of SAF)
- Criterion : The architect continually seeks to understand who the most critical stakeholders are, how they contribute value and what they want.
- AntiPattern - Phone Doesn't Ring
- Pattern - Know Thy Stakeholders
- Criterion: Clear compelling agreements exist between stakeholders
- AntiPattern: Lip-Synching
- Pattern: Reciprocity
The book naturally continues to describe what these patterns are :).
The important takeaway from this is that while knowing every nook and cranny of "the framework formerly known as Indigo" (WCF) will probably won't harm you - technical competency alone will only take you so far as an Architect and you can not afford to neglect growing and working on the aforementioned soft skills .
I recently saw a post by James Gosling (via David Strommer ) called the Eight Fallacies of Distributed Computing . These are eight assumptions on network almost anyone new to distributed computing assumes which proves to be wrong in the long run (and thus cause big problems and headaches).
I thought I'd try a to complement this list by adding few realities on distributed systems and data
- Expect a certain level of entropy in the system - Sites are never fully synchronized (unless you stop new data from pouring in)
- You can only afford to cache immutable data
- It very very hard to be able to scale indefinitely
- Observing global state is only possible via control messages
- It is hard to achieve distributed consensus (membership in a cluster, total order, commitment etc.)
- Expect to debug by log-files
There are probably many others - but these are the first few that came to mind :)
Architecture Description Languages (ADLs) sound like a great idea when you look at the "spec sheet"
ADLs represent a formal way of representing architecture
ADLs are intended to be both human and machine readable
ADLs allows simulation and analysis of architectures – completeness, consistency, ambiguity, and performance
ADLs can support automatic generation of the system (as it is represented by the architecture)
There are quite a few ADLs out there xADL , ACME/ADML, SSEP, Rapide, Wright and the list goes on and on.
I am guessing most of you didn't hear about any of those (well ACME appears in all those Willy E Coyote vs. Road Runner cartoons but that doesn't count). I think that the culprit lies in the fact that most (if not all) these come from the academic world where the focus lies on the models (in terms of semantics, completeness, rigorness etc.) and not on the practical use and applicability to day-to-day issues. Another problem lies in the fact that main-stream tools used by the industry.
By the way, UML isn't considered an ADL for several reasons for example the weak integration between the different model that inhibits automatic analysis.
The reason I am bringing the issue of ADLs up is that looking at the new architecture designers in Visual Studio 2005 (the Application Designer , the logical data center designer ,the System Designer and the Deployment Designer) actually form a set of DSL (Domain Specific Languages) that together can be treated as an integrated ADL. Furthermore Microsoft also provides an SDK for the"System Definition Model" that is the underlying model behind all these DSLs which lets interested parties extend and build additional designers.
While this model is not complete in several aspects ( some designers like the logical data center designer are too limited or there aren't enough views to cover all the architectural description) it is a good starting point in bringing the ADL concept into more practical and usable form.
This is part II of P is for Principles (and Guidelines and Constraints...) - Iteration I
In the previous entry I've said that it is helpful to provide a list of constraints and principles as it helps in limiting the scope of the solution and directing it toward good (and/or required) practices (Architectural, technological or business aligned). However, I also claimed that a simplistic list is problematic - I'll try to demonstrate this through a couple of real-world examples:
I recently reviewed the software architecture document (SAD) of a rather large software project, I saw that they it is mentioned that the project uses "service oriented architecture" - reading on, I saw the architecture builds on a distributed "shared memory" where every client and server has a full image of all the data entities, further more data entities are intertwined without any boundaries whatsoever. when I asked what's service oriented in that, I was told that the underlying distributed "shared memory" engine provides several services like dissemination, scheduling etc. The point here is that a catchy name can mean different things to different people.
On another project some of the stakeholders mentioned it is important that the solution would have an "Open Architecture" - sound good to me, but what the hell does that mean? based on open standards? promote extensibility (easy to add features)? promote replacability (easy to replace components)? all of the above? something else?
Furthermore if you don't understand the implications behind each of the principles you name - it gets very tempting to create a "Buzzword Oriented Architecture"- we want SOA, AOP, Software Factories, Smart clients, GRID, fault tolerance and whatnot... (Note - It is recommended to proceed with caution if you see too many buzzwords in your guidelines/goals. You might be trying to accomplish too much and/or you have too much marketing influence).
So what makes a good (or at least better) description for a principle?I currently use the following template:
Name - something easy to remember (e.g. SOA, Layered Architecture etc.)
Description - What does it mean
Rationale / Benefits - Why do we want to apply this principle
Implications - What does it mean to use it
Alternatives - What else - What are the other options we considered and why we didn't use them.
Scope/Exceptions - when and where does it apply
Note: I used to use a simpler template (whiteout Implications and Scope) the current version is based on a template by Ilia Fortunov from Microsoft UK. I can explain that further but it will probably be easier to understand through examples - so here are a couple from projects I worked on:
Principle Name: Code Generation
Description:Generate specific implementations and allow users to configure generated code via designers.
Rationale/Benefits: Increase longevity of the domain model (help separate from technical implementation). Reduce bugs via use of design tools.
Implications: Need to "templatize" solutions and develop code generators (need to check commercial solutions)
Alternatives: 100% object orientation and generic implementations. rejected due to tight coupling to technology, performance implications and impact on code readability.
Scope/Exception: Domain entities and any "aspect" related implementation (logging, security etc.)
Note: this is for a solution that has to use .NET 1.1 - had it been for a solution that relies on .Net 2.0 Another option - using Generics might have been a viable solution
Another example:
Principle Name: Distributed Database
Description: Each site shall have a separate independent copy of the DB and will have to synchronize its data with connecting sites (Note that I use the term synchronize and not replicate - as replication is a specific technical solution)
Rationale/Benefits: The inter-site communication medium is unreliable plus sites has to maintain autonomy.
Implications: The system has to cope with partial data. There's a need for conflict resolution policy. We need to consider idempotent messages for inter-site communications. Need a distributed primary keys management scheme.
Alternatives: Federated database - problematic since (some of the) data will not be available when sites are disconnected. Another solution weighted is a combination of centralized server and "off-line" capabilities upon disconnection - rejected as it would be more complicated (based on past experience) and have high-dependency on bandwidth (for on-line work)
Scope/Exceptions - Database layer and inter-site communication.
The next thing we have to deal with are constraints.As I've said in the previous post constrains originate from the different stakeholders and limit the scope of the solution. To document the constraints in a meaningful way I use the following template:
Name: something easy to remember
Definition: What does it mean
Implications: What does it mean for the architecture? what are the limitations it places.
Scope: where will we feel the impact
Origin: who placed this constraint and why
Again, lets look at a couple of examples:
Name: Use .NET 1.1
Definition (pretty obvious...)
Implications: use tools/products compatible with .NET 1.1; don't relay on .NET 2.0 capabilities (important for the mapping stage of the SAF)
Scope: All the system
Origin: using .NET is a company policy; we need .NET 1.1 since all the existing tools (e.g. ClearCase, XDE) only support 1.1
And another example:
Name: Deadline
Definition: We must have a working deliverable in 5 months.
Implications: Strive to reuse existing assets; try to migrate code from legacy version; try to model (an extensible) simple architecture (don't try to solve everything now - but try to leave flexibility for future growth)
Scope: Mapping stage; Architecture documentation; quality attributes of the solution
Origin: Customer (time-to-market)
One thing important to remember is that we are not interested in all the project's constraints (we are- only not in this context). The meaningful constraints are those that have implications on the architecture.
To summarize. Principles and Constraints help you limit the scope of the intended solution architecture. Principles are based (mostly) on past experience, constraints must be followed (and are originated from the different stakeholders). Using only a catchy phrase to describe either of these (Principle or Constraint) can prove to be problematic (creates confusion, doesn't really add anything etc.) and it is better to think about the implications of applying the principles and constraints. Lastly, principles (at the first stage) should be taken with a grain of salt - as they may not be suitable for the current requirements - you should be ready to reiterate and update them once you know more about the requirements (The end result would be to have a list of guidelines which are actually used for the solution's architecture).
Both templates (principles and constraints) are available for download on the template section of the SAF page.
Whenever you start a new project, even if you think you start for scratch - you don't really start from scratch. You always bring your past experience into the fray. Developing an architecture for a project is not different. the "Principles" stage of the SPAMMED Architecture Framework (SAF) is about bringing in your "lessons learned" and "best practices" as baseline rules or starting point for the architecture you are trying to build. laying down a good set of principles will help you limit the scope of the solution and focus on proven tactics.
It is important to note that principles you set should be treated as recommendations - the main reason for that is that they are based on past requirements and experience and not on current requirements.
An example for a principle may be "use layered architecture" Layered architecture is the practice where you define several level/areas of processing and limit the communication paths between them (common patterns are: layer can talk to the layer just below; layer talks to any layer below it etc. - the important characteristic is that communication is restricted). Layered Architecture brings a lot of benefits especially in promoting flexibility in deployment, modifying implementation etc. some of you reading this may think this is a trivial principle - however, sometimes it is good to put even trivial assumptions on the table, furthermore not everybody agrees that it is always needed see this for example. Lastly using a layered architecture has its risks for example the different layers can't scale to the same extent you may find yourself with scalability issues down the road. (I'll give more detailed examples for principles in the next post on this subject)
A similar notion to principles, in the sense of limiting the solution scope is constraints. However, unlike principles, these are not recommendations but rather these are limitations you have to follow. Constraints are set by stakeholders (their origin may be company standards, customer standards).There are several types of constraints:
- Technical - limit platform, reuse existing system/solution/component, follow a particular standard e.g. use Windows, .Net (alternatively use Linux, Java), use Web-services
- Organizational - follow a particular process, availability of the customer e.g. use RUP , MSF, company standard # 15 ..
- Business - time/money (deadline, budget). Another interesting example for this is "application freeze" - when an organization forbids change for a period - something some organizations did just before Y2K. [thanks to Andrew Johnston from www.agilearchitect.org for this one]
So how does it works - well, you bring in your architecture team along with some of the technical stakeholders - go through one of those "brainstorming" meetings and come up with a list - e.g.:
- Build on Open standards
- Reuse
- SOA
- Layered Architecture
- support scale-out
- .
- .
- .
Wait, what's wrong with this picture - well, for one it is too simplistic view (it doesn't say much) and even more importantly it not accurate (i.e. it can mean different things to different people) - on the next post on this subject I will show a better way to document and understand principles along with a few examples from real projects.
[thanks to Eliaz Tobias from MS Israel for the link]
The beta version of MSF 4.0 formal is available for download. This is an interactive process which is compliant with CMMI level 3.0 (see http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr029.pdf chapter 7 - maturity levels) and supported by the tools (i.e. Visual Studio Team System 2005). This can be great news to organizations like the one I work for these days who are certified for CMMI 3.0 and can (hopefully) stay compliant with less bureaucracy.
Looking at this process from my perspective (i.e. as an Architect) it also looks interesting. It defines several roles for architects (e.g. in domain technical level as Subject matter experts and in the solution level as an Architect). The process suggested is tailored/aligned with VS2005 capabilities (and thus somewhat limited) however many of the steps are both viable and important. For example it has parts that do with quality of service requirements (what I call Quality Attributes in SPAMMED). My particular favorite step is "Assess Alternatives (LAAAM)" which I helped introduce ( :) ) in my previous job as an Architect for Microsoft Consulting Services
It is not a secret that user involvement increase the success rates of software projects. We can basically look at the architecture as a mini-project inside a project. The users in that case (as my entry on Stakeholders shows (or tries to)) are the stakeholder.
Richard Demers talks about Architecture Control Board (on his smalltalk blog - oh my :) ) as a way to increase the involvement and of stakeholders for large software projects. The idea is to engage as many stakeholders groups as possible on a periodical basis and to set up a review and change board that approves changes in requirement and also review and approve the architecture (I'll talk more about something similar when I'll get to E - Evaluation in SPAMMED).
Richard lists 13 points in regard to the Architecture Control Board - the most important ones (in my opinion) are
- the ACB set the requirements scope for the architecture (what requirement/quality attributes should be accounted for)
- the ACB review and criticize the architecture - they don't, however, design it or vote for its correctness.
- the architecture team has final say on architectural decisions (though an escalation path to upper management should exist)
- the documentation approved by the ACB is the ultimate deliverable of the architecture team (i.e. the "Software Architecture Document")
- go read the rest :)
While establishing a "formal" ACB in smaller-scale projects is probably an overkill you may still want to follow some of these tactics on a less formal basis to increase your stakeholders involvement and more importantly cooperation.
There's a lot of confusion on what Service Oriented Architecture (SOA) is and isn't. Martin Fowler sums it up nicely in his Bliki (Blog+Wiki). Clemens Vasters blogged something similar back in May. Whether you agree or not - one thing is sure - there's so much hype around SOA these days that it is hard to understand the realities.
The Composite UI Application Block (CAB) is an interesting architectural solution for smart client development. It is essentially a plug-in architecture that brings the WebParts concept to the desktop.
The CAB has the following elements:
- WorkItems. These are the classes which represent use cases in your application and contain the business logic for those use cases.
- SmartParts. There are the building blocks of a Windows-based application - similar in concept to WebParts.
- Workspaces. These are helper classes that can display SmartParts with a uniform style.
- UIElements. These are elements such as toolbars and menu bars that are shared by SmartParts within the application.
- Support services. These include:
- Event broker service to manage the publishing and subscribing of events between SmartParts.
- State management service to hold shared state for areas within the application. This provides an option to encrypt the state before it is stored.
The CTP is now available on GotDotNet (needs .NET 2 beta)
It should come as no surprise that the first pillar of the SPAMMED Architecture Framework (SAF) are the stakeholders - after all at least some of these are the people/organizations that are the cornerstone of the software project itself.
What exactly is a stakeholder - EIA 632 , a standard for System Engineering, defines a stakeholder as "An enterprise, organization, or individual having an interest or a stake in the outcome of the engineering of a system".
Sounds good enough to me :) - but before we delve into more details on how to identify these stakeholder, what do we do with that information, we first have to understand why it is important to us as architects. The short answer was already mentioned stakeholders (most obvious examples are Customer, Project Manager) are what makes the project tick. That, however is just the beginning of it. One of the primary responsibilities of the software architect (much like the project manager by the way) is to balance the stakeholders interest to ensure the success of the project. In the SAF sense the stakeholder are important for several reasons:
- The solution is developed to serve their needs or goals (at least some of them i.e. customer, end users, management)
- They serve as the source for constraints (and sometime principles)
- Their concerns can help elevate needed quality attributes
- Stakeholders can (and should) help evaluate the architecture
- The documentation of the software architecture is targets at stakeholders
Several stakeholders are pretty common to any project . The following list shows them along with samples for some of the concerns (or vested interest) they have regarding the project:
- Customer - Functionality, price
- End-User - Ease of use, performance
- Project Manager - On time delivery, development costs
- Management - Price, reuse
- Developers - structure and dependency between components, interesting technology
- Maintainers - ease of debugging, modifiability
- Testers - Testability, Traceability
- Security Analysts - security
- Project New comers - structure and dependency between components, traceability
- Customer’s IT group - ease of installation, stability
This list is a nice starting point but it is just that - a starting point. There are still a few things we may want to do
- Identify additional stakeholder - sometimes there are less common or obvious stakeholders (e.g. System engineers, shareholders, safety analysts, the general public etc.)
- Map stakeholders relevance - Jaap Schekkerman from suggest prioritizing stakeholders by power vs. interest. I believe the privatization should also include the (stakeholders) concern importance:
- Lastly you may want to document your stakeholders so as not to forget what they are interested in. This documentation can include the constraints they place their concerns (translated into quality attributes) and a list of viewpoint that are needed to satisfy them (i.e. explain the architecture to them). A template for documenting stakeholders is available from the SAF page.
Everybody talks about "Software architecture" these days (your humble servant included...) - but what the hell is it?
I mentioned in my first SPAMMED Architecture Framework post. that Software Architecture "deals with the major components or structures, their relationships and interactions. It encompasses the major (read hard to change) decisions and their rationale and every system has an architecture (even if it is a default one)" - OK sounds nice, but is it enough?
Probably not (If I thought it was enough - I wouldn't have wrote this post...). There are literally dozens! of definitions for what (solutions) Software architecture is ( you can see many of them here). I am not going to quote all of them, instead, Lets spend some time looking at some of the more prevalent characteristics found in most definitions
- Architecture is Early - It represents (well at least, should represent) the set of earliest design decisions which are both hardest to change and most critical to get right.
- Every system has an architecture - even if it is just a default one (i.e. it can be described using reverse engineering) it still there.
- Architecture is about breaking a system into components and setting boundaries. It doesn't describe all the components - it usually deals with the major components of the solution. Also it doesn't describe the complete characteristics of components - it mainly deals with their interfaces or other aspects that has to do with their interactions.
which brings us to the next point:
- Architecture is about the relationships and interactions of components. Again we are interested in the behaviors of the components as it can be discerned from other components interacting with it.
- Architecture explains the rationale behind the choices (vs. the choices not made). It is important to understand the reasoning as well as the implications of the decisions made in the architecture since their impact on the project is large. Also it can be beneficial to understand what alternatives where weighted and abandoned (for future reference, when/if things needs to be reconsidered, and for anyone new to the project that needs to understand the situation).
- There isn't a single structure that is the architecture - there's a need to look at the architecture from different directions or viewpoints to fully understand it.
There's a very interesting standard called IEEE 1471-2000 "Recommended Practice for Architectural Description of Software Intensive Systems" which defines the relations between the system stakeholders and the different viewpoints of the architecture. It serves as a good reference for understanding how to document an architecture - it also means that identifying the needs of the different stakeholders will tell us a lot what views (details of the architecture from a specific viewpoint) we need to have.
- Architecture is the first design artifact where a system’s quality attributes are addressed
Stakeholders are also the main source for these "quality" requirements and It is the architect's responsibility to balance the quality attributes of the system. This and the previous point are the main reason that the SPAMMED process first step has to do with identifying stakeholders (And I'll elaborate more on this on the next SPAMMED related post - "Stakeholders Everywhere")
Apart for the quality attributes side of the last point - it also basically state that:
- Architecture is design (but not all design is architecture)
Which raises another interesting question - what's the difference between architecture and design. But this will have to wait for another post as well
Assuming the previous post made you curious - here are a few links to get you started:
The architect doesn't talk, he acts.
When this is done,
the team says, "Amazing:
we did it, all by ourselves!" (17) (The Tao of software architect - Philippe Kruchten)
On the surface - When it comes to agile development the role of the software architect is a little more blurred.
The most obvious aspect where architects is for the technical architecture. An experienced technical architect, can greatly enhance any project by steering the designs into the "best" directions (under the chosen platform constraints). The technical architect can also promote reuse etc.
Additionally while the requirements change a lot and not fully defined, the quality attributes of a system are more stable - if you need performance, then you need performance!. More so, there are some qualities that are inherent to agile process - for example you want to put an emphasis on flexibility and maintainability - if your developed solution does not have these it is going to do refactorings. Thus, depending on the size of the project you may want to use an architect to help set the ground rules in the first couple of iterations.
Another area where architect involvement can be very beneficial is when you try to scale an agile project a good choice is to try to break the project to smaller loosely coupled projects (see this paper for example) - well, this is just what we (architects) live for...
By the way, few agile processes, define the architect role up-front, one such process is MSF 4 Agile. Note that MSF (Microsoft Solutions Framework) version 4 comes in two "flavors" one that is aligned with CMMI and the other a (much) more light version, the afford mentioned MSF 4.
How does the SPAMMED process fare with an Agile project - (surprisingly enough) I would say pretty well
First of the architect should be hands-on i.e. part of the development team (most likely the technical lead)
- Stakeholders, you would probably want stakeholders on-site for any architecture related meeting (just as much as you would want the customer on site for other activities)
- Principles would include things like TDD, Simple Implementations, Refactoring
- Quality attributes would hold Flexibility and maintainability and a couple or so of the important project qualities (performance, availability..)
- For the modeling you would chose very few views and would try to focus on ones that have manifestation in deliverables
A good example for this is the Application designer in Whidbey (see screen shot below) - the result of which is the projects structure for the solution
- Mapping doesn't really change
- Evaluation would be focused on proof of concepts and/or skeletal architecture.
- Deployment - well, being part of the team... you would notice if your decisions were wrong or circumvented
To sum up - yes Agile projects can and many times should use an architect to help it stay on track - hey, even XP has an architectural spike...
I am writing an entry on the architect's role in an agile project and I noticed I am using the term "Technical Architect". My first intention was to go on and explain that within that post, but I then thought it would be better to give a specific entry that paints the complete picture (well, at least, as I see it :) )
There are three basic classes of software architectures:
- Infrastructure Architecture - Has to do with presenting an workable infrastructure solution (types, deployment and configuration of servers, LAN, etc.) it basically deals with the overall layout of the "out of the box" solutions that help solve the problem. An example for this is MS's Windows Server System Reference Architecture
- Business Architecture - Concerned with the business model as it relates to an automated solution. It has to do with the structural part of requirements analysis, and it is usually domain specific (sometimes the job of a business analyst). Domain architecture is ideally technology neutral, although more often then not it is cluttered by technical constrains.
- Technical Architecture - Specific to technology and the use of this technology to structure the technical views (esp. Technology Mapping) of an architecture. Technical architects usually have a very good understanding of a technology (e.g. .NET, J2EE etc.) and how to best solve problems using that technology.
Additionally there are two compound classes
- Solutions Architecture - Specific to a particular business area (or project) but still reliant on being a technical focal point for communications between the domain architect, business interests and development.
- Product Line Architecture - The architecture that has to do with families of related solutions. It is basically the same as solutions architecture only with an extended scope. A product line architect has to promote reuse and identify commonalities between several solutions while on the same time providing each of the solutions with enough specifics as to make it a good viable solution in its own right.
Lastly there's the more comprehensive architecture class which is Enterprise Architecture (EA). EA deals with the governing logic and strategy of the a firm's core business processes and IT capabilities. It is a set of recipes (policies and principles) along with technical constrains (that are set for the different solutions within the enterprise). The EA is concerned with cross project/solution architectures and tries to lay the rules needed to achieve the business standardization and integration requirements of the firm’s operating model.
One of the most interesting things that MS is building these days is the set of tools around their software factories initiative.
Software Factories, is actually a methodology that builds on the concepts of Domain Specific Languages (DSLs) along with frameworks, patterns and guidance.
The idea is to bring the notion of domain modeling and DSLs from horizontal markets (e.g. the form designer in any modern IDE) to the everyday use of vertical markets (i.e. your next application) and help realize the promise of product lines. Many of the ideas in software factories are not new (see for example this article from 1993) - however, today,as the Jack Greenfield et al state in their book, a number of needed technologies has (finally) matured enough to make it feasible. Furthermore What's unique in this effort is that Microsoft is making the effort to back the ideas with tools that will enable architects and developers to put them into use.
A software factory needs 3 elements a schema, a template and tool to support them (a development environment). The schema is basically the recipe containing the overall information on how things should work together. The template is the set of DSLs, samples, frameworks that are used to create the factory and lastly we need the IDE that supports them (e.g. Visual Studio "Whidbey" - although, at least technically this can also be achieved in Eclipse etc.)
I am, personally, very interested in this initiative. We already use code generation all over the place (DAL, serializations, Data Entities etc) and employ a domain driven approach even today. I would love to have "user-friendly" tools that will let me as an architect achieve these goals more easily.
One final note - yes I am aware of MDA (Model Driven Architecture) but all the MDA tools I've encountered thus far are lacking in their usability. This, by the way, brings me to one caveat with software factories - it is not based on UML 2.0. I would rather have a DSL that is based on stereotypes, tagged values and OCL. This would allow for a specific view using the appropriate designer but also for backward compatibility (and a downgraded view) on other tools. I do agree though, that UML 2.0 has its own share of problems - but that's a matter for another discussion altogether
The previous post on SPAMMED introduced the different pillars of the process. This one focuses on their interactions
The state chart below (modeled using Sparx System's excellent Enterprise Architect) shows the possible transitions between the different process steps
Eliciting stakeholders is usually the first step to take - the reason is that stakeholders serve as the base for several of the following steps for example stakeholders concerns serve as a guideline for deciding which views to document during the modeling step.
The next step is documenting principles and goals - which are based on past experience. The real reason however that this step follows the Stakeholders elicitation is that the step also includes documenting constrains, and most of these are set by the different stakeholders (e.g. use .NET/Java because it is a company standard).
Quality Attributes builds on the former two steps, deciding which quality attributes are important and balancing them is based again on the stakeholders concerns and the principles you've set.
You would then follow this with some modeling and then technology mapping.
Up to this point the process has been pretty much "waterfall-like", however after you evaluate your former steps, you may find that you need to revise any of the preceding steps. This is why you want to get here as soon as possible. do not try to complete and "finalize" the architecture and only then perform the evaluation. Developing the architecture, is very much like the other parts of the development life-cycle, and can benefit greatly from a short feedback loop. Evaluating early helps prevent the architecture from being a bottleneck holding all the development process and even more importantly, building too much structure based on false assumptions/decisions.
Deploying the architecture, i.e. releasing it to the designers and developers is the next step (assuming the evaluation was OK). Once the architecture is "out-there", the realities of the chosen technology/product, changing requirements, budget/talent/ time constrains and most likely the mere iterative nature of modern software development will probably still make you evaluate and retrace some of your steps as a result. I believe this refactoring is a healthy procedure - in the first few iterations (the actual number depends on methodology used, iterations length as well as project size) of your project. Nevertheless if the number of times the architecture has to be reevaluated and changed or alternatively the number of changes is large - it is probably a sign that your architecture is in need of a serious overhaul.
Microsoft has recently went public with their Architect certification program - While (naturally) they expect knowledge of the Microsoft platform ("about a quarter of the emphasis" to quote the site), it also has a lot of of emphasis on architect competencies and general know-how. Way to go MS :)
On the Java side of the fence there are already architect certification programs e.g. Sun's or BEA's.
There are also two (that I know of) non-vendor specific certifications - one from the Open Group (they also brought us TOGAF - which is worth its own discussion) and the other from the Software Engineering Institute (SEI @ Carnegie Mellon university)
I 'd be happy to learn of any other architect's certification programs if you know them
Well, I haven't heard about that one yet - but there are few other groudps and organizations out there. Here are the ones I know about:
General overview of the SPAMMED process for developing software architectures
|