One of the most interesting presentations in Architecture & Design world was the eBay Architecture presentation by Randy Shoup and Dan Pritchett. The presentation was only one hour long, so Randy and Dan didn't cover all the topics in the slide. Here are some of the insight I took from this presentation.

Architecture evolution  -  eBay actually went through several architecture revolutions. Their initial architecture cannot even begin to scale to their current loads. It was, however, a very good fit for their initial quality attributes - specifically, the emphasis on time to market and costs.  This shows the importance of balancing quality attributes. Sure an architectural change is painful but if they'd future proofed too much I doubt they would ever get something working.

V2 demonstrated that traditional 3-tier architecture would only scale so far. It was nice to see how it evolved though. Also with the move from version 2.4 to 2.5 and later to 3 we see eBay learning about  CAP - the hard way. In its final (current) incarnation eBay's data architecture prefers partitioning and availability over consistency. This doesn't mean they forgo consistency altogether - just that they trade the comfort zone of ACID transactions with the BASE approach. Where BASE - stands for Basically Available, Scalable/Soft state & Eventually Consistent. .
eBay partitions thier data in two levels one is a SOA like division by business areas (users, items etc.) and the second level is an horizental partitioning based on access paths.This BASE approach to data was dubbed by Dathan Pattishall (from Flicker and Friendster) as sharding (via HighScalability). This approach means things like high partitioning, no distributed transactions (also see below), denormalization etc. (you might also want to read the item I wrote on denormalizaiton in InfoQ yesterday).
The more major implication here is that when it comes to internet scale, the database looses its importnace - or as Bill de Hora nicely puts it:

The use of RDBMSes as data backbones have to be rethought under these volumes; as a result system designs and programming toolchains will be altered. When the likes of Adam Bosworth, Mike Stonebraker, Pat Helland and Werner Vogels are saying as much, it behooves us to listen.

As I said the data architecture of eBay is SOAish -  partitioned their components and data along business lines, and they apply many of SOA principles. They don't however unite data and components to create a service and  they don't (seem) to have the same contract boundaries that SOA promotes (Randy told me that they are currently contemplating SOA).

Returning to the  eBay do not use transactions. "no transactions" which seems very controversial  - but if we just consider some of the points I made on transactions between services in previous posts - it is the only logical way to ensure scaling. By the way, as can be expected  they do use transactions - when they are local e.g. if the users table is spread over a couple of table both will be updated together).

The application layers also follow the segmentation by business areas. eBay cacse metadata/immutable data as much as possible. keep the application stateless (i.e. state comes from client/db) e.g. they don't use sessions. The DAL virtualized the horizontal partitioning mentioned above for the rest of the code.

It was also interesting to  that eBay developed its own messaging infrastructure - though Randy and Dan did not provide alot of details on that

Development process - It seems that eBay is using some hybrid of feature driven development with waterfall (i.e. the development is feature by feature - but the development of a feature is waterfallish). The do have a constant delivery rate which they synchronize using the concept of a train. if you have a features that is it will be added to the train which is scheduled to arrive around the time your feature will be ready. Several features are delivered as a package which gives a predictable (weekly). I guess it also gives them some nice metaphors to use such as a feature that doesn't make it - misses the train or the train leaves on time etc.

The slides of the presentation can be downloaded from Dan Pritchett's site (They not from the same event but they are pretty much the same slides. Also you can read Elliotte Rusty Harold's account of the presentation.

I won't say anything about my presentations (that's for others to say :) ). The point of this post is just to let you download them. So here they are:

• SOA Patterns (2.14mb) - Takes a look at different strategies (patterns) to solve common SOA pitfalls
  
• Getting SPAMMED for architecture (4.56mb) - Takes a look at the activities architects can/should do when they think about software architectures. The presentation also covers architecture in agile projects.
  

I've got a few emails from people who asked about the commit and consensus protocols (Paxos, 3PC etc.) mentioned in the previous post.
I was going to write something up but then I find a very good overview by Mark MC Keown which provides "A brief history of Consensus, 2PC and Transaction Commit". This short writeup provides links to the most important papers like the Fischer, Lynch and Paterson (FLP) "Impossibility of distributed consensus with one faulty process", Jim Gray (who, unfortunately, was lost at see earlier this year)  and Leslie Lapmort's "Consensus on Transaction Commit". In addition to Mark's post and all the papers there you may want to check out the presentation Jim and Leslie made on paxos commits

By the way, if on the other hands, you already know all this (and a little more) than check out Werner Vogel's (Amazon CTO) blog as there's a interesting  job opening for you there

Following the previous post I had a chance to exchange a few email with Mark Little (the director of engineering in the JBoss division of Red Hat). Mark thinks the topic of transactions and SOA has been beaten to death already and wonder's why does it need to resurface (see his post "Is anyone out there?") - I don't see a problem with discussions resurfacing when new people are faced with situations others already solved (but that's a matter for another post)

Anyway, the reasons we're here is  that I think that during this conversation  mark made a few interesting observations and I think the end result is pretty interesting. I decided (with his permission) to post it here ( It is only minimally edited: no deletions, few additions (in []) and a few time shifting to make it more coherent as a single conversation)

Mark : From what I can see it's [the arguments on transaction and services - are] the same old arguments that have gone round and round, ignoring the important fundamental issues and not doing enough background research.
Sagas are transactional - it's just an Extended Transaction model and not an ACID transaction model. Don't get hung up on the word "transaction", which is way to overloaded in our industry to actually mean anything by itself. Plus, 2PC is a consensus protocol too; it does not impose any other aspect of ACID than the A. Even the D is optional until/unless you want to tolerate failures.

Arnon I know this is an old argument - but that doesn't mean it isn't worthwhile

Mark It isn't worthwhile if people aren't going to listen I've been involved in these debates so many times over the past 7 years (for Web Services transactions) and longer for extended transactions, that it gets a bit old after a while. Maybe we should create a wiki page and point people at that ?

Arnon I guess, but you should keep in mind that people who are solely in the .NET camp only got WS-AT recently with Windows Communications Foundation so you can expect the issues to resurface. By the way a wiki might not be a bad idea 

[regarding 2PC.] 2PC is a distributed consensus protocol and in principle doesn’t have to be related to ACID transaction. But I think the common view and use of it is for ensuring distributed ACID behavior. Looking back at my experience with XA and COM+ transactions it seems it does a good job at achieving this ACIDness


Mark This is an education issue. The literature is clear on this. People who know and understand transactional protocols don't make the mistake of equating 2PC to ACID properties.

Arnon Yes it is an educational issue . But I am not sure that it is that common knowledge. It is expected that middleware vendors who build the tools to support these protocols to understand it better - I don’t think it is that widely known outside these circles. Most of the architects I’ve don’t (maybe It time to look for new friends )

By the way as 2PC is not resilient to failures of the coordinator so in a highly distributed environment like SOA it might have been a better idea to go with paxos commits if at all you go down that path.

Mark The reason WS-AT and WS-ACID chose 2PC is: interoperability. All TP monitors support it. Try getting IBM, MSFT, Oracle, BEA etc. to change to Paxos, 3PC, flat-commit, or anything else and you'll be waiting for the heat death of the universe.

Arnon Can’t argue with that

Mark [also] 2PC is resilient to failures if the coordinator eventually recovers. Paxos has its own failure assumptions too: Jim never disputed this. Same as 3PC and other consensus protocols. As with *any* fault tolerance approach (transactions, recovery blocks, replication, etc) it's always probabilistic. All we're doing is making it highly unlikely that the system cannot complete, but we can never make it entirely safe. Even in the airline industry they can "only" go to a probability that failures happen .000000001

Arnon You are probably right that in an SOA situations the chances of not getting an ACID transaction are worse than in a controlled environment - which actually make the situation even worse since people using WS-AT perceive it as allowing them ACID interaction (e.g. Juvals podcast) .


Mark WS-AT is *all* about ACID, in the same way WS-ACID is about ACID transactions. It is *nothing* to do with SOA though. Web Services are not purely the domain of SOA implementations!

Arnon I totally agree that Web-services and SOA are not directly related and can each exist independently of each other. Again this is an educational issue but,  SOA==Web-services is a very common misconception (I guess the word “service” in web-service doesn’t help )

[in any event] I think distributed transactions in general should be used carefully period.


Mark Absolutely. They are not a global panacea and people who push them as such do more harm than good.

Arnon WS-AT is more problematic than regular distributed transactions as by definition in an SOA you do not know who and how many other services will participate in your transactions so you are much more likely to run into problems.

Sagas which embrace the temporal shift don't give an illusion of ACIDness and allow to focus on achieving distributed consensus while keeping all parties involved consistent. I think that it is a much better option if you need transaction-like behavior

Mark For SOA, yes. Although Sagas are only good for a certain type of use case. That's why we've always tried to develop "live documents" that allow people to add new models when/if needed. With a couple of exceptions during the BTP days, there has always been consensus that one size does not fit all (http://www.webservices.org/weblog/mark_little/blackadder_and_the_micro_kernel_approach_to_web_services_transactions).

Web Services *anything*, whether it's WS-AT, WS-Sec, or WS-Addressing all have their non-SOA aspects because Web Services aren't developed purely with SOA in mind. If that were to happen then Web Services as a technology would lose some of their important benefits immediately.

Arnon This whole discussion is in the context of SOA (at least from my side ) – naturally there’s a place for ACID transactions for other uses.

Regarding Sagas - calling them "Extended transactions that are not ACID" is just semantics - my point was that they are not ACID transaction. I think most people equate transactions with ACID transactions as well (but I may be wrong)

Mark Many people do and that again is an education problem. The term Extended Transactions (don't need to say "that are not ACID") has a well defined meaning in the R&D community. There have been many good models and implementations around Extended Transactions. They really took off in the vendor community through the Additional Structuring Mechanisms for the OTS, back in the 1990's. If you check that out you'll see that it formed the basis of WS-TX and WS-CAF. Even in Jim's original technical report he discussed relaxing all of the ACID properties in a controller manner to get more flexibility. That was the first extended transaction. In fact, ACID transactions are just one type of extended transaction. There are many many others, including nested transactions, coloured actions, epsilon transactions, sagas etc.

Unless I qualify it beforehand, I try to never use the term "transaction" in isolation because it has different meanings to different people. For example, when talking to developers working in trading infrastructures, a "transaction" isn't an ACID transaction at all. In telecos it's different again.

Evan H asked a question about distributed transactions and services in the MSDN architecture forum:

Are distributed transactions (ie.. WS-Transaction) a violation of the "Autonomous" tenant of service orientation?   Yes or No and Why?  Kudos if you can address concurrency and scalability (in an enterprise with multiple interacting services).

I answerd this questions back in april when I wrote a couple of posts that explained why cross-service transactions are a bad idea:cross service transactions and some more thoughts on cross service transactions.
Roger Sessions also agrees with this view (well, it seems actually, he wrote about it well before I did :) ):

When the WS-Transaction specification was first proposed, back in 2002, I wrote an article explaining why I thought the idea of allowing true transactions to span services was a bad idea. I published the article in The ObjectWatch Newsletter, #41: http://www.objectwatch.com/newsletters/issue_41.htm. Nothing since then has changed my mind. Atomic transactions require holding locks, and spanning transactions across services requires allowing a foreign, untrusted service to determine how long you will hold your very precious database locks. Bad idea. Just because IBM and Microsoft agreed on something doesn't make it good!

The reason I am bringing this issue back is that Juval Lowy (who wrote the article that triggered my first post on the subject) has recorded an Arcast with Ron Jabobs. Where he re-iterated the idea that "Transactions is categoricaly the only viable programming model" and you should strive to use it whenever you can. It seems Juval admits you sometimes need to use Sagas (which he called "long running transactions" - you can see in my link why I think that's a wrong name). He also agrees that you can also use a transactionable transport and then only do internal transactions from each service to the transport (a pattern I call "Transactional Service"). However, at the end of the day, he still thinks you should use WS-AtomicTransactions whenever you can.

I agree that transactional programming is important. I think it is the simplest programming model (from the developers side). I would probably never write an interaction with a database that is not transactional; I look very favorably at initiatives for in-memory ACI (no Durability) transactions such as the one Ralf talks about.  Until we get to Distributed Transactions...

First, we should note that transactions are not "the only viable" option.As Martin Fowler notes Ebay seems to be doing fine without distributed transactions. Not only that, they abandoned distributed transaction and went "transactionless"because they needed one simple thing... Scalable performance .

In most COM+ scenarios you have a single server or a few internal servers where the distributed transaction happen - and even there you should plan your transactions carefully if you want to get any kind of decent performance. In SOA scenarios the situation is more complicated as the distribution level is expected to be higher (even if you don't involve services from other companies). More distribution means longer times to complete transactions (especially if a participant can flow the transaction and extend it). It also means increasing the chances of failure (see Steve Jones series of posts on five nines for SOA). In my opinion, the more distributed components you have the more you want their interaction to be decoupled in time - i.e. the opposite of transactions.

Juval also said he doesn't buy the denial of service problem I mentioned (supporting a transaction means you allow locks - if an external party doesn't commit you retain the lock..). Juval said he assumes that a solutions has both authentication and authorization so this shouldn't be an issue. For one, I have seen too many projects where security was something that was neglected or quickly patched in at the latest moment - so I would hardly assume security. Even with security on - you increase your attack surface.
But that's just the half of it. Even if all your service consumers have good intentions - you still don't know anything about their code. SOA is not like the "good old days" where you owned the whole application  - this means you cannot trust their security to be ample. Also you don't know anything about their code quality. Services are likely (in the general case) to be deployed on different machines, even if they start co-located. I think that a Service boundary should be treated as a trust boundary just like a tier boundary. I strongly believe you should have reduced assumptions on what's on the other side of the service's boundary - transactions are not reduced assumptions

SOA and distributed transactions do not go hand in hand - it isn't just autonomy at stake here. It is a problem for performance and scalability and even security period.

To finish this post - I would also highly recommend looking at Pat Helland's paper "Life Beyond Distributed Transactions: an Apostate's Opinion" and a post he recently made called  "SOA and Newton's Universe", where he explains more eloquently than I ever could why SOA is not a good fit for distributed transactions.

Steve Jones has (yet another) great post called "Le Tour SOA - why support services are critical, but not important".
You should go read the article - but in a nutshell, Steve explains that important services are the ones that bring business values and critical services are the supporting ones that help keep the light on for the important services to function properly. 

While the post has SOA in the title. I think it is more general and is also applicable to applications or any other IT generated components. In fact it can also be applicable to IT itself as Nicholas Carr noted in 2003 when he published his paper "IT doesn't matter". Nicholas argues that IT will become akin to electricity and as such be critical for the business to continue operating but not important. As a side note I'd say I think this is might be true for traditional businesses but not for businesses where the IT is the business (such as banks, insurance companies, etc.)

Back to critical vs. important - I think this is an important for architects to make this distinction to be able to prioritize work and not confuse business value with semblance of business value due to criticality for operations. This doesn't mean you can neglect critical tasks (after all they are critical...). It is the important stuff that will bring your business the competitive edge.

A couple of quick  observations following the Events and temporal coupling post

Events, Current data and aggregated data all have Time-to-live aspects.

• Events value usually diminishes over time until the TTL reaches
• Current data usually have a constant value while their TTL lasts (until a new value is the current data) - unless we are talking about version data which is a  component of or a step  in the direction of aggregated data.
• Aggregated data  has the longest TTL, it is interesting to note that its value increases over time

Also while the Current data TTL is determined by the producer both Events and Aggregated data TTLs are determined by consumers

Yeah, I know these are not not earth shattering observations but I still think they are  interesting
 

You raise an event when something interesting happens to you, you think it is important, but you don't care enough to know who is interested. you are even less interested in to personally going and to each an every interested party and letting them know. So - instead, you raise and event, and let the poor buggers take care of any implications by themselves. We raise the event "now" when the change happened - it is only important now anyway...


Looking from the "poor buggers" -the event consumer point of view things are more complicated. There are events which are cyclic in nature like stock price updates, the blips from a sonar etc. if you missed one, then it isn't really important you'd get the right information in the next update (actually, that isn't entirely true - see later in this post). Then, there are the events which only occur once. sometime it isn't important for you to listen to them if you are not up and running in the same time. Other times you can't afford to lose an event for instance if your ordering service (or component for that matter) communicates with the invoicing one using events you don't want to miss the event of a new order else you would loose money.

This basically means that the event producer and the event consumer are coupled in time - one way to solve it is to make sure both of these services are available at the same time i.e. if the invoices crashed, then processing orders should be suspended (note that this doesn't mean that you don't accept orders just that you don't process them).
Ok - maybe we can just raise the event "transactionally" - this would probably work, but we need to remember that the event producer doesn't really care about the event consumers, why would it want to fail because of them?!
Maybe a better way would be to "raise" the event over some reliable transport  - this has a few problems. one is that we've passed the problem to the connection between the event producer and the transport. It might be acceptable to have a transaction between the event producer and the transport. However, as I've already said the producer doesn't care much about the consumers..
We can have persistent subscriptions for existing consumers to prevent events from getting lost which make both creates a er minor problem that new consumers can't see past events but also has the risk of existing subscribers disappearing and their queue can then grow endlessly (or until an administrator would remove the subscription)

Ok, let's try to look at the problem from a different angle. looking at the events, what we can really see is that an event has a time-to-live (TTL) as far as the event consumer is concerned. For instance in the case of the cyclic events the TTL is the interval until the next  event. Actually, even with cyclic events the TTL might be larger - if we are also interested in analyzing trends or  ab normal occurrences (which is why I said it isn't entirely true we don't care about old events). In case of one-time events the TTL might be indefinite or maybe even then it might be some definite value (one day, week, year etc.). Since we can't know about the TTL of consumers it can be a good idea to make past events available somehow.

Thus, when you design an event centric architecture like  EDA (whether on top of SOA or not)  it is important to think about event consumers - we don't want to think about  specific  consumers since it negates the benefits of thinking in events, but I would say that you want to think about event consumers in general, after all your component is also an event consumer (do unto others as you would have them do unto you)

One option, which I already talked about, is to make past events available as a feed. Event consumers can then come at their own leisure and consume past event  (this can be in addition  to  to raising the events in real-time). This provides a partial solution as the maximal TTL is determined by the event producer (after which the event is deleted from the feed). This may be acceptable but you must be aware of that.
The other option is to to log all the events and provide an API to retrieve past events. In a sense the max TTL is still at the hands of the event producer only if you use a database it would probably be a large time compared with a feed. Alternatively the events can be logged on by a central "always present" event aggregator (in a manner similar to the aggregated reporting pattern I described for SOA).

To sum all this - events they seem only to matter in the instance in time they are created, we are used to that thinking from building OO systems where all the components are co-located in the same address-space and time (even there I can think of scenarios where we would want past events) - in a distributed world events need to have a TTL, the TTLs can vary and are determined by the events consumers. Lastly, as I demonstrated in the paragraph above, there are several strategies we can use to help solve the event TTL dilemma (and there are probably a few others).

Few months ago I wrote here about solving the mismatch between Service Oriented Architecture (SOA)  and Business Intelligence (BI) (see papers and articles section). Recently I got the following question from Ben:

One major question I have is around large data sets. As an experienced BI/DW architect and developer I have worked on a number of large scale data warehouses. Retrieving large data sets (i.e. millions of records) doesn't seem to fit well into SOA. As you state in your article, we could have another point-to-point interface, where the service which houses data we need gets a request and writes out a batch file (xml or plain ascii text). Then using typical ETL, we grab the file and load it. The underlying source system (service) can use optimization in generating a large data set (vs. record by record) and
the data warehouse can correspondingly load in bulk.

Like most architectural questions - the answer is "it depends"
For instance, if you do a run-of-the-mill ETL as a on-time setup then it is just that- a one time setup and I, personally, don't see any contradiction between SOA goals or tenets and that.

I do think that iit is better to enhance SOA with EDA interactions to provide a long term solution to the BI problem. You can also have a dedicated component that aggregated the information that flows in in these events and builds batch files that are suited for the ETL you've used during the setup phase (mentioned above).
It is true though that moving an SOA which is already in-place to EDA is not a small feat, but adding EDA layers does not have to mean that the old interfaces go away - especially not immediately (remember to treat services as products)

If you have a business that generated millions of records on a daily basis - then the situation is more complicated. Now you have to think about the trade-offs between "compromising" SOA and adding a dedicated interface (or a backdoor to the database) for the ETL vs. the implications of performance, bandwidth, transition costs, ROI  etc. of pushing that information with EDA.
 I, personally believe in pragmatism and the "no-silver-bullet" approach so I can't say that EDA is always the best solution (As an aside, this is part of the reason I write my book as patterns not as "best-practices guidance"). You may find that ETL is the best trade off in your situation. Yes I know that it isn't a definitive answer - but real life is (usually) a little more complicated than black and white solutions. As architects we need to find the best trade off for the situation at hand.

I thought I has this  RESTful web services thing figured out, but following one of the threads on the Yahoo group on Service-Oriented-Architecture I came to the conclusion that maybe I don't.

Steve Jones tried to see if he understands REST by giving an example and that example was corrected by Anne Thomas Manes (who is a research director with the Burton Group which recently stated that the future of SOA is REST).
Here are the examples from the above mentioned thread:

POST http://example.org/customer
HTTP message body contains a representation of "anne"
server creates a subordinate resource called http://example.org/customer/anne

GET http://example.org/customer/anne
returns a representation of "anne"

GET http://example.org/customer/personByName?name=anne
returns a representation of "anne"
or perhaps returns the URI of the "anne" resource
or perhaps returns a list of URIs of all people named "anne"
might also be specified more simply as
GET http://example.org/customer?name=anne

GET http://example.org/customer/personByAge?age=27
returns a list of URIs of people whose age is 27
or perhaps returns a collection of representations of all people aged 27
might also be specified more simply as
GET http://example.org/customer?age=27

PUT http://example.org/customer/anne
HTTP message body contains a representation of "anne"
either creates a new resource called "anne" (if none exists)
or replaces the existing "anne" resource

PUT http://example.org/company/newco
HTTP message body contains a representation of "newco"
either creates a new resource called "newco" (if none exists)
or replaces the existing "newco" resource

If you prefer the server to assign the URI you would instead say

POST http://example.org/company
HTTP message body contains a representation of "newco"
server creates a subordinate resource called http://example.org/company/newco

POST http://example.org/customer/anne?addCompany=http://example.org/company/newco
this would append the newco company reference to the "anne" resource

You can see another example for what I am talking about here on Jon Udell's blog giving an example from RESTful Web Services, by Leonard Richardson and Sam Rubycovering  of doing a transaction in RESTful style

If all these are indeed "legal" or "correct" RESTful interactions I have 2 observations to make
First, I guess Pat Helland is right when he said "Every noun can be verbed" since I don't see the real difference between having a contract with a PersonsByAge request which returns a document* of Persons and a REST request like " GET http://example.org/customer/personByAge?age=27" or even " GET http://example.org/customer?age=27".

The second observation has to do with the so called "uniform interface". I would argue that the resources and their attributes (age=27, name="anne") are the interface. the POST, GET etc. uniform interface does not mean much more than the "uniform" SEND, BROADCAST  interface of messaging.
Further more if resources and their attributes are indeed "the interface" - than not only does REST not have a uniform contract - it actually has a dynamic one which changes in run-time as new resources are created - such as the "POST http://example.org/company"  which creates a new resource "http://example.org/company/newco" in the example above

---

* I think it is very important for SOA to have document oriented messages and not RPC one I\ll blog in a separate post about the differences. for now it is suffice to say that the REST hypermedia notion of returning the URIs of all the relevant persons should also be present (one way or another) in a good document oriented message even if you are using WS-* or plain messaging as transport

In addition to the drafts of selected patterns I publish on my site, you can now purchase my book via the Manning Early Access Program (MEAP).
MEAP means you can get chapter drafts as I write them and the complete book when its done (ebook or printed). Here is Manning's explanation:

"Buy now through MEAP (Manning Early Access Program) and get early access to the book, chapter by chapter, as soon as they become available. You choose the format - PDF or ThoutReader - or both. By subscribing to MEAP chapters, you get an opportunity to participate in the most sensitive, final piece of the publishing cycle by offering feedback to the author. Reader feedback to the author is welcome in the Author Online forum. As new chapters are released, announcements are made in the MEAP Announcement Forum. After all chapters are released, you will be able to download the complete edited ebook. If you order the print edition, we will ship it to you upon release, direct from the bindery, weeks before it is widely available elsewhere.

By the way, this is probably also a good time to mention that I'll be speaking about quite a few of the patterns in Architecture & Design World 2007 which will take place this July.

There is still a lot of work, but I already like to thank all the people in manning that helped me get this far. especially to Cynthia Kane my editor (hey, maybe now she'll give me more slack :) )
Ok, 'nuff blubbering, back to completing chapter 5...

Agile and documentation? what gives? First things first documentation is not something that is prohibited by Agile Manifesto. Working software is definitely preferred over "comprehensive documentation" but there can also be some value in documentation

The first question is why would we want to document anything if we have a working software. I think there are several stakeholders like project newcomers, maintainers etc. who will be interested in something that will let them get up to speed and provide them an overview of what's going on before they delve into code. You can read more on that in  a post I wrote almost a year a