Retrospectives, every "agile" team does retrospectives.What are retrospectives anyway?

A retrospective is a meeting where the team takes a look and inspect the past, in order to adapt and improve the future.

Agile or not, our team does a retrospective at the end of each iteration (every two weeks in our case). We try to look at what worked, what didn't , how we are meeting our goals etc, how is the product going etc.. These meetings provide a lot of value for steering us at the right direction.
On going retrospectives that look at the near past allows for suppleness and change adaptation and they are very powerful at that - However it is sometimes worthwhile to reflect over longer periods of time.

One area where longer perspective is important is the architecture of the project. Evolving an architecture you run the risk of accepting wrong decisions - mostly because architectural decisions have long term implications, while YAGNI, time constraints and life in general drive you toward short term gains.

Again, taking an example from my current project, working towards the first release, we took a few major decisions during the development e.g.

• federated resource management - Taking into consideration the fallacies of distributed computing we decided that we'd have local resource managers that will take care of resource utilization and allocation. The resource managers will have a hierarchy where they'd communicate with each other to gain the "bigger picture"
• Introduce Parallel Pipelines - handle image understanding by dividing the work between specialized components.
  
• RESTful control channel - to use a "lingua franca" between all component types so that we can easily integrate across platforms and languages
  
• local failure handling - resources and components handle failure by themselves
• Communication technology (WCF in our case) is isolated from the business logic by an Edge Component
  
• etc.

Once we finished delivering the first release. We took a few "days off" to consider what we've done thus far. updated our quality attribute list per our knowledge working with the system and looking at some customer scenarios. studies the things we liked/didn't like in the design and architecture of the working system. and revised a few of our decisions for instance

• We found that rushing to a working system we introduced some excess coupling to a specific technological solution (for video rendering). We initiated a few proof of concepts and found out how to both isolate the technology from the rest of the system as well as allow more technology choices.
• We found that the some of the data flows were not as clean as we thought they'd be - adding new features caused more resource interactions than we thought when we partitioned the resources. We redefined some of the resource roles to get less message clutter (and higher cohesion)
  
• The federated resource management works well, but introduce needless latency in session initiation. We now opted for introduce "Active services" which are more autonomous.
• Add a blogjecting Watchdog in addition to local failure handling to both increase the chances of failure identification and recovery as well as get a better picture in a centralized Service Monitor.
• RESTful control channel worked well and will continue for later release
• Some of the scale issues will be handled by introducing "Virtual Endpoints" while some would continue to use autonoumous endpoint creation and liveliness dissemination (hopefully learning from the mistakes of others)
• etc.
  

The result of these and the other decisions we've maid is a rework plan that will (hopefully anyway) make our overall solution better.
What we see is that we evolved our architecture as we went forward. While all the the decisions we made seemed right at the time we took them, only through reviewing them in a wider perspective (architecture retrospective) we identified the decisions that we need to change and the ones that we have to enhance. The insight you gain after working on a project for awhile are much better than the initial thoughts you have or the understanding you master in the initial interations.
I think it is essential to review the architecture once you've gained more experience with the realities of the system you write (vs. the precieved realities you have on the get go)

By the way if you work with a waterfall approach your situation is worse. Since in this case you take your decisions before you write any code so, you don't even have the benefit of POCs, and working code to enhance your insights


PS
if you have the MEAP version of SOA Patterns you can read more on the patterns I've mentioned here: Active service in chapter 2, blogjecting watchdog in chapter3, Service Monitor in chapter 4, Parallel Pipelines in chapter 3, Edge Component in chapter 2


 

Johanna writes about how you should fund project in an incremental manner especially when it comes to the most risky ones:

incrementally, in some sort of a timebox. (Gotcha!) But here’s why it make sense to do that:

• If you show value to someone, preferably your customer, you are much more likely to get more funding.
• If you’re not showing value early and often, you get feedback early enough to change before you start a death march for something your customers don’t want.
• You can start highly risky projects, because you’re not committing a ton of money and time to that much risk. You’re just committing 2, 3, or 4 weeks.

I have to say I agree 100%. One nice anecdote in this regard, happened to me a few years ago. I was working for a rather bureaucratic, waterfallish organization. I wanted to outsource part of a project I was running and I found an agile sub-contractor for the job. I somehow managed to convince both my management and the methods and practices group that this project should be using SCRUM (the only "cost" for me was that I had to write a Software Development Plan , proving that SCRUM can be ISO 90003 compliant, but that's another story).
We set up a rough plan (release plan) with timeboxed releases every three months (every 3 iterations) and we were all set. Almost, that is - when I approached the procurement group they told me to hold my horses. They were not willing to go through the procurment process every three month. They also weren't very keen on the cost plus nature of the plan (the deliverables weren't prefixed as we were going to evolve the solution). To went over this hurdle I had to prepare a pseudo plan were they had "real" mile stones (basically based on the release planning mentioned above). We ended up budgeting the whole thing in advance, however I did convince them to establish sort of a blanket purchase order, so that we were able to release funds incrementally as we progressed.

 

While I am on the subject of Project management, One nice feature of Mingle (I am not sure if it is new in v.2 or it was there in v1) is requirements traceability.
Requirements traceability is a requirements management discipline which makes it possible to attribute artifacts (design/code etc) to the requirements that originated them. It also provides a way to mange changes (if you know which artifacts were created to fulfill a requirement you also know the artifacts that are likely to change when the requirement change).

However, above all that, Requirements traceability is a pain in the ass* ! Which means it is hardly ever done. Especially in agile projects (working code over extensive documentation, remeber?)

Anyway, as I said, a nice feature of mingle is that, if you set it to work with your subversion repository (I created a user for mingle) and when you check in code you add the card number to the check-in comments (e.g. #123 - fixed this and that bug). Mingle will let you navigate that relation. i.e. in the History -> revisions you can see the card number and click through to the card it-self. If you also do the same from tasks to stories you get a good traceability with very little effort

---

And I can tell you that from personal experience having once volunteered(!) to do the whole traceability matrix for a 1000+ man/year project from the analysis to the customer requirements ( it took more than 2 weeks in case you are wondering)
 

When you are working towards a specific goal esp. if it is a looming mile stone or something to that effect, you tend to leave stuff behind, cut some corners, just a wee bit of slack... This "stuff" has a nice metaphor to describe it. It is called "Technical Debt" .

If you don't mind your technical debt it can grow so much you'd end up with a big-ball-of-mud which is hard to maintain, change and, well, do anything useful with.

The best way, I've found to deal with this is to "add a card" every time I feel the urge to add a //TODO comment. Or in other words add the technical debt as a task into the product/task backlog.
Having the technical debt on the backlog has several benefits such as

• It will not be forgotten - it will be documented...
  
• It will not be hidden - The true state of the product will be in the open for management/product owner to see. As a manager I want to know the true state of the product. If I know what I can and can't have I can get ready for that. If I think everything is rosy and then the system blows up in my face, that's not so good..
• It will be managed - The importance/relevance of the "debt" will be reevaluated every time the product backlog get prioritized.
  

Technical debt will occur in your project, whether it is agile, "water-falled" , incremental or what not. Don't ignore it



 

I never really understood would I want to use sticky notes/cards  in my agile projects rather than use a software tools. After all we write software for others - how can we say that software is not a good enough solution for us ?
For instance, in our team we use Mingle by Thoughtworks. I think it is one of the best tools for project management I've ever used.  Its serves us well for sprint planning, daily scrums etc. Among other things it gives you "digital" sticky notes which you can drag around to move between statuses or whatnot - just perfect.

And yet, last week when we were pushing for completing our first major milestone I decided to try real sticky notes, and finally I understood the difference - constant visibility. Computerized tasks seem to be just as accessible as physical cards but in fact they aren't. Yes, everyone can see the status whenever they want. But the fact that you can doesn't mean that you do. Digital cards don't have the "in-your-face" kind of effect that always visible notes has. I've seen a notable  difference in maintaining the dev team's focus and making managers (the CEO in my case) feel they are in the loop and that progress is constantly made.
My conclusion is clear. While I still use electronic tools, I guess, until we'd have large enough e-ink boards to allow us to get the same effect physicals cards give us I'll just have to keep restocking my sticky notes inventory :)

 

Jeremy D. Miller writes that he prefers user stories better than use cases - and I basically agree with him. Mostly because user stories are finer grained which lends to make them more "estimateable"  and manageable then use cases are.

However, having used both, I think there's one thing use cases do better then user stories which is eliciting related scenarios. When you write a use case that covers a  business scenario the use case template makes you think about variations and exceptions. When you try to think of a theme of user stories if you're not careful to do that in a structured manner you may miss some.

One important point regarding use cases is that you don't have to treat the use case a single monolithic block. When you work with use cases it doesn't have to mean that you sign-off use cases in a waterfall-ish manner- that is something that has to do with your development methodology.
You can develop use cases incrementally i.e. only do some of the use cases per "release". You can also elaborate use cases   iteratively by only elaborating/identifying some of the scenarios in each iteration (in a manner similar to user stories). I have successfully use both incremental and iterative use case elaborations  on several projects and I find it very useful (I even wrote paper summarizing my use cases experience a few years ago (warning - more than 60 pages with the examples and all..)

In fact, even today I find it useful to use both tools where I would usually start with identifying a user story and writing that down. Then map it to a use case (either existing  or a new one) - where a use case roughly equals a user story theme. As I mentioned previously I can them identify more user stories and add them to the product backlog. The use case stays a skeleton with links to the user stories and helps provide more context to individual use stories (ok, so that isn't a use case in the traditional sense, but I find it useful anyway).

Generally speaking, it is important to note that while it is tempting to equate a user story with a scenario in a use case, that isn't always true. sometime a user story would be a step in a use case and sometimes a story is shared by several use cases - since a fine grained feature that delivers business value is can be used in many business processes which is what use cases are oriented to


As a side note, if you have a project where you do have to write all the requirements up-front (as if they won't change anyway...) then I find that use cases are way superior than IEEE 830 style requirements ("The system shall...")  - but hey, that's another story :)

 

Jeremy Miller recently tried to "answer hard questions about Agile development". One question he didn't really answer was that of agile and fixed bids :

"I don't really think the Agile answer to a fixed bid is any different from any other process.  I do think that Agile practices and project management can give you far more control and feedback on the "Iron Triangle" of resources, time, and features"

Jeremy says that this would enable the team to fail "softly" (i.e with some functionality etc.).
I think (well, actually, I know since I did that) we can do better than just use Agile and hope for the best. For instance, here are a few  strategies I successfully used:

1. Pre-budget the whole project with rough milestones and estimates (works best if you have prior experience in the domain)
2. Coordinate expectations with the client and trade - the single project to many smaller ones. What I did was to get a blanket purchase order for the estimated amount and each smaller order/project  "ate" some of that budget until it was done.
3. Since the project was for a CMMI level 3/ ISO 90003 we also added a few documents to the backlog - and let the product owner prioritize them vs. deliverables. For instance the "Architecture document" was delivered after 6 iterations when the architecture stabilized
4 exchange requests instead of change request - new functionality comes instead of old promised one

Of course we also did the other SCRUM practices of releasing frequently, demonstrating progress, retrospective etc.

I would love to take credit for the ideas above - however, Pascal Van Cauwenberghe details most of them (and more) in two papers called "Agile Fixed Price Projects part 1: The Price is Right" and "Agile Fixed Price Projects part 2: Do you want agility with that" (go read them..)