Arnon Rotem-Gal-Oz's Cirrus Minor - Project Management

What is maintainability anyway?

Arnon Rotem-Gal-Oz — Sat, 01 Aug 2009 13:59:18 GMT

Reacting to a comment left by Frans Bauma, Ayende recently wrote about “Maintainability”

Maintainable is a value that can only be applied by someone who is familiar with the codebase. If that someone find it hard to work on the codebase, it is hard to maintain. If someone with no knowledge of a codebase find it hard to work with it, tough luck, but that doesn’t say anything about the maintainability of a code base.

I usually agree with what Ayende has to say, but not this time. First I hope that by “someone who is familiar with the codebase” he doesn’t refer to the person that actually wrote the code – since if the person who wrote the code can’t understand what he/she wrote than the code base is doomed anyway.

In the wider-sense “someone who is familiar with the codebase” is just part of the picture – a code base is only maintainable is a reasonably professional developer can get to a point where she is familiar enough with the code to be able to maintain it. This doesn’t imply that the time it takes to be productive with the code base is zero – but the lower the time it takes to get up to speed means the more maintainable is the code.

In any event, for a codebase to be maintainable, it has to show several quality attributes .For the most part I agree withthe definition of Maintainability in ISO 9126:2001 Software Engineering Product Quality*

6.5 Maintainability
The capability of the software product to be modified. Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications.

6.5.1 Analysability - The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified.

6.5.2 Changeability - The capability of the software product to enable a specified modification to be implemented.
NOTE 1 Implementation includes coding, designing and documenting changes.
NOTE 2 If the software is to be modified by the end user, changeability may affect operability.

6.5.3 Stability - The capability of the software product to avoid unexpected effects from modifications of the software

6.5.4 Testability - The capability of the software product to enable modified software to be validated.

6.5.5 Maintainability compliance -The capability of the software product to adhere to standards or conventions relating to maintainability.

Naturally, being a standard it has the “compliance” thingy which is usually only relevant for large organizations and project but for the most part the different aspects mentioned above are the parts you need to take care of when you want someone besides yourself to make changes to the software.

The view of Maintainability Ayende uses is problematic esp. when we consider that (successful) software will spend most its life in maintenance and not in development (you can read Robert L. Glass’s excellent “Software Maintenance is a Solution, Not a Problem” paper in this regard). Assuming someone maintaining the code will always be familiar with it is expecting the same developer(s) to stay at the same project for as long as the project will live (which is not likely) and/or assuming the project will have a short life (not something I’d want from my projects)

So don’t forget that other people will have to maintain your code and they probably won’t live the code-base as you do or as they say in “Code for the maintainer” in the C2 wiki

“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live. “ :)

* ISO 9126 is a multi-part standard for QA. I think ISO9126:2001 is good quick reference for quality attributes ( i.e. something you can look at when you try to elicit quality attributes for an architecture). I, personally think the other parts of the standard are pretty useless but that's another story :)

Continuous Deployment

Arnon Rotem-Gal-Oz — Wed, 18 Mar 2009 19:35:26 GMT

Image via Wikipedia

You might have noticed my blogging is slowing down a little, in case you're wondering the culprit is xsights - As we're getting closer to
production (2-3 weeks if all goes well) everything else slows down...

We all know about continuous integration (CI). We (xsights) are seeing a lot of ROI on the investments we made into signing up to CI esp. when we augmented by automated testing. Everyday we can and do actually release several versions and run load tests on them on different server seeking the best release to "freeze". Where "freezing" means the system that will be used in production. That's the way it is done, right? You work on a release for x months (sometimes years..) and after an integration period at the client's site you go live and forget about them (save for show-stopping bugs) until the next major update several months later. For instance, setting up our "methodology" I figured a 6 months release cycles for major versions plus 1 month releases of minor version would suffice.

These days, however the more I think about it, The more I believe this approach is not the right one for our situation. We are going to go with Continous Deployment. There are many reasons for that primarily:

The requirements stream (new reqs and changes) is still pouring in - and in increased rate.
We are building a service platform not an installable product - This means we get to control where how it is run so there's less overhead in
streamlining updates
It would be our first public installation - were going to roll out bug fixes anyway (I personally never write bugs... but you know ;) ). We'll need a proper procedure for updates anyway
We are doing this anyway - every new demo/internal run we have is using the "stablest" latest and greatest :)

What does Continuous Deployment means? It means we'll be rolling out new production versions on a daily basis (or even more frequently!)
How that's going to work? Well, First there are the prerequisites:

Version control - something that can track and tag versions...
Continous integration which is set up and running. You cannot release continously if every new development offer breaks the system. An automated smoke test is
Automates tests that provide a good approximation of real usage
(.e.g. usage patterns, loads etc.) - You need to be able to have a high
confidence level that whatever you have.
Staging environment - An environment as close as possible to the production system (just like you'd have for a web-site)
Monitoring and alert system - You need to be able to identify problems quickly - the running

Then there's the process:

identify a baseline Stable release -The "fallback".
pick the most promising build of the day
run it through more extensive tests (at least burn-down and load - which means a release maybe 1-2 days old by the time it hits the production system)
if everyhting is cool deploy to staging - else fix bugs and repeat from 2
repeat tests in staging environment
Update system (This is a point that still needs work so we can support a version hot-swap i.e. without a shutdown)..
If the monitoring system alerts unforeseen bugs (i.e. problems with the release) - roll back to stable release, try to add test for the problematic behavior and repeat from 2
if the release worked good enough - mark it as the next Stable release

That's the plan. However as we all know, talk is cheap so be sure to check in here in 2-3 months to hear how did we fare :)

The baker's dozen - my best posts for 2008

Arnon Rotem-Gal-Oz — Sat, 20 Dec 2008 22:10:06 GMT

The year is almost done so I'd thought it would be a good time for a short retrospective into what I blogged here. The 13 posts below are the ones I liked best this year. Turns out these posts touch on a lot of different subjects: requirement, software management, agile development, architecture, SOA and programming.

Broken build white noise

Arnon Rotem-Gal-Oz — Sun, 16 Nov 2008 22:52:24 GMT

One important (yes, yes important) aspect of continuous integration (CI) is the way broken builds are reported. I'll tell you why but before that let's quickly recap on CI in general.
The point is to get an automated build process that run's after every check-in and takes the system for a quick spin to verify all the major parts are still OK. CI helps maintain the quality of the overall code by ensuring that components don't drift too much apart and that the current version is still valid to a reasonable degree.

Martin Fowler's article on CI (referenced above) lists the following practices

Maintain a Single Source Repository.
Automate the Build
Make Your Build Self-Testing
Everyone Commits Every Day
Every Commit Should Build the Mainline on an Integration Machine
Keep the Build Fast
Test in a Clone of the Production Environment
Make it Easy for Anyone to Get the Latest Executable
Everyone can see what's happening
Automate Deployment

What I am saying that a practice missing from this list is to maintain a clear concise broken build report. You see, whether we like it or not builds break. Not every checking (well hopefully anyway:) and not every day but that isn't a scarce sight either.
In our setup we get an email every time the build breaks. That's a good thing - since it increases the visibility of the problem. The message tells us who broke the build and what's the error (something doesn't compile, unit test broken or smoke test broken) and which file.

The problem is that is (was) that up until recently these details were unreliable for instance

The error message was so long nobody bothered to read where the problem is
Once a build was broken every new checkin also got the broken build notice - this time pointing the new person (doing the checking ) as the problem's source (even though the previous person was to blame"
etc.

The problem here is that unreliable reports quickly turn into white noise nobody pays attention to (someone at the team said it was like the boy who cried wolf -by the time it is really "your" problem you are no longer listening. When nobody pays attention, problems get to age - in the long run this can also invalidate the whole CI approach - somewhat like the broken windows theory. So for no broken builds the team has to pay attention to what's wrong. The way to do that is to invest in the build script to make it more easy to understand and as reliable as possible - this is something we did (and still working on). I know I don't want to take the risk of broken build white noise...

Evolving Architectures - Architecture Retrospective

Arnon Rotem-Gal-Oz — Tue, 12 Aug 2008 21:40:21 GMT

Retrospectives, every "agile" team does retrospectives.What are retrospectives anyway?

A retrospective is a meeting where the team takes a look and inspect the past, in order to adapt and improve the future.

Agile or not, our team does a retrospective at the end of each iteration (every two weeks in our case). We try to look at what worked, what didn't , how we are meeting our goals etc, how is the product going etc.. These meetings provide a lot of value for steering us at the right direction.
On going retrospectives that look at the near past allows for suppleness and change adaptation and they are very powerful at that - However it is sometimes worthwhile to reflect over longer periods of time.

One area where longer perspective is important is the architecture of the project. Evolving an architecture you run the risk of accepting wrong decisions - mostly because architectural decisions have long term implications, while YAGNI, time constraints and life in general drive you toward short term gains.

Again, taking an example from my current project, working towards the first release, we took a few major decisions during the development e.g.

federated resource management - Taking into consideration the fallacies of distributed computing we decided that we'd have local resource managers that will take care of resource utilization and allocation. The resource managers will have a hierarchy where they'd communicate with each other to gain the "bigger picture"
Introduce Parallel Pipelines - handle image understanding by dividing the work between specialized components.
RESTful control channel - to use a "lingua franca" between all component types so that we can easily integrate across platforms and languages
local failure handling - resources and components handle failure by themselves
Communication technology (WCF in our case) is isolated from the business logic by an Edge Component
etc.

Once we finished delivering the first release. We took a few "days off" to consider what we've done thus far. updated our quality attribute list per our knowledge working with the system and looking at some customer scenarios. studies the things we liked/didn't like in the design and architecture of the working system. and revised a few of our decisions for instance

We found that rushing to a working system we introduced some excess coupling to a specific technological solution (for video rendering). We initiated a few proof of concepts and found out how to both isolate the technology from the rest of the system as well as allow more technology choices.
We found that the some of the data flows were not as clean as we thought they'd be - adding new features caused more resource interactions than we thought when we partitioned the resources. We redefined some of the resource roles to get less message clutter (and higher cohesion)
The federated resource management works well, but introduce needless latency in session initiation. We now opted for introduce "Active services" which are more autonomous.
Add a blogjecting Watchdog in addition to local failure handling to both increase the chances of failure identification and recovery as well as get a better picture in a centralized Service Monitor.
RESTful control channel worked well and will continue for later release
Some of the scale issues will be handled by introducing "Virtual Endpoints" while some would continue to use autonoumous endpoint creation and liveliness dissemination (hopefully learning from the mistakes of others)
etc.

The result of these and the other decisions we've maid is a rework plan that will (hopefully anyway) make our overall solution better.
What we see is that we evolved our architecture as we went forward. While all the the decisions we made seemed right at the time we took them, only through reviewing them in a wider perspective (architecture retrospective) we identified the decisions that we need to change and the ones that we have to enhance. The insight you gain after working on a project for awhile are much better than the initial thoughts you have or the understanding you master in the initial interations.
I think it is essential to review the architecture once you've gained more experience with the realities of the system you write (vs. the precieved realities you have on the get go)

By the way if you work with a waterfall approach your situation is worse. Since in this case you take your decisions before you write any code so, you don't even have the benefit of POCs, and working code to enhance your insights

PS
if you have the MEAP version of SOA Patterns you can read more on the patterns I've mentioned here: Active service in chapter 2, blogjecting watchdog in chapter3, Service Monitor in chapter 4, Parallel Pipelines in chapter 3, Edge Component in chapter 2

Funding projects incrementally

Arnon Rotem-Gal-Oz — Sun, 03 Aug 2008 21:44:38 GMT

Johanna writes about how you should fund project in an incremental manner especially when it comes to the most risky ones:

incrementally, in some sort of a timebox. (Gotcha!) But here’s why it make sense to do that:

If you show value to someone, preferably your customer, you are much more likely to get more funding.

If you’re not showing value early and often, you get feedback early enough to change before you start a death march for something your customers don’t want.

You can start highly risky projects, because you’re not committing a ton of money and time to that much risk. You’re just committing 2, 3, or 4 weeks.

I have to say I agree 100%. One nice anecdote in this regard, happened to me a few years ago. I was working for a rather bureaucratic, waterfallish organization. I wanted to outsource part of a project I was running and I found an agile sub-contractor for the job. I somehow managed to convince both my management and the methods and practices group that this project should be using SCRUM (the only "cost" for me was that I had to write a Software Development Plan , proving that SCRUM can be ISO 90003 compliant, but that's another story).
We set up a rough plan (release plan) with timeboxed releases every three months (every 3 iterations) and we were all set. Almost, that is - when I approached the procurement group they told me to hold my horses. They were not willing to go through the procurment process every three month. They also weren't very keen on the cost plus nature of the plan (the deliverables weren't prefixed as we were going to evolve the solution). To went over this hurdle I had to prepare a pseudo plan were they had "real" mile stones (basically based on the release planning mentioned above). We ended up budgeting the whole thing in advance, however I did convince them to establish sort of a blanket purchase order, so that we were able to release funds incrementally as we progressed.

Requirements Tracability in Mingle

Arnon Rotem-Gal-Oz — Mon, 02 Jun 2008 21:17:32 GMT

While I am on the subject of Project management, One nice feature of Mingle (I am not sure if it is new in v.2 or it was there in v1) is requirements traceability.
Requirements traceability is a requirements management discipline which makes it possible to attribute artifacts (design/code etc) to the requirements that originated them. It also provides a way to mange changes (if you know which artifacts were created to fulfill a requirement you also know the artifacts that are likely to change when the requirement change).

However, above all that, Requirements traceability is a pain in the ass* ! Which means it is hardly ever done. Especially in agile projects (working code over extensive documentation, remeber?)

Anyway, as I said, a nice feature of mingle is that, if you set it to work with your subversion repository (I created a user for mingle) and when you check in code you add the card number to the check-in comments (e.g. #123 - fixed this and that bug). Mingle will let you navigate that relation. i.e. in the History -> revisions you can see the card number and click through to the card it-self. If you also do the same from tasks to stories you get a good traceability with very little effort

And I can tell you that from personal experience having once volunteered(!) to do the whole traceability matrix for a 1000+ man/year project from the analysis to the customer requirements ( it took more than 2 weeks in case you are wondering)

Make technical debt explicit

Arnon Rotem-Gal-Oz — Mon, 02 Jun 2008 20:56:50 GMT

When you are working towards a specific goal esp. if it is a looming mile stone or something to that effect, you tend to leave stuff behind, cut some corners, just a wee bit of slack... This "stuff" has a nice metaphor to describe it. It is called "Technical Debt" .

If you don't mind your technical debt it can grow so much you'd end up with a big-ball-of-mud which is hard to maintain, change and, well, do anything useful with.

The best way, I've found to deal with this is to "add a card" every time I feel the urge to add a //TODO comment. Or in other words add the technical debt as a task into the product/task backlog.
Having the technical debt on the backlog has several benefits such as

It will not be forgotten - it will be documented...
It will not be hidden - The true state of the product will be in the open for management/product owner to see. As a manager I want to know the true state of the product. If I know what I can and can't have I can get ready for that. If I think everything is rosy and then the system blows up in my face, that's not so good..
It will be managed - The importance/relevance of the "debt" will be reevaluated every time the product backlog get prioritized.

Technical debt will occur in your project, whether it is agile, "water-falled" , incremental or what not. Don't ignore it

Sticky notes vs. the computer

Arnon Rotem-Gal-Oz — Sun, 30 Mar 2008 22:07:28 GMT

I never really understood would I want to use sticky notes/cards in my agile projects rather than use a software tools. After all we write software for others - how can we say that software is not a good enough solution for us ?
For instance, in our team we use Mingle by Thoughtworks. I think it is one of the best tools for project management I've ever used. Its serves us well for sprint planning, daily scrums etc. Among other things it gives you "digital" sticky notes which you can drag around to move between statuses or whatnot - just perfect.

And yet, last week when we were pushing for completing our first major milestone I decided to try real sticky notes, and finally I understood the difference - constant visibility. Computerized tasks seem to be just as accessible as physical cards but in fact they aren't. Yes, everyone can see the status whenever they want. But the fact that you can doesn't mean that you do. Digital cards don't have the "in-your-face" kind of effect that always visible notes has. I've seen a notable difference in maintaining the dev team's focus and making managers (the CEO in my case) feel they are in the loop and that progress is constantly made.
My conclusion is clear. While I still use electronic tools, I guess, until we'd have large enough e-ink boards to allow us to get the same effect physicals cards give us I'll just have to keep restocking my sticky notes inventory :)

Use Cases and User Stories

Arnon Rotem-Gal-Oz — Wed, 14 Nov 2007 00:15:55 GMT

Jeremy D. Miller writes that he prefers user stories better than use cases - and I basically agree with him. Mostly because user stories are finer grained which lends to make them more "estimateable" and manageable then use cases are.

However, having used both, I think there's one thing use cases do better then user stories which is eliciting related scenarios. When you write a use case that covers a business scenario the use case template makes you think about variations and exceptions. When you try to think of a theme of user stories if you're not careful to do that in a structured manner you may miss some.

One important point regarding use cases is that you don't have to treat the use case a single monolithic block. When you work with use cases it doesn't have to mean that you sign-off use cases in a waterfall-ish manner- that is something that has to do with your development methodology.
You can develop use cases incrementally i.e. only do some of the use cases per "release". You can also elaborate use cases iteratively by only elaborating/identifying some of the scenarios in each iteration (in a manner similar to user stories). I have successfully use both incremental and iterative use case elaborations on several projects and I find it very useful (I even wrote paper summarizing my use cases experience a few years ago (warning - more than 60 pages with the examples and all..)

In fact, even today I find it useful to use both tools where I would usually start with identifying a user story and writing that down. Then map it to a use case (either existing or a new one) - where a use case roughly equals a user story theme. As I mentioned previously I can them identify more user stories and add them to the product backlog. The use case stays a skeleton with links to the user stories and helps provide more context to individual use stories (ok, so that isn't a use case in the traditional sense, but I find it useful anyway).

Generally speaking, it is important to note that while it is tempting to equate a user story with a scenario in a use case, that isn't always true. sometime a user story would be a step in a use case and sometimes a story is shared by several use cases - since a fine grained feature that delivers business value is can be used in many business processes which is what use cases are oriented to

As a side note, if you have a project where you do have to write all the requirements up-front (as if they won't change anyway...) then I find that use cases are way superior than IEEE 830 style requirements ("The system shall...") - but hey, that's another story :)

Fixed Bids and Agile Projects

Arnon Rotem-Gal-Oz — Sun, 02 Sep 2007 21:02:10 GMT

Jeremy Miller recently tried to "answer hard questions about Agile development". One question he didn't really answer was that of agile and fixed bids :

"I don't really think the Agile answer to a fixed bid is any different from any other process. I do think that Agile practices and project management can give you far more control and feedback on the "Iron Triangle" of resources, time, and features"

Jeremy says that this would enable the team to fail "softly" (i.e with some functionality etc.).
I think (well, actually, I know since I did that) we can do better than just use Agile and hope for the best. For instance, here are a few strategies I successfully used:

1. Pre-budget the whole project with rough milestones and estimates (works best if you have prior experience in the domain)
2. Coordinate expectations with the client and trade - the single project to many smaller ones. What I did was to get a blanket purchase order for the estimated amount and each smaller order/project "ate" some of that budget until it was done.
3. Since the project was for a CMMI level 3/ ISO 90003 we also added a few documents to the backlog - and let the product owner prioritize them vs. deliverables. For instance the "Architecture document" was delivered after 6 iterations when the architecture stabilized
4 exchange requests instead of change request - new functionality comes instead of old promised one

Of course we also did the other SCRUM practices of releasing frequently, demonstrating progress, retrospective etc.

I would love to take credit for the ideas above - however, Pascal Van Cauwenberghe details most of them (and more) in two papers called "Agile Fixed Price Projects part 1: The Price is Right" and "Agile Fixed Price Projects part 2: Do you want agility with that" (go read them..)