March 23, 2010
@ 10:29 PM

A few days ago I was contacted by Lianping Chen a doctoral researcher from the Irish Software Engineering Research Centre. Lianping is doing research on “how to elicit architectural significant requirements” and he asked me a few questions, which I though, might be interesting to a wider audience.


1. Do you agree that architecture design and requirements elicitation are usually in parallel or have a big time overlap? In other words, Architectural design usually starts before requirements elicitation ends, and usually a big time overlap exists between these two activities.

While it seems to me Lianping is thinking mostly about large waterfall (or waterfall-ish) projects. I’d say this is true for all kinds of projects (agile, lean, iterative and waterfall). The tradeoff usually is waiting for some unknown point in time where the requirements will be set, known, approved and whatnot and starting to move forward. Architecture usually needs to start rolling when the data (in the sense of what exactly the architectural requirements are) is incomplete. This is exactly why I think Architecture should evolve over time (see the previous two posts on “Evolving Architectures” – part I, part II).

In my experience, in large projects, it is beneficial for the architects to be part of the requirements elicitation team. It is true some architectural requirements can be derived from “pure” functional requirements. However most of the architectural requirement can (and in my opinion, should) be formed  by a deliberate effort. I personally, like the scenario based approach of  creating a utility tree. Note that this does not contradict the point above, which says that some of these requirements will probably be off of the real requirements which will surface during the project.


2. Do you agree that when perform architecture design, making some decisions depends on the outcome of some other decisions? I other words, a sequence/order exists for the set of architectural decisions need to be made.

Yes some architectural decisions depends on others. For example, technological choices can change other architectural decisions – after all technologies usually come with their own set of architectural decisions made by the people that created them. On the other hand, it is important to understand that some decisions can progress or be made  in parallel as well. For example, UI architecture design can progress and form while the distribution architecture is still being debated.


3. Do you agree that, in most cases, for a particular architectural decision, only a mall portion of the whole requirements actually influences the decision making? In other words, there is mapping exist from the architectural decisions you need to make and the requirements required when you make those decisions.

Right, most of the functional requirements drive design and the implementations. Only a relatively small part of the requirements drive the architecture.


4. Do you agree that requirements engineers usually do not know clearly what items/aspects of requirements are architectural significant (i.e., it is not easy to distinguish architectural significant requirements from normal non architectural significant requirements)? Thus, they may ignore/miss some items of requirements (some of them may just look like trivial details) that are actually architectural significant during their requirements elicitation.

Yes, this is natural. “Requirement engineers” (or product owners in agile projects) are mostly concerned with the business aspects of the system  - and rightfully so. It is important to have architects involved in analyzing requirements. Also, as I mentioned in a previous answer, it is even more important to have architects work with the different stakeholders (req engineers included) to specifically elicit architectural requirements. Sessions dedicated to quality attributes and forming them as scenarios in the system can help bridge this gap (the scenarios provide the connection to the functional reqs, and architectures are build around quality attributes)


5. Do you agree that requirements engineers usually are not aware of which items/aspects of requirements are required urgently by architects and which items/aspects of requirements can be scheduled a bit later to elicit? Thus, they may first elicit the requirements required for making the decisions in the tail of the decision making sequence, and schedule the elicitation of requirements required for the architectural decisions in the front of the decision sequence to the end of the requirement elicitation phase.

I am not even sure architects are fully aware which items and requirements are important :)  . In any event, as mentioned above,there are basically two types of architectural requirements. The first kind is requirements that can be elicited by direct, intentional analysis of the system from quality attributes perspective (thinking about scenarios where security, availability, scalability etc. come into play). For example in my current system (xsights) one requirement under extensibility – effort to change (data) we have the  scenario “Under normal conditions, refreshing the system’s data (links, interactions etc.) shall not require a system restart.”

The other type of architectural requirements  are derived from functional requirements i.e.  specific functional requirements whose implementation can have significant implications on the system. For example in a previous version of our system we had the requirement to handle 3G video call. Components that participate in this need to be stateful (as there is constant streaming of video in and out)

The conclusions are that, again, architects needs to be involved in the requirement gathering effort; architects need to lead specific session(s) that involve eliciting requirements based on quality attributes analysis. Architectures need to evolve over time as requirements significant to the architecture may (and a lot of times do) pop up during the development effort


6. Do you agree the following statement: If the requirements engineers are informed of what items/aspects of requirements are required and when they are required by the architects for making architectural decisions, the requirements engineers will be able to properly schedule their requirements elicitation activities and elicit the required requirements in enough detail and precision. So they will have higher chance to provide the required requirements to the architects before the architects make those architectural decisions.

Yes to some extent, but architects involvement as mentioned above, would probably yield better results (in my opinion of course)


7. What are the main issues in eliciting architectural significant requirements? What researchers can do to solve these issues

I think I already answered that – but if not feel free to ask for clarifications


 
Tags: Agile | Q&A | Software Architecture | SPAMMED Process

I’m writing a short series of posts for MS Israel MCS blog (in Hebrew) and I’d thought I’d translate them to English, as it seems to me they are interesting enough.

In this series I am going to talk about Evolutionary Architecture or , some of the aspect of dealing with software architecture in agile projects. The topic is interesting since architecture and agile seems to have some conflicting forces at work to better understand that let’s start by defining software architecture

There are many definitions for software architecture, with the simplest one (attributed, I think, to Kent Beck) that software architecture is what software architects does. Leaving the fact that (unfortunately) sometimes software architects are very far from building software architectures, the definition doesn’t tell us much. There are many definitions around some are good and some are bad. My current definition is

“Software architecture is the collection of decisions affecting the system’s quality attributes; which have global effects and are hardest to change. Software architecture provides the frame within which the design (code) is built.”

Let’s review the components of this definition

  • “affecting the system’s quality attributes” – I’ve written a lot about quality attributes in the past. In a nut shell, quality attributes (often going by the name “non-functional reqs.”) includes aspects of the system like scalability, security, availability etc. Architectural decisions have a direct effect on the system’s ability to meet these types of goal;
  • “Global effects” – Design decisions effect the module or the class where they happen. Decisions with macro effect (e.g. choosing a technology, scaling approach) can completely alter a solution
  • “Hardest to change” – The most interesting part of the definition, at least in regard to evolutionary architecture.The definition mentions that the code is built within the frame and rules set by the architectural decisions. Change in these decisions can have significant consequences. As a (over simplified) example - You can’t take a standalone system developed in Access and move it to a service oriented, Hadoop based solution with out major changes in the code, data flows and what not.

The definition of Software Architecture above, seems to prescribe that for best results we need to do a lot of up front design to get the architecture right. If that’s true than we have a severe mismatch with agile and/or lean where handling requirement and design up-front is a big no-no (YAGNI- you ain’t gonna need it, comes to mind) – Is there any way to make them work together.

I think yes, and as you’ve probably guessed, the answer is evolving the architecture over time. While this may sound simple it isn’t – I’ll try to give a few strategies to make that work later in the series. Before that, the next part, we’ll examine why design can be emergent which architectures need to evolve


 
Tags: Agile | Software Architecture

August 1, 2009
@ 02:59 PM

Reacting to a comment left by Frans Bauma, Ayende recently wrote about “Maintainability”

Maintainable is a value that can only be applied by someone who is familiar with the codebase. If that someone find it hard to work on the codebase, it is hard to maintain. If someone with no knowledge of a codebase find it hard to work with it, tough luck, but that doesn’t say anything about the maintainability of a code base.

I usually agree with what Ayende has to say, but not this time. First I hope that by “someone who is familiar with the codebase” he doesn’t refer to the person that actually wrote the code – since if the person who wrote the code can’t understand what he/she wrote than the code base is doomed anyway.

In the wider-sense “someone who is familiar with the codebase” is just part of the picture – a code base is only maintainable is a reasonably professional developer can get to a point where she is familiar enough with the code to be able to maintain it. This doesn’t imply that the time it takes to be productive with the code base is zero – but the lower the time it takes to get up to speed means the more maintainable is the code.

In any event, for a codebase to be maintainable, it has to show several quality attributes .For the most part I agree withthe definition of Maintainability in ISO 9126:2001 Software Engineering Product Quality*

6.5 Maintainability
The capability of the software product to be modified.  Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications.

  • 6.5.1 Analysability - The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified.
  • 6.5.2 Changeability - The capability of the software product to enable a specified modification to be implemented.
    NOTE 1 Implementation includes coding, designing and documenting changes.
    NOTE 2 If the software is to be modified by the end user, changeability may affect operability.
  • 6.5.3 Stability - The capability of the software product to avoid unexpected effects from modifications of the software
  • 6.5.4 Testability - The capability of the software product to enable modified software to be validated.
  • 6.5.5 Maintainability compliance -The capability of the software product to adhere to standards or conventions relating to maintainability.

Naturally, being a standard it has the “compliance” thingy which is usually only relevant for large organizations and project but for the most part the different aspects mentioned above are the parts you need to take care of when you want someone besides yourself to make changes to the software.

The view of Maintainability Ayende uses is problematic esp. when we consider that (successful) software will spend most its life in maintenance and not in development (you can read Robert L. Glass’s excellent “Software Maintenance is a Solution, Not a Problem” paper in this regard). Assuming someone maintaining the code will always be familiar with it is expecting the same developer(s) to stay at the same project for as long as the project will live (which is not likely) and/or assuming the project will have a short life (not something I’d want from my projects)

So don’t forget that other people will have to maintain your code and they probably won’t live the code-base as you do or as they say in “Code for the maintainer” in the C2 wiki

“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live. “ :)


* ISO 9126 is a multi-part standard for QA. I think ISO9126:2001 is good quick reference for quality attributes ( i.e. something you can look at when you try to elicit quality attributes for an architecture). I, personally think the other parts of the standard are pretty useless but that's another story :)
 
Tags: Agile | Project Management

March 18, 2009
@ 07:35 PM

Image via Wikipedia
You might have noticed my blogging is slowing down a little,  in case you're wondering the culprit is xsights - As we're getting closer to
production (2-3 weeks if all goes well) everything else slows down...

We all know about continuous integration (CI). We (xsights) are seeing a lot of ROI on the investments we made into signing up to CI esp. when we augmented by automated testing. Everyday we can and do actually release several versions and run load tests on them on different server seeking the best release to  "freeze". Where "freezing" means the  system that will be used in production.  That's the way it is done, right? You work on a release for x months (sometimes years..) and after an integration period at the client's site you go live and forget about them (save for show-stopping bugs) until the next major update several months later. For instance, setting up our "methodology" I figured a 6 months release cycles for major versions plus 1 month releases of minor version would suffice.

These days, however the more I think about it, The more I believe this approach is not the right one for our situation. We are going to go with Continous Deployment. There are many reasons for that  primarily:
  • The requirements stream (new reqs and changes) is still pouring in - and in increased rate.
  • We are building a service platform not an installable product -  This means we get to control where how it is run so there's less overhead in
    streamlining updates
  • It would be our first public installation - were going to roll out bug fixes anyway (I personally never write bugs... but you know ;) ). We'll need a proper procedure for updates anyway
  • We are doing this anyway - every new demo/internal run we have is using the "stablest" latest and greatest :)
What does  Continuous Deployment means? It means we'll be rolling out new production versions on a daily basis (or even more frequently!)
How that's going to work? Well, First there are the prerequisites:

  • Version control - something that can track and tag versions...
  • Continous integration which is set up and running. You cannot release continously if every new development offer breaks the system. An automated smoke test is
  • Automates tests that provide a good approximation of real usage
    (.e.g. usage patterns, loads etc.) - You need to be able to have a high
    confidence level that whatever you have.
  • Staging environment - An environment as close as possible to the production system (just like you'd have for a web-site)
  • Monitoring and alert system - You need to be able to identify problems quickly - the running

Then there's the process:
  1. identify a baseline Stable release -The "fallback".
  2. pick the most promising build of the day
  3. run it through more extensive tests (at least burn-down and load - which means a release maybe 1-2 days old by the time it hits the production system)
  4. if everyhting is cool deploy to staging - else fix bugs and repeat from 2
  5. repeat tests in staging environment
  6. Update system (This is a point that still needs work so we can support a version hot-swap i.e. without a shutdown)..
  7. If the monitoring system alerts unforeseen bugs (i.e. problems with the release) - roll back to stable release, try to add test for the problematic behavior and repeat from 2
  8. if the release worked good enough - mark it as the next Stable release

That's the plan. However as we all know, talk is cheap so be sure to check in here in 2-3 months to hear how did we fare :)

Reblog this post [with Zemanta]



 
Tags: Agile | Project Management | xsights

The year is almost done so I'd thought it would be a good time for a short retrospective into what I blogged here. The 13  posts below are the ones  I liked best this year. Turns out these posts touch on a lot of different subjects: requirement, software management, agile development, architecture, SOA and programming.



 
Tags: Agile | Project Management | SOA | SOA Patterns | Software Architecture | TDD

December 2, 2008
@ 10:03 PM
It looks like December is the conference season here. Dec. 9th there's going to be a SCRUM conference here in Tel-Aviv, featuring Ken Schwaber as the keynote speaker and Scott Ambler for the closing session. See more details here


 
Tags: Agile

November 23, 2008
@ 07:36 AM
Even if you don't write your tests first on a regular basis, you probably want to adopt that at least for debugging.
All the developers I know want to repro a bug before they make any changes to a the code (I know I do). When you write your test before you debug you do just that and you get several other benefits since you can
  1. Make sure that the bug is indeed reproducable
  2. Easily (and repeatedly) step through your code to pinpoint the bug
  3. Verify that the bug is fixed
  4. Get a regression test to make sure the bug isn't reintroduced
The only question now is what is the difference between a bug and a new feature?
One of my favorite things about UserVoice -- which we use for Stack Overflow -- is the way it intentionally blurs the line between bugs and feature requests. Users never understand the difference anyway, and what's worse, developers tend to use that division as a wedge against users. Nudge things you don't want to do into that "feature request" bucket, and proceed to ignore them forever. Argue strongly and loudly enough that something reported as a "bug" clearly isn't, and you may not have to to do any work to fix it. Stop dividing the world into Bugs and Feature Requests, and both of these project pathologies go away. (coding horror)

But that's something for another post :)

Powered by ScribeFire.


 
Tags: Agile | TDD

November 16, 2008
@ 10:52 PM
One important (yes, yes important) aspect of continuous integration (CI) is the way broken builds are reported. I'll tell you why but before that let's quickly recap on CI in general.
The point is to get an automated build process that run's after every check-in and takes the system for a quick spin to verify all the major parts are still OK. CI helps maintain the quality of the overall code by ensuring that components don't drift too much apart and that the current version is still valid to a reasonable degree.

Martin Fowler's article on CI (referenced above) lists the following practices
  • Maintain a Single Source Repository.
  • Automate the Build
  • Make Your Build Self-Testing
  • Everyone Commits Every Day
  • Every Commit Should Build the Mainline on an Integration Machine
  • Keep the Build Fast
  • Test in a Clone of the Production Environment
  • Make it Easy for Anyone to Get the Latest Executable
  • Everyone can see what's happening
  • Automate Deployment
What I am saying that a practice missing from this list is to maintain a clear concise broken build report. You see, whether we like it or not builds break. Not every checking (well hopefully anyway:) and not every day but that isn't a scarce sight either.
In our setup we get an email every time the build breaks. That's a good thing - since it increases the visibility of the problem. The message tells us who broke the build and what's the error (something doesn't compile, unit test broken or smoke test broken) and which file.

The problem is that is (was) that up until recently these details were unreliable for instance
  • The error message was so long nobody bothered to read where the problem is
  • Once a build was broken every new checkin also got the broken build notice - this time pointing the new person (doing the checking ) as the problem's source (even though the previous person was to blame"
  • etc.
The problem here is that unreliable reports quickly turn into white noise nobody pays attention to (someone at the team said it was like the boy who cried wolf -by the time it is really "your" problem you are no longer listening. When nobody pays attention, problems get to age - in the long run this can also invalidate the whole CI approach - somewhat like the broken windows theory. So for no broken builds the team has to pay attention to what's wrong. The way to do that is to invest in the build script to make it more easy to understand  and as reliable as possible  - this is something we did (and still working on). I know I don't want to take the risk of broken build white noise...

Powered by ScribeFire.


 
Tags: Agile | Project Management

Retrospectives, every "agile" team does retrospectives.What are retrospectives anyway?

A retrospective is a meeting where the team takes a look and inspect the past, in order to adapt and improve the future.

Agile or not, our team does a retrospective at the end of each iteration (every two weeks in our case). We try to look at what worked, what didn't , how we are meeting our goals etc, how is the product going etc.. These meetings provide a lot of value for steering us at the right direction.
On going retrospectives that look at the near past allows for suppleness and change adaptation and they are very powerful at that - However it is sometimes worthwhile to reflect over longer periods of time.

One area where longer perspective is important is the architecture of the project. Evolving an architecture you run the risk of accepting wrong decisions - mostly because architectural decisions have long term implications, while YAGNI, time constraints and life in general drive you toward short term gains.

Again, taking an example from my current project, working towards the first release, we took a few major decisions during the development e.g.
  • federated resource management - Taking into consideration the fallacies of distributed computing we decided that we'd have local resource managers that will take care of resource utilization and allocation. The resource managers will have a hierarchy where they'd communicate with each other to gain the "bigger picture"
  • Introduce Parallel Pipelines - handle image understanding by dividing the work between specialized components.
  • RESTful control channel - to use a "lingua franca" between all component types so that we can easily integrate across platforms and languages
  • local failure handling - resources and components handle failure by themselves
  • Communication technology (WCF in our case) is isolated from the business logic by an Edge Component
  • etc.
Once we finished delivering the first release. We took a few "days off" to consider what we've done thus far. updated our quality attribute list per our knowledge working with the system and looking at some customer scenarios. studies the things we liked/didn't like in the design and architecture of the working system. and revised a few of our decisions for instance
  • We found that rushing to a working system we introduced some excess coupling to a specific technological solution (for video rendering). We initiated a few proof of concepts and found out how to both isolate the technology from the rest of the system as well as allow more technology choices.
  • We found that the some of the data flows were not as clean as we thought they'd be - adding new features caused more resource interactions than we thought when we partitioned the resources. We redefined some of the resource roles to get less message clutter (and higher cohesion)
  • The federated resource management works well, but introduce needless latency in session initiation. We now opted for introduce "Active services" which are more autonomous.
  • Add a blogjecting Watchdog in addition to local failure handling to both increase the chances of failure identification and recovery as well as get a better picture in a centralized Service Monitor.
  • RESTful control channel worked well and will continue for later release
  • Some of the scale issues will be handled by introducing "Virtual Endpoints" while some would continue to use autonoumous endpoint creation and liveliness dissemination (hopefully learning from the mistakes of others)
  • etc.
The result of these and the other decisions we've maid is a rework plan that will (hopefully anyway) make our overall solution better.
What we see is that we evolved our architecture as we went forward. While all the the decisions we made seemed right at the time we took them, only through reviewing them in a wider perspective (architecture retrospective) we identified the decisions that we need to change and the ones that we have to enhance. The insight you gain after working on a project for awhile are much better than the initial thoughts you have or the understanding you master in the initial interations.
I think it is essential to review the architecture once you've gained more experience with the realities of the system you write (vs. the precieved realities you have on the get go)

By the way if you work with a waterfall approach your situation is worse. Since in this case you take your decisions before you write any code so, you don't even have the benefit of POCs, and working code to enhance your insights


PS
if you have the MEAP version of SOA Patterns you can read more on the patterns I've mentioned here: Active service in chapter 2, blogjecting watchdog in chapter3, Service Monitor in chapter 4, Parallel Pipelines in chapter 3, Edge Component in chapter 2


 
Tags: Agile | Project Management | REST | SOA | SOA Patterns | Software Architecture

August 3, 2008
@ 10:44 PM
Johanna writes about how you should fund project in an incremental manner especially when it comes to the most risky ones:

incrementally, in some sort of a timebox. (Gotcha!) But here’s why it make sense to do that:
  • If you show value to someone, preferably your customer, you are much more likely to get more funding.
  • If you’re not showing value early and often, you get feedback early enough to change before you start a death march for something your customers don’t want.
  • You can start highly risky projects, because you’re not committing a ton of money and time to that much risk. You’re just committing 2, 3, or 4 weeks.

I have to say I agree 100%. One nice anecdote in this regard, happened to me a few years ago. I was working for a rather bureaucratic, waterfallish organization. I wanted to outsource part of a project I was running and I found an agile sub-contractor for the job. I somehow managed to convince both my management and the methods and practices group that this project should be using SCRUM (the only "cost" for me was that I had to write a Software Development Plan , proving that SCRUM can be ISO 90003 compliant, but that's another story).
We set up a rough plan (release plan) with timeboxed releases every three months (every 3 iterations) and we were all set. Almost, that is - when I approached the procurement group they told me to hold my horses. They were not willing to go through the procurment process every three month. They also weren't very keen on the cost plus nature of the plan (the deliverables weren't prefixed as we were going to evolve the solution). To went over this hurdle I had to prepare a pseudo plan were they had "real" mile stones (basically based on the release planning mentioned above). We ended up budgeting the whole thing in advance, however I did convince them to establish sort of a blanket purchase order, so that we were able to release funds incrementally as we progressed.

 
Tags: Agile | Project Management

CrossTalk, the journal of defense software just published its Aug. 2008 issue. Which, incidentally is its 20th anniversary issue.
This issue includes a couple of very interesting articles one is called "Good Old Advice" by Alistair Cockburn and the other is called "What Have 20 Years Accomplished and What Is Left to Accomplish in the Next 20 Years?" by Gerry Weinberg.
Cockburn explains very eloquently something I also wrote about a year ago that the ideas and lessons expressed in literature of yore are not just valid today but they drive home the points of the "coolest" ideas we have today. Alistair does that by demonstrating how the principles behind the agile manifesto builds on the works of Fred Brooks, Barry Boehm, TomDemarko, Gerry Weinberg and many others.
We have known for decades that is is all about people , users, communications between them. Alistair summarize his article as follows (emphasis mine):
"

I derived the four values of the Agile Manifesto from much older, recognized, and highly reliable articles. The point I wish to make is that the authors of the Agile Manifesto did not throw out all previous experience as they wrote it. On the contrary, what they did was cherry-pick four of the most important issues from among hundreds. The appropriateness of their selection is seconded by decades of prior experience and data, as evidenced by the numerous articles referenced in this article.

For decades, we have learned, documented – and ignored – the same lessons:

  • Attend to the individuals and their interactions.
  • Put the mercilessness of running software to good use – write, and learn from the writing.
  • Get inside the heads of your users and customers, and get them on your side.
  • Plan coarse-grained long term and fine-grained short term, and find a way to replan quickly – because you’ll have to.

And, finally:

  • Do not believe in any single process or methodology because each works only in a particular and limited context.

Those who do not learn from history are doomed to repeat it. Let’s stop pretending we don’t know everything and let’s stop repeating painful history by following this good, old advice."

Weinberg writes a short retrospective of the last 20 years. Here are some interesting quotes (again emphasis mine):

Individual managers, however, have certainly improved. The good news is that we now have many more excellent managers who can mentor new managers. The bad news is that because the field has grown so much, there simply are not enough experienced managers to go around. We still see people abused or poorly used by their managers. All of this in spite of the fact that we have much more information about how to work with people in software development – information that, sadly, is frequently ignored. Too many managers have failed to learn that the world will not end due to a late delivery or a defect in production. Nor have they learned to relax, breathe, learn from their mistakes, and try again.
and

The idea of certification is only one of the management myths that has not changed in a generation. We still have an excess of heroic environments, often leading to death march projects. Many managers still maintain the fallacious perception that quality can be tested into slapdash software. Some still try to develop software without being bothered by potential users of that software. And some still believe that process models are the answer to all of our problems.

These are the things I notice on my pessimistic days. On my optimistic days, I notice more managers realizing that it is the people who make a difference, that they can hire talent, but then must also build the relationships to build a team. Quite a few organizations are now using process models successfully, but as only one of the tools providing information for informed cultural changes to their software community.

Though looking from different angles, both articles somehow hash the same ideas like 'work iteratively", "b e wary of magic or single processes" etc. I do believe that what underlines both these articles and, indeed, software development itself is that at the end of the day software is developed by the people and for the people and we should not allow ourselves to forget it.

*with pardon to Abe for the misqoute
 
Tags: Agile | General

Technical documentation of the code/project  seem like a worthy goal. After all if we aren't living in a vacuum someone will need to understand our code/design and maintain it, use it to develop new stuff etc.
On the other hand writing documentation can get tedious, boring, hard to maintain and whatnot - so you get all sorts of ways to deal with it. Consider, for example, the following:
  • On a recent article in IBM's DeveloperWorks Paul Duvall suggests you use automation to generate various documentation artifacts like UML diagrams based on the source code (using ANT tasks and UMLGraph), ERDs (with SchemaSpy) etc.
  • Another example, is a .Net tool called GhostDoc. This tool automagically generates C# XML documentation comments ("///") e.g. (all comments below are ghostdoc work):
/// <summary>
/// Appends the HTML text.
/// </summary>
/// <param name="htmlProvider">The HTML provider.</param>
public void AppendHtmlText( IHtmlProvider htmlProvider )
/// <summary>
/// Adds the specified item.
/// </summary>
/// <param name="item">The item.</param>
public void Add( string item )
/// <summary>
/// Determines the size of the page buffer.
/// </summary>
/// <param name="initialPageBufferSize">Initial size of the page buffer.</param>
/// <returns></returns>
public int DeterminePageBufferSize( int initialPageBufferSize )

[from Introduction to GhostDoc]
 What I think is that while both of these efforts can help satisfy a customer specific requirement for "comprehansive documentation"* they have very little value in making anyone understand anything about your code. UML diagrams can only help if they are created at a higher level of abstraction than the code (which means they'd be hand-crafted) and if GhostDoc can understand your code enough to create anything useful it means that your method and parameter names are self-descriptive anyway.

In a previous post I mentioned that I prefer to rely on tests, short methods and meaningful names for readability. I'll talk about tests in another post, for this installment lets look at the other two. I think it would be better demonstrated by an example.
Consider the following horror of a method:
       public void HandleWithFrame(FrameProp Frame)
        {

         int FreeProcessNum = 0;
         int FreeProcessId = 0;

         if (Frame != null)
         {
             rwl.EnterWriteLock();

             if (m_WaitingFrame.ContainsKey(Frame.m_SessionId))
                 m_WaitingFrame[Frame.m_SessionId].m_Frame = Frame;
             else
                 m_WaitingFrame.Add(Frame.m_SessionId, new WaitingFrame(Frame));

             IncrementPriority();

             rwl.ExitWriteLock();
         }

           rwl.EnterUpgradeableReadLock();
           foreach (var keyValuePair in m_ProcessMap)
          {
            if (keyValuePair.Value.m_Busy == false)
            {
               FreeProcessId = keyValuePair.Key;
               ++FreeProcessNum;

               if (FreeProcessNum > 1)
                     break;
            }

          }

           if (FreeProcessNum == 0)
           {
               rwl.ExitUpgradeableReadLock();
               return;
           }
       

           if (FreeProcessNum >= 2  )
           {

               rwl.EnterWriteLock();
            
                m_ProcessMap[FreeProcessId].SendFrame2CV(Frame);
              
                m_WaitingFrame[Frame.m_SessionId].m_NumProcess += 1;
                m_WaitingFrame[Frame.m_SessionId].m_Priority = 0;
                m_WaitingFrame[Frame.m_SessionId].m_Frame = null;

                m_ProcessMap[FreeProcessId].m_Busy = true;
              
 
              rwl.ExitWriteLock();
              rwl.ExitUpgradeableReadLock();
              return ;
          }


        


          WaitingFrame MaxPriority = new WaitingFrame();
          MaxPriority.m_NumProcess = 1000;
          MaxPriority.m_Priority = -1;

       
            foreach (var Item in m_WaitingFrame)
            {
                 if (Item.Value.m_NumProcess < MaxPriority.m_NumProcess)
                      if (Item.Value.m_Priority > MaxPriority.m_Priority)
                          MaxPriority.m_Frame = Item.Value.m_Frame;
            }
       
         rwl.EnterWriteLock();

       //  Console.WriteLine("ProcessId={0} Assign", FreeProcessId);
          m_ProcessMap[FreeProcessId].SendFrame2CV(MaxPriority.m_Frame);
          m_ProcessMap[FreeProcessId].m_Busy = true;

        
          m_WaitingFrame[MaxPriority.m_Frame.m_SessionId].m_NumProcess += 1;
          m_WaitingFrame[MaxPriority.m_Frame.m_SessionId].m_Priority = 0;
          m_WaitingFrame[MaxPriority.m_Frame.m_SessionId].m_Frame = null;
        
        
        rwl.ExitWriteLock();
        rwl.ExitUpgradeableReadLock();
        return ;

       }
This class needs a lot of explanations if you want to understand what exactly going on here. So you can set your self up to writing a lot of comments and trying to figure things our - or assuming this class was fully tested (which it wasn't, but that's another story)  try to refactor it until we get something meaningful (I will omit the added tests though for brevity)

Step1 - First If

It seems that when we have a first frame we want to keep it aside, so let's extract method all the if code to EnqueueFrame

           if (Frame != null)
               EnqueueFrame(Frame);

Ok so now we look at EnqueueFrame, The code we see here talks with the  m_WaitingFrame private member (which is a Dictionary of <Guid, WaitingFrame>();. The first thing we'll do is to rename it to FramesQueue. Now the more interesting thing is that the code here has to do with managing this FramesQueue and isn't directly related to the containing class.
We can either subclass the Dictionary class or we can add an extnention method to Dictionary<Guid,WaitingFrame> to handle this for us.
Do we'll do that and then refactor the If again:

           if (Frame != null)
           {
               rwl.EnterWriteLock();
               FramesQueue.Enqueue(Frame);
               rwl.ExitWriteLock();
           }

and Enqueue looks like (in a separate interanal static class)

        public static void Enqueue(this Dictionary<Guid, WaitingFrame> queue, FrameProp Frame)
        {

            if (queue.ContainsKey(Frame.m_SessionId))
                queue[Frame.m_SessionId].m_Frame = Frame;
            else
                queue.Add(Frame.m_SessionId, new WaitingFrame(Frame));

            queue.Prioritize();

        }
The advantage of what we've achieved thus far is both better OO design (separation of concerns) and enhanced readability by using intention revealing names and notation.

Step 2 - The foreach loop
Again we'll start with Extract Method, we can now remove the definition of FreeProcessNum from the beginging and we get
var FreeProcessNum = GetFreeProcessNum(ref FreeProcessId);

but this code is not really clear, for one we have to rename FreeProcessNum to FreeProcessesCount to make it more legible. and we have an ugly and hard to follow ref variable. It is probably better to apply the Single Responsibility principle and  seperate this into two distinct methods,  so we'd get
var FreeProcessesCount = ProcessesList.CountFree();
var FreeProcessId = ProcessesList.GetNextFreeId(); // we don't really need/want the ID but we'll fix that later

(as in the previous example we add extension methods to ProcessesList to make the code more intention revealing and for better seperation of concerns)
All we want to do in CountFree is count how many proccesses are not marked as busy so we can rewrite

var FreeProcessNum = 0;
foreach (var keyValuePair in ProcessesList)
{
        if (keyValuePair.Value.m_Busy == false)
        {
               FreeProcessId = keyValuePair.Key;
               ++FreeProcessNum;

               if (FreeProcessNum > 1)
                       break;
         }

 }
  return FreeProcessNum;
into
        public static int CountFree(this Dictionary<int, ProcessStatus> processesList)
        {
            return processesList.Count(item => item.Value.m_Busy == false);
        }

Thank you MS for adding Linq and Lambda expressions :). The same can be done for GetNextFreeId
 

Step 3 the two Ifs and the rest
Taking a deep look at the code we can see that the rest of the method tries to find a free processor and if there are enough processors send the frame, otherwise it should send the top prioritized. We can also spot a bug here that two different threads can get the same Processor and then try to send a message to it one after the other. Another potential bug comes from the way the maximal priority is found. There's an assumption there that the max priority would be 1000. While it isn't likely to happen it is still a hard coded assumption.
Anyway, if we continue and apply the same principles that got us here (Single Responsibility Principle, Don't Repeat Yourself, Intention revealing methods, coherence and opening classes to add specific functionaliy) we get the original method to look like the following:

       public void ProcessFrame(FrameProp nextFrame) //was HandleWithFrame
        {
           rwl.EnterWriteLock();
           try
           {
               if (nextFrame != null)
                   FramesQueue.Enqueue(nextFrame);

               TryDispatchTopFrame();
           }
           finally
           {
               rwl.ExitWriteLock();
           }

       }
Compare this with the original method....
Also note that it isn't that the functionality disappeared - it is just neatly distributed and grouped in short related methods in related classes e.g.

    internal static class FramesQueueExtnesions
    {
        public static void Enqueue(this Dictionary<Guid, WaitingFrame> queue, FrameProp Frame)
        {

            if (queue.ContainsKey(Frame.m_SessionId))
                queue[Frame.m_SessionId].m_Frame = Frame;
            else
                queue.Add(Frame.m_SessionId, new WaitingFrame(Frame));

            queue.UpdatePriorities();

        }

        public static void ResetSlot(this Dictionary<Guid, WaitingFrame> queue,Guid slotId)
        {
            queue[slotId].m_NumProcess += 1;
            queue[slotId].m_Priority = 0;
            queue[slotId].m_Frame = null;
        }


        public static void UpdatePriorities(this Dictionary<Guid, WaitingFrame> queue)
        {

            foreach (var Item in queue)
            {
                if (Item.Value.m_Frame != null)
                    Item.Value.m_Priority += 1;
            }


        }
        public static FrameProp FindTopPrioritized(this Dictionary<Guid,WaitingFrame> queue)
        {
            var maxPriority=  queue.Max(item => item.Value.m_Priority);
            return queue.First(item => item.Value.m_Priority == maxPriority).Value.m_Frame;
        }
    }

You should note that this is not the end of the refactoring (e.g. we should still handle the WaitingFrame, FrameQueue and the ProcessesList which we are called here) we just took a look at a single method.

While there might still be a need for an occasional explanatory remark , I think this little exercise demonstrate that we can gain a lot in the way of clarity by refacroting code and keeping up a few simple principles. Oh yea, and what we got at the end of the process is not just readable code, but also a more maintainable, better designed code that can move forward and evolve further as the system changes.



* I don't underestimate the value of generating full documentation when there's such a requirement from a customer. I would prefer to convince a customer that having such a Write-Only document is a complete waste of time  and trees but sometimes you can't help it. Generating documents in these situations can be a life-saver.


 
Tags: .NET | Agile | refactoring

While I am on the subject of Project management, One nice feature of Mingle (I am not sure if it is new in v.2 or it was there in v1) is requirements traceability.
Requirements traceability is a requirements management discipline which makes it possible to attribute artifacts (design/code etc) to the requirements that originated them. It also provides a way to mange changes (if you know which artifacts were created to fulfill a requirement you also know the artifacts that are likely to change when the requirement change).

However, above all that, Requirements traceability is a pain in the ass* ! Which means it is hardly ever done. Especially in agile projects (working code over extensive documentation, remeber?)

Anyway, as I said, a nice feature of mingle is that, if you set it to work with your subversion repository (I created a user for mingle) and when you check in code you add the card number to the check-in comments (e.g. #123 - fixed this and that bug). Mingle will let you navigate that relation. i.e. in the History -> revisions you can see the card number and click through to the card it-self. If you also do the same from tasks to stories you get a good traceability with very little effort



And I can tell you that from personal experience having once volunteered(!) to do the whole traceability matrix for a 1000+ man/year project from the analysis to the customer requirements ( it took more than 2 weeks in case you are wondering)
 
Tags: Agile | Project Management

June 2, 2008
@ 09:56 PM
When you are working towards a specific goal esp. if it is a looming mile stone or something to that effect, you tend to leave stuff behind, cut some corners, just a wee bit of slack... This "stuff" has a nice metaphor to describe it. It is called "Technical Debt" .

If you don't mind your technical debt it can grow so much you'd end up with a big-ball-of-mud which is hard to maintain, change and, well, do anything useful with.

The best way, I've found to deal with this is to "add a card" every time I feel the urge to add a //TODO comment. Or in other words add the technical debt as a task into the product/task backlog.
Having the technical debt on the backlog has several benefits such as
  • It will not be forgotten - it will be documented...
  • It will not be hidden - The true state of the product will be in the open for management/product owner to see. As a manager I want to know the true state of the product. If I know what I can and can't have I can get ready for that. If I think everything is rosy and then the system blows up in my face, that's not so good..
  • It will be managed - The importance/relevance of the "debt" will be reevaluated every time the product backlog get prioritized.
Technical debt will occur in your project, whether it is agile, "water-falled" , incremental or what not. Don't ignore it



 
Tags: Agile | Project Management

March 30, 2008
@ 11:07 PM
I never really understood would I want to use sticky notes/cards  in my agile projects rather than use a software tools. After all we write software for others - how can we say that software is not a good enough solution for us ?
For instance, in our team we use Mingle by Thoughtworks. I think it is one of the best tools for project management I've ever used.  Its serves us well for sprint planning, daily scrums etc. Among other things it gives you "digital" sticky notes which you can drag around to move between statuses or whatnot - just perfect.

And yet, last week when we were pushing for completing our first major milestone I decided to try real sticky notes, and finally I understood the difference - constant visibility. Computerized tasks seem to be just as accessible as physical cards but in fact they aren't. Yes, everyone can see the status whenever they want. But the fact that you can doesn't mean that you do. Digital cards don't have the "in-your-face" kind of effect that always visible notes has. I've seen a notable  difference in maintaining the dev team's focus and making managers (the CEO in my case) feel they are in the loop and that progress is constantly made.
My conclusion is clear. While I still use electronic tools, I guess, until we'd have large enough e-ink boards to allow us to get the same effect physicals cards give us I'll just have to keep restocking my sticky notes inventory :)


 
Tags: Agile | Project Management | SCRUM

January 9, 2008
@ 10:51 PM
A while ago I wrote about use cases and user stories and how I use both, starting with user stories for ease of estimation and manageability and expanding them into use cases as a way to elicit more requirements.
I just read a recent blog (wiki) entry by Alistair Cockburn (of "writing effective use cases" fame) called  "why I still use use cases".
Alistair explains that he sees (with companies he consults to) three main problems with user stories:
    1. lack of context (what's the largest goal)
    2. sense of completeness - that you covered all bases relating to a goal.
    3. no mechanism for looking ahead at upcoming work.


I think the first reason is partially dealt with by using themes, although use cases provide more context if we also consider the brief description, pre conditions and post conditions.
Also I am not sure I agree (maybe I don't understand) the third one - since you can have a backlog of future work regardless of the requirements methodology you use.

The second point, seems to be around  the same reason I have for using use cases - elicitation of requirements. Indeed, you can see that reading reasons 3 and 4 (of the 5 reasons Alistair  mentions for still using use cases) :
"
3. The extension conditions of each use case provide the requirements analysts a framework for investigating all the little, niggling things that somehow take up 80% of the development time and budget. It provides a look ahead mechanism, so the customer / product owner / business analyst can spot issues that are likely to take a long time to get answers for. These issues can and should then be put ahead of the schedule, so that the answers can be ready when the development team gets around to working on them. The use case extension conditions are the second part of the completeness question.

4. The use case extension scenario fragments provide answers to the many detailed, often tricky business questions progammers ask: "What are we supposed to do in this case?" (which is normally answered by, "I don't know, I've never thought about that case.") In other words, it is a thinking / documentation framework that matches the if...then...else statement that helps the programmers think through issues. Except it is done at investigation time, not programming time. "

Alistair also says that working with short iterations means you have to break the use case into stories :
These days, iteration/sprint lengths are so short that it is not practical to implement an entire use case in just one of them. That means additional work is needed, to create user stories or backlog items for each use case, track that each one get developed, and ensure that the complete set of user stories or backlog items do indeed deliver the subset of the use cases needed for the particular release.

I think that working just from the use case toward stories  is limiting, since it is many times easier to think up a user story and figure out the larger context later on (along with the other details of the use case). Also, as I've mentioned in the above mentioned blog entry, user stories are also easier to estimate not just easier to build.



 
Tags: Agile | Requirements

December 6, 2007
@ 11:43 PM
Microsoft uses the "live labs" to release all sorts of test balloons. Sometimes we get really nifty stuff like Photosynth or SeaDragon. Unfortunately, sometimes we get stupid not so bright ideas like Volta.

Ok, so what is Volta? Here's what the project's homepage has to say (emphasis mine):
" The Volta technology preview is a developer toolset that enables you to build multi-tier web applications by applying familiar techniques and patterns. First, design and build your application as a .NET client application, then assign the portions of the application to run on the server and the client tiers late in the development process. The compiler creates cross-browser JavaScript for the client tier, web services for the server tier, and communication, serialization, synchronization, security, and other boilerplate code to tie the tiers together.

Developers can target either web browsers or the CLR as clients and Volta handles the complexities of tier-splitting for you.  Volta comprises tools such as end-to-end profiling to make architectural refactoring and optimization simple and quick. In effect, Volta offers a best-effort experience in multiple environments without any changes to the application."

The idea sounds very compelling - I kid you not. So what's the problem?

The first issue is that, as a platform/framework (MS would say factory), Volta tries to accomplish too much. On the one hand Volta is another go at the web/desktop convergence trend. On the other hand it is supposed to be a solution for "painless" tier-splitting. Both of these tasks are very heavy. My opinion is that the Single Responsibility Principle (while originally defined for objects) applies here. And Volta should choose one thing and try to excel in that.

What's more disturbing to me, is the automatic handling of the "complexities of tier-splitting". Here's another excerpt from the Volta site which further explains the "tier-splitting" concept:
Objective

We have an application that runs in a monolithic environment, say the browser. We want parts of this application to run in other environments, such as servers. We don’t want to litter the application with plumbing code.

Rationale

The standard techniques for distributed applications infuse our code everywhere with information about what parts run where. This makes the code hard to change. Typically, once we make these decisions we can’t change them because it is too expensive. However, environments, requirements, and performance profiles change and we’re stuck with applications that can’t adapt to new realities. We need to separate the concerns about what the application does from the concerns about where parts of the application run.

Without Volta, we are forced to decide where code runs before we know everything it is going to do, in particular before we know the communication frequencies and delays. Development methodologies force us to make irreversible decisions too early in the application lifecycle. Volta gives us the means to delay decisions until we have adequate information to base them on.

Recipe

Volta tier splitting automates the creation of the communication plumbing code, serialization, and remoting. Simply mark classes or methods with a custom attribute that tells the Volta compiler where they should run. Unmarked classes and methods continue to run on the client.

We may base our decisions about tier assignment on any criteria we like, such as performance or location of critical assets and capabilities. Because Volta automates boilerplate code and processes for dispersing code, it is easy for us to experiment with and change assignments of classes and methods to tiers.

Wow, Agile development at its best, allowing us to postpone architectural decisions,  that just sound too good to be true. Well, the problem is that it is too good to be true. Abstracting the network out, and providing location transparency without thinking about the  implications of distribution is the reason "distributed objects" failed. e.g. Here is what Harry Pierson (DevHawk) had to say about distributed objects:
"...back in 2003, mainstream platforms typically used a distributed object approach to building distributed apps. Distributed objects were widely implemented and fairly well understood. You created an object like normal, but the underlying platform would create the actual object on a remote machine. You'd call functions on your local proxy and the platform would marshal the call across the network to the real object. The network hop would still be there, but the platform abstracted away the mechanics of making it. Examples of distributed object platforms include CORBA via IOR, Java RMI, COM via DCOM and .NET Remoting.

The (now well documented and understood) problem with this approach is that distributed objects can't be designed like other objects. For performance reasons, distributed objects have to have what Martin Fowler called a "coarse-grained interface", a design which sacrifices flexibility and extensibility in return for minimizing the number of cross-network calls. Because the network overhead can't be abstracted away, distributed objects are a very leaky abstraction.


So here comes Volta and tells us just put a [RunAtOrigin] attribute on the code you want on another tier and if you don't like that you can change it to another place in your application and what not. Note that the notion that you can automate some or maybe even all of the distribution "boilerplate" code may be viable. The problem is in the premise that you can seamlessly move that boundary around. There's a fundamental  difference between tiers and layers. Tiers should be treated as a boundary .Volta designers do talk about Security but they seem to forget a few of the other fallacies of distributed computing...



 
Tags: .NET | Agile | OO | Software Architecture | Trends

In the previous installment I talked about the architect and the architectural decisions, I also said (ok, wrote) that architects do more than that. Well, here are a few of the duties I think  architects should have (sometimes not exclusively)

project CTO - Tom Berray has an excellent paper describing 4 models for the role of a CTO. 3 of them can be applied to software architects (within their projects)
  • "Big Thinker" - This is somewhat akin to the role discussed in the previous post.
  • "External Facing Technologiest" - I usually saw this in larger projects, but it is also applicable for smaller ones. There are many occasions where the technical capabilities of the project have to be presented and/or negotiated with external stakeholders. Architects are in a good position to perform this as they should have good understanding of both the business and the technology. Additionally making architectural decisions already requires the architect to understand the different stakeholders' needs
The third model is called "Technology Visionary and Operations Manager" - Making sure that technology works to deliver business goals - but how is that done?

In their book on organizational patterns, Jim Coplien and Neil Harrison, talk about the "Quattro Pro for Windows" (QPW) development team. According to the case study, Borland had a team of 4 architects who worked together to produce what the authors call prototypes*. 6 month later these architects were joined by additional developers to produce the product. During the development the architects kept meeting on a daily basis to coordinate their efforts (sort of like a daily stand-up in a scrum of scrums).

The situation in the QPW is probably close to the ideal architect involvement in a project - coding architects that work closely with the team, while driving technical and architectural decisions. The availability of multiple architect (but few to prevent the "design by committee" effect) also enhances the overall quality of the solution.

Another aspect of the architect work is to act as a coach/tutor. It isn't enough for the architect to "know best". We already know that architect must also be able to reason about their recommendations/decisions, but that's just part of the story. Helping other team members get better in what they do means that they'd be able to do their job better, they'd be able to come up with their own ideas (and get more fresh ideas into the discussions) and produce better software. Since the architect is ultimately responsible  for the quality of the solution, making others perform better should be a top priority for the architect. Being considered as a source of knowledge will help an architect perform his/her role, even when they don't have an architect title


Actually, What they did were POCs or spikes  (see "Architecture Evaluation in code" for an explanation of the differences)


 
Tags: Agile | Software Architecture

Jeff Atwood (of Coding Horror) writes about Brian Foote and Joseph Yoder's "Big Ball of Mud" paper but completely misses the ball (pardon the pun). Jeff says that the paper describes
"classic architectural mistakes in software development."

while the paper describes the exact opposite. with the exception of the "Big Ball of Mud" pattern itself which can be seen as an anti-pattern (as,by the way, the authors explain) the rest contain exceptionally good advice on how to prevent problems. Let's look at them one by one
  • Throwaway Code -  when you just want to make sure something works (prototype, spike etc.) or even a quick fix - you don't write elaborate and heavy code - instead you write something simple to solve the problem. Throwaway code only becomes a problem if you don't actually throw it away...
  • Piecemeal Growth - The waterfallish "big plan" in advance has failed numerous times - so instead we should build incrementally, don't over plan or over design, build things into libraries only when it is proven to be recurrent problem, refactor etc.
  • Keep it working - The pattern is basically about building often and keeping iterations short, having tests that always prove your code still works as you make changes to it
  • Shearing Layers - This pattern is about identifying related components. We all know change is inevitable - however things don't change at the same rate. For instance we can probably get interfaces to be a little more stable than the implementations behind them. Frameworks evolve at a different rate than the business they support etc. The pattern is about dividing the system so that things that change at similar rates are together i.e. in the same package,  CSCI  or whatever you use to partition your system.
  • Sweeping It Under the Rug - When you find that you do have badly designed or badly implemented code - the pattern suggest you localize and isolate it to a fixed area (e.g. behind a facade) to prevent it from propagating into the rest of the system.
  • Reconstruction - This pattern is about understanding when the code base is so bad that it is better to start-over and rebuild from scratch rather then try to patch it. In many  way reconstruction is not a good or easy thing to do however the point here is to identify when it  is the lesser evil. 
Oh, and what about "Big Ball of Mud" itself - essentially from the architectural perspective it is indeed an anti-pattern - you have something that is not very maintainable, hard to understand and what not. However we should keep in mind that the idea behind designing an architecture is not to get the best, cleanest architecture. The idea is to make the right tradeoffs so that you'd be able to deliver the best overall solution under the constraints you face (budget, time, the team's skill and what not). If you're biggest constraint is time-to-market and your architect spends all eternity planning the 8th wonder of the world, fire his ass. I'd rather live with a Big Ball of Mud for the first release than not ever make a release...
Big Ball of Mud can be considered a pattern for pragmatic approach to building working software. This is probably not acceptable in the long term  - but it can be a good option for short term if you are aware that that's what you are doing and willing to treat what you get as "Throwaway code".

So again, except maybe big ball of mud, these patterns are not "project pathologies" as Jeff calls them - these are very good ways to keep delivering business value and working softwares



 
Tags: Agile | Design | Software Architecture

November 14, 2007
@ 12:15 AM
Jeremy D. Miller writes that he prefers user stories better than use cases - and I basically agree with him. Mostly because user stories are finer grained which lends to make them more "estimateable"  and manageable then use cases are.

However, having used both, I think there's one thing use cases do better then user stories which is eliciting related scenarios. When you write a use case that covers a  business scenario the use case template makes you think about variations and exceptions. When you try to think of a theme of user stories if you're not careful to do that in a structured manner you may miss some.

One important point regarding use cases is that you don't have to treat the use case a single monolithic block. When you work with use cases it doesn't have to mean that you sign-off use cases in a waterfall-ish manner- that is something that has to do with your development methodology.
You can develop use cases incrementally i.e. only do some of the use cases per "release". You can also elaborate use cases   iteratively by only elaborating/identifying some of the scenarios in each iteration (in a manner similar to user stories). I have successfully use both incremental and iterative use case elaborations  on several projects and I find it very useful (I even wrote paper summarizing my use cases experience a few years ago (warning - more than 60 pages with the examples and all..)

In fact, even today I find it useful to use both tools where I would usually start with identifying a user story and writing that down. Then map it to a use case (either existing  or a new one) - where a use case roughly equals a user story theme. As I mentioned previously I can them identify more user stories and add them to the product backlog. The use case stays a skeleton with links to the user stories and helps provide more context to individual use stories (ok, so that isn't a use case in the traditional sense, but I find it useful anyway).

Generally speaking, it is important to note that while it is tempting to equate a user story with a scenario in a use case, that isn't always true. sometime a user story would be a step in a use case and sometimes a story is shared by several use cases - since a fine grained feature that delivers business value is can be used in many business processes which is what use cases are oriented to


As a side note, if you have a project where you do have to write all the requirements up-front (as if they won't change anyway...) then I find that use cases are way superior than IEEE 830 style requirements ("The system shall...")  - but hey, that's another story :)


 
Tags: Agile | Everything | Project Management | Requirements

Let's assume I convinced you that some projects need architects (see part I). Convinced, you go and hire an architect. now what?

Let's start by looking at  "architectural decisions" - which is sure sounds like something we'd want an architect to do. I read once (I think that was Martin Fowler) that an architectural decision is a decision that in hindsiight you wished you made right. if we look at a formal definition of software architecture (say from IEEE 1471) we see that the architecture embodies the fundamental decisions about the system its components, their relations and their properties. Using this definition an architectural decision is a fundamental decision about the system (which pretty much explain why we want to make them right etc.)

Well, here are two observations on what I've said thus far. One is that we would want to postpone architectural decision as much as we can, since changing them will cause us a lot of headache. The problem is that in order to postpone an architectural decision we need to build flexibility into the system which is an architectural quality in itself - which might not be the top of the list if we prioritize it vs. other architectural qualities we need.

The second observation is that if we "refactor" the pretty language out from both of these definition - we can see that an architectural decision is basically a guess, hopefully that's an educated guess but it is a guess none-the-less. and as Albert Einstein once said it is  hard to make  predictions - especially about the future.

This is why architects  breadth of knowledge - which helps explain the architect training program I posted about a few weeks ago (see Architect training program Part I and Architect training program Part II). Another aspect is experience. And to get a wider perspective it can be helpful if this experience includes other roles besides developer such as project manager or business analyst etc. Another important component is domain knowledge and understanding of the business.

Using all these you (as an architect) may come up with a reasonable architectural decision (e.g. use MVC pattern) and a design to match it and that's it.

Well, actually, not quite since as I said earlier it is still a guess. Remember  an architectural decision (and any design for that matter) is a mirage no matter how beautiful the power point slide looks (or white board or UML sketch etc.)

Alas, power point compilers are still in the making. Which means that as an architect, you must be able to prove your point in writing - that is coding. While you are at it, you also need to know a thing or two about the technology you are using because it too has an architecture, features etc. which can have a significant effect on the end result. (You can read a little bit more on this in the "Architecture Deployment" paper I published a while ago).

The result of trying to postpone architectural decisions, ever changing requirements along with adding details as we unfold the  architectural abstraction level to a working system, is that the architect can't just appear at the inception of a project and disappear afterwards - they need to stick around for the game. This is especially true if you want to have an evolving architecture

An architect needs to do more than "architectural decisions". There are also additional reasons why the architect should have continuous interaction with the rest of the development team. However that will have to wait for Part III. :)




 
Tags: Agile | Everything | Software Architecture

A side effect of my decision not to become an independent consultant at this time means that I have to shelve some of the projects I was considering. One of these projects was to create a training program for software architects which I was discussing with a couple of training centers here in Israel.
Since It seems I am not going to promote it, I thought I'd share what I think a training program for .NET/Java architects should look like in the hope that someone would find it useful and do promote it (or parts of it)

Soft Skills
The way I see it architects needs a bunch of soft skills to be able to perform their roles.
Here is the list I identified in the past (by the way, I began a series of posts on each of these skills and never got to finish it - maybe it is time that I will :) )
  • Leadership. Influencing others to accomplish tasks and following your guidance
  • System thinking. Understand decisions and constrains in the
    wide scope pertaining to whole of the solution at hand. This includes
    the ability to abstract problems.
  • Strategic thinking. Understanding decisions and constrains and their alighments to the overall business of the company.
  • Organizational politics. Understand the environment you operate in and how it influences you.
  • Communications. Making sure you get your point across.
  • Human relations. Understand the "people" aspects or human
    factors and dynamics. This includes things like negotiation, pragmatism
    etc
I am not sure if you can teach all of them, but few courses that can help (in my opinion) include:

  • Presentation Skills - While getting the architecture and technology right is what matters, if you can't explain it to the different stakeholders you're toast.
  • Strategic Planning - This has to do with the vision thing I expect architects to manifest. Note that having a vision should not be confused with future-proofing a solution. future-proof means excess work not needed. Having a vision is knowing where you want to end - it can still be perfectly valid to completely re-write your applications along the way


Project Management
While the architect is not the project manager (mostly anyway), I think understanding the constraints coming from the project management point of view is very important. Since most environments call for a mix of agile and formal disciplines (hey, you've got to be pragmatic). I would train architect both in SCRUM and RUP (or some other formal methodology)

Also while not all environments needs this I would give an 2 days overview of important standards. The first would be IEEE 1471, which defines a standard for documenting software architectures. I would also teach ISO 90003 and CMMI.

It should be noted that the ISO 90003 is much better than the previous incarnation (ISO 9003) as it basically lets you define what you want to do to cover the different areas. The standard just helps you make sure you think about the various parts of project management (requirements, environment etc.). For instance I demonstrated how key areas of 90003 can be mapped to SCRUM to get it approved on my last project.


Languages, Design  and Patterns

I would want the .Net/Java architect training program to include at least 2 of the following languages:
Ruby, Scala, Erlang, F#, Python , Groovy ,OCaml
The reason for this is that these languages have different design goals than .NET and Java so learning them gives you additional perspectives and broaden your horizons for other ways of thinking (even if you don't use them in your project directly). You might have noticed that there's no .NET or Java training here. The reason for that is that's a prerequisite as far as I am concerned. You should master at least one

Object Oriented principles - hopefully aspiring architects already know this. However, I often see people who discovered some of the principles by themselves but haven't heard about all of them.
I am talking about principles such as Liskov Substitution  Principle , Open Closed Principle, Single Responsibility Principle , IoC containers, Don't Repeat Yourself  and YAGNI (I summarized my opinion on most of them in this paper)

The next step is to cover some design issues like Domain Driven Design, UI Design, Database modeling, Database alternatives l(after all the database is dead, right? )

Advanced design patterns - When most people hear the term design patterns they think about the GoF patterns. There are however literally hundreds of design patterns. Some of them are even worth learning :) . For instance there are patterns for concurrent and parallel systems like Proactor, Reactor Half-sync/Half-Async etc; Workflow patterns like Cancelling partial Join, Recovery Action etc; SOA patterns (ok, so I am still working on that :) )



Architecture Workshops
Another important part of the training, in my opinion, is to do some workshops and actually try to apply some of the material covered.

  • Architecture Evaluation - workshop 2-3 days - It is probably worthwhile to delve a little on scenario based evaluation techniques such as LAAAM and ATAM. While I prefer evaluating architecture in code, the scenario based thinking is very valuable for eliciting architectural requirements
  • Distributed Systems Architectures workshop - I'll expand on this in the next post


Lastly, there are also a few miscellaneous subjects like architect 101, the SPAMMED architecture framework , Agile architecture, Behavior Driven Design , common frameworks (though hopefully this would  not be needed ) like Spring/Spring.Net, Hibernate/NHibernate, iBatis  etc.


PS

Note that there are a few architect training programs available out there
One is offered by the Software Engineering Institute (SEI) and includes a 6 courses. SEI program seems to be focuses on formal sides of architecture as it includes courses on documenting software architecture and ATAM (You can see an old presentation I have on ATAM here)
Dana Bredemeyer  also offer architect training. Dana offers several workshops that cover the software architecture profession.
TOGAF (which is more of an enterprise architecture framework) offers both a certification and courses
Lastly, IASA is considering creating a software architect program and has a few courses in development
If you know any others I'd be happy to hear about them


 
Tags: .NET | Agile | Design | Everything | Java | OO | Software Architecture

September 2, 2007
@ 10:02 PM
Jeremy Miller recently tried to "answer hard questions about Agile development". One question he didn't really answer was that of agile and fixed bids :
"I don't really think the Agile answer to a fixed bid is any different from any other process.  I do think that Agile practices and project management can give you far more control and feedback on the "Iron Triangle" of resources, time, and features"

Jeremy says that this would enable the team to fail "softly" (i.e with some functionality etc.).
I think (well, actually, I know since I did that) we can do better than just use Agile and hope for the best. For instance, here are a few  strategies I successfully used:

1. Pre-budget the whole project with rough milestones and estimates (works best if you have prior experience in the domain)
2. Coordinate expectations with the client and trade - the single project to many smaller ones. What I did was to get a blanket purchase order for the estimated amount and each smaller order/project  "ate" some of that budget until it was done.
3. Since the project was for a CMMI level 3/ ISO 90003 we also added a few documents to the backlog - and let the product owner prioritize them vs. deliverables. For instance the "Architecture document" was delivered after 6 iterations when the architecture stabilized
4 exchange requests instead of change request - new functionality comes instead of old promised one

Of course we also did the other SCRUM practices of releasing frequently, demonstrating progress, retrospective etc.

I would love to take credit for the ideas above - however, Pascal Van Cauwenberghe details most of them (and more) in two papers called "Agile Fixed Price Projects part 1: The Price is Right" and "Agile Fixed Price Projects part 2: Do you want agility with that" (go read them..)


 
Tags: Agile | Everything | Project Management | SCRUM

August 1, 2007
@ 09:52 PM
I won't say anything about my presentations (that's for others to say :) ). The point of this post is just to let you download them. So here they are:
  • SOA Patterns (2.14mb) - Takes a look at different strategies (patterns) to solve common SOA pitfalls
  • Getting SPAMMED for architecture (4.56mb) - Takes a look at the activities architects can/should do when they think about software architectures. The presentation also covers architecture in agile projects.


 
Tags: .NET | A&D2007 | Agile | Everything | SOA | SOA Patterns | Software Architecture | SPAMMED Process

Just got back from Chicago, where I attended Architecture & Design World. The event was great. it had many interesting talks and sessions  which I'll blog about in  few of the following posts (Also I'll upload the slides from my sessions later this week). One session however, was very disappointing for me, Ivar Jacobson's Keynote on "Next Generation Process with Essential Modeling".

I was really looking forward for this keynote; After all Ivar is one of the "Three Amigos", father of use cases and all. Also I had the chance to attend a couple of his presentations several years ago and he had a very interesting and convincing appearances at those times. Unfortunetly this session was nothing like those previous events.
Ivar talked about his new methodology. The basic idea behind the essential software process is (in my opinion) correct. Instead of big bang prescribed processes - it is better to tailor a process from a set of practices to get something that fits the organization/project at hand. It might be a little unnerving to hear this from one of the people who brought (upon) us the RUP, but, I guess it is good to see people who are constantly learning.

The problem was that while the main idea can be summed up in one line (see above) Ivar went on fro the better part of the hour to explain how cool it was that his base set of practices comes on a set of que cards that can be sorted and arranged. He also went on to explain the game board (or something to that effect) where you can lay different cards to both describe your current process as well as the bunch of practices you want to use for your future process.
If that wasn't enough, when he (briefly) shown us the cards that talk about the architecture and architecture description practices the advice/guidance there was very mediocre and general
I expected much more from someone of the stature of Ivar Jacobson.
On the up-side this was only one keynote and the other sessions I listened to where much better - as I already said, I'll blog about few of them in upcoming posts.

 
Tags: A&D2007 | Agile | Everything | General

Agile and documentation? what gives? First things first documentation is not something that is prohibited by Agile Manifesto. Working software is definitely preferred over "comprehensive documentation" but there can also be some value in documentation

The first question is why would we want to document anything if we have a working software. I think there are several stakeholders like project newcomers, maintainers etc. who will be interested in something that will let them get up to speed and provide them an overview of what's going on before they delve into code. You can read more on that in  a post I wrote almost a year ago called "Who needs a software architecture document" but in essence the main motivation for documentation is that assuming that the software is successful it will outlive the team - i.e. the people that built the software will not be the ones that will have to develop, maintain and support it for most of the software's life.

If we agree that we need a software architecture document the question is what to document and when.

There are two main "things" you can document in regard to architecture, the first is the obvious one: the architecture itself. In my experience the most value can be derived from documenting recipes i.e. how  to do stuff that is common in the architecture. These recipes can be a short description of the context and then a
pointer to a test (or tests) and an implementation that exercises this. You can think of the recipes as a type of a tutorial to the architecture.

Other documentation worthy elements related to the architecture are an overview and technology mapping (including what a developer needs to install to start working). The overview allows a newcomer to understand where to find what, the technology mapping allows for understanding what she needs to learn and install to be productive. Note that to be useful the overview should be at a higher level of abstraction than the code - otherwise you run the risk of missing the forest for the trees or at least not saving any time.

It is obvious that documenting any of this before your architecture is stable more or less is useless, as a rule of thumb I would say this can be around the 5th-6th iteration - assuming the team has to grow during the project. If the team stays stable for the duration of the project, this documentation can take place towards the end of the project (though I would probably add recipes to a wiki or something similar during the project as development patterns emerge).

The second "thing" you can document in regard to architecture are the decisions you decided against, in my opinion this is more important than all of the other items mentioned above together.  The reason for that, while it might take a while to understand a well written software and infer its architecture it can be done, but it is virtually impossible to understand the options that were disqualified from looking at the chosen solution.
Understanding the options that weren't used can save time for the person reading that description. both in understanding why things are the way they are. Furthermore it can save time trying things that didn't work or provide clues to options when the circumstances change (since, as we all know, requirements change..)
The best time to document decisions you decided not to take is when you opt not to use them - this is when you remember best "why". for instance, in my current project we use x.509 certificates to authenticate clients and we use decided to use Kerberos tickets to authenticate components within the service. There's a reason for making that translation*,   there's also a reason for making the transition by replacing the client certificate with the edge component's credentials instead of mapping the client's certificate to a Kerberos ticket using an Identity provider*. we had two developers spike different options for two weeks until we came to the current solution instead of the more obvious choice of passing the x.509 certificate from the edge into the service and using the client's credentials. This question is likely to come up when/if someone else would take over the project, when the technology will be updated etc. Again, if we know why we didn't make that choice we can better decide what to do when the circumstances change.

To sum up, there are few architecture related issues that are worth documenting even in agile projects. some of them can be postponed some of them are worth documenting a little earlier. In any event it is better to document after the fact and to keep the documentation light.





* It all has to do to to limitations of WCF in regard to  the transports we use (HTTP, MSMQ and TCP) and the request/reaction pattern (asynchronous communication) we use.



 
Tags: Agile | Everything | Software Architecture

Yesterday I attended Jim Coplien's presentation on "Organizational Patterns - a Key for Agile Systems Development". Overall I think It was a very good presentation. Jim makes a few interesting claims, some of which are controversial within both in the traditional and the agile spaces
Few examples
  • Process guidance (ISOs etc.) doesn't work - roles are stabler than processes, processes always change.
  • Jim says that in order to make a change you need to make it at the organizational structure level. The processes will then support these changes
  • TDD is evil - it is just an re-incarnation of bottom-up procedural design. It is better to follow "Design by contract"
  • He says XP is not a good methodology (He thinks SCRUM is good)
  • etc.
Additionally he talked about some of the organizational patterns he and Neil Harrison discovered studying organizations for more than a decade. You can  read the Top ten patterns on his site.

Jim covered 2 patterns that are related to software architecture: Architect controls product and Architect also Implements
Architect controls Product basically says that you should have an architect and that she should oversee that the direction of the project is flowing in the right direction.

Architect also implements - this pattern says that in order for the architect to broaden her leadership without sacrificing depth and pragmatics she must also participate in the implementation (beyond advising and communicating). Jim gave the example of the development of Borland's Quatro pro for windows in 1993 where the team's architect had a daily meeting (akin to scrum stand-ups) for synchronization and would then go and code with the developers. The Quatro pro team had 4 architects out of 12 persons that made the team.If a third of the development team is architect I'd say he is right -  My experience, however with most organizations I see is that you hardly have one architect per  project (sometimes you only have one for several projects). In these cases I hardly see the architect writing production code as part of the team since she would not have time to fulfill her architectural responsibilities. She must know how to code though and she must be able to prove her designs in code or be able to offer a candidate implementation if needed (I also wrote about that in the past see "Should architect's code" part 1, part 2, part 3

By the way if you are located in Israel, Jim will be here for a couple of weeks and he is giving a few courses like Agile Architecture, Patterns of Agile Project Management etc. You can find more information on pacificsoft's site


 
Tags: Agile | Everything | General | Software Architecture