I decided to do another article collection post. As I mentioned in the last post that I am extremely busy these days  as we're preparing to launch the first public pilot of our service. On top of that I cannot disregard the ROI (views vs. hours invested) in previous collection post I made on "10 papers architect should read". It isn't that I don't have other articles and papers with 40K+ reads but compiling a list takes much less time than, say, writing an article on what is SOA, explaining the fallacies of distributed computing, a mini-book on Use cases - not to mention the effort that goes into writing  SOA patterns/anti patterns book (Saga, Edge Component, The Knot etc.).
But enough about me...

  • The mythical man-month (1975) Fred Brooks - it is the second chapter of a book with the same name. A classic paper where, I think, for the first time, the ideas that some tasks need to be done sequentially and can not be solved by adding more man-power or in Fred’s words “The bearing of a child takes nine months, no matter how many women are assigned”. It is also famous for stating that adding more people to a late project will only delay it further.
  • Why the Vasa Sank: 10 problems and some antidotes for software projects (2003) by Richard E. Fairley and Mary J. Wilshire -it isn't free but worth it (though you can read the gist of it for free in "Why the Vasa Sank: 10 Lessons learned" by Richard E. Fairley) – The tale of the Vasa ship is a very nice analogy for developing software project. I first heard it from Ivar Jacobson several years ago. This article presents it very well.
  • The Cathedral and the Bazaar (1997) Eric S. Raymond - the classic essay on open source vs. closed source models for developing software
  • Software Maintenance is a Solution, Not a Problem - Robert L. Glass – it really is when you think about it.
  • IT doesn't Matter (2003) - Nicolas Carr – A thought provoking article on the place and importance  of IT – Carr basically states that software is getting to a point where it will become a utility like electricity and in that will loose it strategic place. In a sense we’re seeing more of this direction today with all the hype and move to cloud computing.
  • Chapter 3 of the "Report of the Defense Science Board Task Form on Defense Software" (2000) – various – An extensive analysis of software intensive projects. Besides revealing interesting facts on the ever growing importance of software (eg. 80% of the functions of F-22 are performed in software vs. 8% in the 1960’s F-4) it also acknowledge the advantages of iterative lifecycle over waterfall even in large, safety critical,complex  projects. 
  • Observations on Balancing Discipline and Agility (2003) Barry Boehm and Richard Turner – provide a few rules of thumb for assessing a project’s suitability for agile methods (e.g. team’s level, criticality etc.)
  • Succeeding with "Agile Fixed Price" part I & part II Pascal Van Cauwenberghe  - a two part paper with practical advice to employing/injecting agile methods into fixed price contracts.

 
Tags: General | Papers

March 18, 2009
@ 07:35 PM

Image via Wikipedia
You might have noticed my blogging is slowing down a little,  in case you're wondering the culprit is xsights - As we're getting closer to
production (2-3 weeks if all goes well) everything else slows down...

We all know about continuous integration (CI). We (xsights) are seeing a lot of ROI on the investments we made into signing up to CI esp. when we augmented by automated testing. Everyday we can and do actually release several versions and run load tests on them on different server seeking the best release to  "freeze". Where "freezing" means the  system that will be used in production.  That's the way it is done, right? You work on a release for x months (sometimes years..) and after an integration period at the client's site you go live and forget about them (save for show-stopping bugs) until the next major update several months later. For instance, setting up our "methodology" I figured a 6 months release cycles for major versions plus 1 month releases of minor version would suffice.

These days, however the more I think about it, The more I believe this approach is not the right one for our situation. We are going to go with Continous Deployment. There are many reasons for that  primarily:
  • The requirements stream (new reqs and changes) is still pouring in - and in increased rate.
  • We are building a service platform not an installable product -  This means we get to control where how it is run so there's less overhead in
    streamlining updates
  • It would be our first public installation - were going to roll out bug fixes anyway (I personally never write bugs... but you know ;) ). We'll need a proper procedure for updates anyway
  • We are doing this anyway - every new demo/internal run we have is using the "stablest" latest and greatest :)
What does  Continuous Deployment means? It means we'll be rolling out new production versions on a daily basis (or even more frequently!)
How that's going to work? Well, First there are the prerequisites:

  • Version control - something that can track and tag versions...
  • Continous integration which is set up and running. You cannot release continously if every new development offer breaks the system. An automated smoke test is
  • Automates tests that provide a good approximation of real usage
    (.e.g. usage patterns, loads etc.) - You need to be able to have a high
    confidence level that whatever you have.
  • Staging environment - An environment as close as possible to the production system (just like you'd have for a web-site)
  • Monitoring and alert system - You need to be able to identify problems quickly - the running

Then there's the process:
  1. identify a baseline Stable release -The "fallback".
  2. pick the most promising build of the day
  3. run it through more extensive tests (at least burn-down and load - which means a release maybe 1-2 days old by the time it hits the production system)
  4. if everyhting is cool deploy to staging - else fix bugs and repeat from 2
  5. repeat tests in staging environment
  6. Update system (This is a point that still needs work so we can support a version hot-swap i.e. without a shutdown)..
  7. If the monitoring system alerts unforeseen bugs (i.e. problems with the release) - roll back to stable release, try to add test for the problematic behavior and repeat from 2
  8. if the release worked good enough - mark it as the next Stable release

That's the plan. However as we all know, talk is cheap so be sure to check in here in 2-3 months to hear how did we fare :)

Reblog this post [with Zemanta]



 
Tags: Agile | Project Management | xsights

the previous installment provided some context as to why I want to implement this pattern. This
installment will look at some of the implementation options.

As I noted before, WCF provides quite a lot of extension points on the route the message pass from arriving on the service to the point WCF
calls the actual method in the service instance. Several of those extension points are possible candidates for the Service Firewall for instance

  • Contract Filter-The contract filter is responsible to route messages to the appropriate contract. It needs to be a subclass of a MessageFilter. It looks that the contract filter is a good option since it intercepts the call rather early so it means it would probably be the fastest option. Also its name (filter..) implies it is a good option
  • Message Inspector - The Message inspector is responsible for looking at or modifying messages when they enter a service and looks like a natural candidate for the job. There are two kinds of Message Inspectors: Those who look at messages on the client side (implement the IClientMessageInspector interface) and those that look at the server side (implement the IDispatchMessageInspector). It seems that the latter is the type of inspector we need here.
  • Service Authorization Manager - responsible for evaluating policies, claims etc. of the client to make sure that a call is valid from the security perspective. This looks like it would be a good class to use for a real service firewall. It seems it won't be a good fit for the purpose of what we need here.

When I need to choose  between several technical options that seem to be similar I usually do a POC - proof of concept.  A piece of throwaway code to get a feel of the different options and better understand their strengths and weaknesses (in the context of the solution I seek).

What I did was to take a class I prepared for some of the integration tests of the EventBroker and build a few extensions that interact with them. Here is some of the setup code of the environment:

testServer = new Tester();
 service1 = new ServiceHost(testServer, new Uri(string.Format("http://localhost:{0}", TestServerPort)));
 var binding = new WebHttpBinding
 {
     ReaderQuotas = { MaxArrayLength = 600000 },
     MaxReceivedMessageSize = 800000,
     MaxBufferSize = 800000

 };

 var ep = service1.AddServiceEndpoint(typeof(TestingContract), binding, string.Format("http://localhost:{0}/S1", TestServerPort));
 ep.Behaviors.Add(new WebHttpBehavior());
 ep.Behaviors.Add(new InspectorBehavior());
 service1.Authorization.ServiceAuthorizationManager = new TestAuthorizer();
 var cp = service1.AddServiceEndpoint(typeof(ImContract), binding, string.Format("http://localhost:{0}/Control", TestServerPort));
 cp.Behaviors.Add(new WebHttpBehavior());

The two redlines above are the ones responsible for injecting the POCs the InspectorBehavior is reponsible for inserting the ContractFilter and the MessageInspector and the TestAuthorizer is the Authorization Manager test implementation.

We also need some code to raise an event:

public void SendMessage()
 {
     var evnt = new TestingEvent { sagaId = Guid.NewGuid() };

     moqRA.Expect(x => x.GetChannel<TestingContract>(evnt.sagaId, true)).Returns(channel1);
     moqRA.Expect(x => x.GetChannel<TestingContract2>(evnt.sagaId, true)).Returns(channel3);

     eb.BeginNewSagaEvent(evnt.sagaId, evnt);
     eb.CloseSaga(evnt.sagaId);

 }

And now we can look at the different options. The InspectorBehavior is just a helper class to wite the filter and/or inspector to the endpont. (The Authorization Manager is setup at the service level (i.e. for all endpoints))

public class InspectorBehavior : IEndpointBehavior
 {
     
     public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
     {
     }

     public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
     {
         throw new NotImplementedException();
     }

     public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
     {
      var inspector = new TestInspector();
      endpointDispatcher.DispatchRuntime.MessageInspectors.Add(inspector);
      endpointDispatcher.ContractFilter = new TestFilter(endpointDispatcher.ContractFilter);
         
     }

     public void Validate(ServiceEndpoint endpoint)
     {
     }

The first thing I tried was the "ContractFilter". It is actually very simple to use. You inherit from MessageFilter and there are two "Match" method you need to override. One that accepts a buffer and one that accepts a (WCF) Message. WCF calls the Match method which accepts a Message.

WCF's Message class is interesting in the sense that it has a one-time touch feature. i.e. only one piece of code can read/copy it and the next piece of code which will try to do the same will fail

So the match method you can do something like the following:

public override bool Match(Message message)
 {
     var buffer = message.CreateBufferedCopy(Int32.MaxValue);
     message = buffer.CreateMessage();
     var r = buffer.CreateMessage().GetReaderAtBodyContents();
     .
     .
     .
 }

Which basically means get a buffer of the message, create one copy to preserve and the get another copy for internal use and work with that to parse and verify the actual message. Unfortunetly, this doesn't really work - the message parameter is not passed as ref so the original message is lost on the first line of the method and that's it. Note that you can access the header part of the message without problem, however that's not a good fit for what I am trying to do.

The next thing I looked at the MessageInspector. Again implementing it is rather simple, you just need to implement the IDispatchMessageInspector interface. This interface has two methods BeforeSendReply and AfterReceiveRequest. We'll look at the AfterReceiveRequest method. Again we try the message copy trick:

public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
 {
     var buffer = request.CreateBufferedCopy(Int32.MaxValue);
     request = buffer.CreateMessage();
     var temp = buffer.CreateMessage().GetReaderAtBodyContents();
     .
     .
     .
 }

This time it works since we get the request parameter as ref. At first it seemed to me that while you can inspect and alter the message as your heart wishes there is no way to say that the message is bad. One option is to alter the message to a faulty message and let the application handle it - but that means too much coupling between infrastructure and application. Another, better, option is to throw an exception.

So using the MessageInspector is a usable option. It is very good if you want to alter the incoming message but throwing an exception when the message is bad is not very clean

Which brings us to our third option Authorization Manager which, surprisingly  turned out to be the best option

public class TestAuthorizer :ServiceAuthorizationManager
 {
     public override bool CheckAccess(OperationContext operationContext, ref Message message)
     {
         var autorized= base.CheckAccess(operationContext, ref message);
         var buffer = message.CreateBufferedCopy(Int32.MaxValue);
         message = buffer.CreateMessage();
         var testMessage = buffer.CreateMessage();
         .
         .
         .
         return autorized;
     }
     
 }

Like the message inspector it receives the message as ref and like the filter it allows a single yes/no answer to decide if a message should continue or be discarded. Additionally it notifies the client that the message was rejected if that is what you choose to do (in the WebHttpBinding I used that means a 400 bad request return code)

Ok, so we've seen some of the options for implementing the Service Firewall and briefly went over thier different behaviors. The next part in this series will take a look at some of the actual implementation I did



 
Tags: .NET | SOA | SOA Patterns | WCF