November 15, 2009
@ 12:07 PM

Jesse Ezell  left the following comment on my previous WCF rant (Windows Trick-or-treat Foundation)

You wouldn't expect WCF to take care every TCP/IP registry setting as well would you? At some point, the things WCF exposes have to stop so transport specific settings come into play. What would really suck is if WCF completely abstracted every detail of every transport and came up with new names for things like cookies and specific http request and response headers in the name of creating a "unified" experience across every transport.
The point of WCF is to give you a unified communication API. You don't have to change your code to switch from HTTP, to MSMQ, to TCP, etc. However, the point of WCF is not to hide all the specifics of each transport from you. There are a lot of transport specific options in WCF that you won't find in a WSDL file. However... none of these settings actually require you to change any code to take advantage of them. IMO, WCF does a really good job of providing a uniform communication API and limiting transport specific details to configuration settings. Even in this case, you were able to resolve the issue by configuration settings rather than code changes.
If you had used http request manually, this is one of very few settings you would actually have control over via configuration. What about other popular competing APIs? Look at something like NServiceBus. NServiceBus has a transport model that abstracts communication to some degree... but what happens when you want to change the format of the messages on the wire? For example, maybe you want to switch from raw XML messages to messages with SOAP envelopes on an endpoint or limit the depth of XML node hierarchies to protect against XML attacks. Maybe you want to switch to a completely new transport that was provided by a vendor that has never seen your product. Maybe you want to add compression, or chunking, or certificate based security. Can you do that all via configuration files without ever touching code in any other .NET API? Maybe this is my own ignorance... but I don't know of any .NET communication API that offers even half of the flexibility of WCF.

Well, I would actually expect WCF to do one of two things either provide a complete API of all the communications need (next version of .NET communication , unified communication model and all that..) and retire the other .NET communication libraries or on the other hand provide a thin layer of abstraction that will make it clear you need to move to the specific underlying protocols.

What we’ve got now is something that isn’t quite there on the first and way away from the second – which means that when you try to do serious stuff with WCF you hit these unexpected (ok now they are) snags where you don’t know where to go – until you realize that this is a specific thing regarding TCP this or HTTP that which is not readily apparent and is not well documented or even worse you need to set it outside of WCF altogether.

In my experience , you can’t in fact, “just change the binding” and expect everything to work unless you are doing very simple stuff. For instance when  if you move from HTTP binding to TCP you’d find that the channels are suddenly getting closed after periods of inactivity and you need to “keep them alive” or if you move in the other direction from TCP to HTTP you’d find that the size of messages gets larger by an order of magnitude  etc.

Not to mention the  “training wheels” approach to setting defaults (at least some of it is fixed for .NET 4) which I talked about a few times in the past. Also there cryptic error messages that make you scratch your head looking for the configuration item you need to set. for instance if your send a large message (>8K) you won’t see any problem in the sending side but you’d get the following

“An exception of type 'System.ServiceModel.Dispatcher.NetDispatcherFaultException' occurred in mscorlib.dll but was not handled in user code

Additional information: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://tempuri.org/:ClientPrintResult. The InnerException message was 'There was an error deserializing the object of type System.String. The maximum string content length quota (8192) has been exceeded while reading XML data. This quota may be increased by changing the MaxStringContentLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 2, position 40523.'.  Please see InnerException for more details.”

I am sure all of you immediately understood you need to set  ReaderQuotas, MaximumReceiveMessageSize and MaxBuferSize on the binding e.g.:

var binding = new WebHttpBinding()
        {
            ReaderQuotas = { MaxArrayLength = 20 * 8192},
            MaxReceivedMessageSize = 20 * 8192,
            MaxBufferSize = 20 * 8192

        };

Don’t get me wrong, WCF isn’t all bad or anything like that but it does get annoying like hell at times.

As for the reference to other frameworks - I can’t speak for NServiceBus  because I didn’t write it (though I think the answer would be the same) - but if I consider the communication framework I did write for xsights (which builds on WCF by the way) it is not as presumptuous as WCF. It does not pretend to be an all-encompassing communication layer for .NET and  It is build to provide a specific architectural approach – which is why it shouldn’t be judged by the same standards.


 
Tags: .NET | Software Architecture | WCF

October 30, 2009
@ 10:58 PM

Yes, this is another WCF rant…

We were getting ready to launch an open-for-all version of our service, we were also adding more cores to the system, to make sure our computation engine will be able to handle higher request loads (We’re basically implementing the Gridable Service pattern -  if it is interesting, I can expand on that in another post). We tried a few load tests, which on first look, seemed OK. However we then noticed that we get WCF timeouts on some of the calls.

WCF timeouts!? what gives? we already took care of all the annoying throttling defaults. After busting my head (and Google) for a few hours, I found that apparently, when using http bindings in WCF you only get 2 outgoing channels for each server (IP) you are connecting to. yep WCF thinks your services are novice users using a  browser. So if you have a service that tries to access a remote server with more than two requests simultaneously (say, under load…) you may find, like us, that you’d get occasional timeouts.

Yes, there’s also a solution, which also took time to find – it is called ConnectionManagment element in the network configuration (I am yet to find a programmable way to do this). So now the services app.config (see below) sets the value to 16 instead of 2 and everything is well again, at least until the next default hits us..

<system.net>
   <connectionManagement>
     <add address = "*" maxconnection = "16" />
   </connectionManagement>
 </system.net>

 

From trying to work with WCF in the last few years, it seems its abstractions are very leaky.They are almost leaky  to the point  of rendering it useless as a “unified” framework. So, in the spirit of the times, and as the title suggest – maybe they should just rename WCF to Windows Trick-or-treat Foundation – Alas, we are getting more and more of the “trick’ part rather that the “treat”, but at least the acronym fits well.


 
Tags: .NET | SOA Patterns | WCF

This is another post (<Rant>) about WCF default behavior and how it can make the life of developers miserable ( you can also check out “WCF defaults limit scalability”  and “Another WCF gotcha - calling another service/resource within a call”)

Anyway, the trigger for this is a post by Ayende called “WCF works in mysterious ways”.  Ayende posted some code he wrote which was throwing a serialization exception. You can see his post for the full code, but in a nut shell he was defining a large object graph (8192 objects that contain other objects) and was trying to send that over the wire. Here’s a short excerpt from the service definition:

   1: [ServiceBehavior(
   2:        InstanceContextMode = InstanceContextMode.Single,
   3:        ConcurrencyMode = ConcurrencyMode.Single,
   4:        MaxItemsInObjectGraph = Int32.MaxValue
   5:        )]
   6:    public class DistributedHashTableMaster : IDistributedHashTableMaster
   7:    {
   8:        private readonly Segment[] segments;
   9:  
  10:        public DistributedHashTableMaster(NodeEndpoint endpoint)
  11:        {
  12:            segments = Enumerable.Range(0, 8192).Select(i =>
  13:                                                        new Segment
  14:                                                        {
  15:                                                            AssignedEndpoint = endpoint,
  16:                                                            Index = i
  17:                                                        }).ToArray();
  18:        }
  19:  
  20:        public Segment[] Join()
  21:        {
  22:            return segments;
  23:        }
  24:    }
  25:  
  26:    [ServiceContract]
  27:    public interface IDistributedHashTableMaster
  28:    {
  29:        [OperationContract]
  30:        Segment[] Join();
  31:    }
  32:  
  33:    public class NodeEndpoint
  34:    {
  35:        public string Sync { get; set; }
  36:        public string Async { get; set; }
  37:    }
  38:  
  39:    public class Segment
  40:    {
  41:        public Guid Version { get; set; }
  42:  
  43:        public int Index { get; set; }
  44:        public NodeEndpoint AssignedEndpoint { get; set; }
  45:        public NodeEndpoint InProcessOfMovingToEndpoint { get; set; }
  46:  
  47:        public int WcfHatesMeAndMakeMeSad { get; set; }
  48:    }

As you can see in line 4 – the service is properly decorated with an attribute to enlarge the number of objects in graph. so looking at the code I initially suggested he add a few ServiceKnowType and DataContract/DataMember attributes on the data classes (as the serialization sometimes needs some guidance. After that didn’t help I actually ran the code and then I noticed that the code was missing setting that same attribute – on the client side. So to fix the problem, the client side code below

   1: var channel =
   2:     new ChannelFactory<IDistributedHashTableMaster>(binding, new EndpointAddress(uri))
   3:         .CreateChannel();
   4: channel.Join();
Need to change to something like
   1: var channelFactory =
   2: new ChannelFactory(binding, new EndpointAddress(uri));
   3:  
   4: foreach (var operationDescription in channelFactory.Endpoint.Contract.Operations)
   5: {
   6:  
   7: var dataContractBehavior =
   8:  
   9: operationDescription.Behaviors[typeof(DataContractSerializerOperationBehavior)]
  10:  
  11: as DataContractSerializerOperationBehavior;
  12:  
  13: if (dataContractBehavior != null)
  14: {
  15:  
  16: dataContractBehavior.MaxItemsInObjectGraph = int.MaxValue;
  17:  
  18: }
  19:  
  20: }
  21: var channel=channelFactory.CreateChannel();
  22: channel.Join();

The main problem I find with this piece of code is the fact that it is needed at all. As the post’s title suggest I find this behavior greatly affects the loose coupling of anything that uses WCF (services or other components).

WCF requires that any change you make to the channel on the server side would be reflected in the channel on each and every client (e.g. we have a similar setting where we enlarge message sizes for webHttpBinding and there are many other such examples).

Sure, you say, that is just like adding a new field in the contract isn’t it? – Well no it isn’t since unlike anything else which appears in the (verbose as it is) SOAP contract these changes in default values, which are purely a WCF design choice, are not documented. Again, the changes in default values are not part of the contract. These are things you need to remember to pass on to you service consumer. So not only do I pay the overhead of having an explicit contract (e.g. vs. REST) – it really doesn’t work.  It means that two components who use the same contract may not  be interchangeable if one returns more data (in this case). It means that the two sides are coupled by the need to change these defaults and for what? WCF is smart enough to know how long is the message; WCF is smart enough to handle the message (if I encourage it by setting a behavior) why can’t it add 2 and 2 by itself?

Sometimes I just wish WCF had a TrainingWheels or DemosOnly attribute I could just set to false and make all this crap go away…

</Rant>


 
Tags: .NET | SOA | WCF

the previous installment provided some context as to why I want to implement this pattern. This
installment will look at some of the implementation options.

As I noted before, WCF provides quite a lot of extension points on the route the message pass from arriving on the service to the point WCF
calls the actual method in the service instance. Several of those extension points are possible candidates for the Service Firewall for instance

  • Contract Filter-The contract filter is responsible to route messages to the appropriate contract. It needs to be a subclass of a MessageFilter. It looks that the contract filter is a good option since it intercepts the call rather early so it means it would probably be the fastest option. Also its name (filter..) implies it is a good option
  • Message Inspector - The Message inspector is responsible for looking at or modifying messages when they enter a service and looks like a natural candidate for the job. There are two kinds of Message Inspectors: Those who look at messages on the client side (implement the IClientMessageInspector interface) and those that look at the server side (implement the IDispatchMessageInspector). It seems that the latter is the type of inspector we need here.
  • Service Authorization Manager - responsible for evaluating policies, claims etc. of the client to make sure that a call is valid from the security perspective. This looks like it would be a good class to use for a real service firewall. It seems it won't be a good fit for the purpose of what we need here.

When I need to choose  between several technical options that seem to be similar I usually do a POC - proof of concept.  A piece of throwaway code to get a feel of the different options and better understand their strengths and weaknesses (in the context of the solution I seek).

What I did was to take a class I prepared for some of the integration tests of the EventBroker and build a few extensions that interact with them. Here is some of the setup code of the environment:

testServer = new Tester();
 service1 = new ServiceHost(testServer, new Uri(string.Format("http://localhost:{0}", TestServerPort)));
 var binding = new WebHttpBinding
 {
     ReaderQuotas = { MaxArrayLength = 600000 },
     MaxReceivedMessageSize = 800000,
     MaxBufferSize = 800000

 };

 var ep = service1.AddServiceEndpoint(typeof(TestingContract), binding, string.Format("http://localhost:{0}/S1", TestServerPort));
 ep.Behaviors.Add(new WebHttpBehavior());
 ep.Behaviors.Add(new InspectorBehavior());
 service1.Authorization.ServiceAuthorizationManager = new TestAuthorizer();
 var cp = service1.AddServiceEndpoint(typeof(ImContract), binding, string.Format("http://localhost:{0}/Control", TestServerPort));
 cp.Behaviors.Add(new WebHttpBehavior());

The two redlines above are the ones responsible for injecting the POCs the InspectorBehavior is reponsible for inserting the ContractFilter and the MessageInspector and the TestAuthorizer is the Authorization Manager test implementation.

We also need some code to raise an event:

public void SendMessage()
 {
     var evnt = new TestingEvent { sagaId = Guid.NewGuid() };

     moqRA.Expect(x => x.GetChannel<TestingContract>(evnt.sagaId, true)).Returns(channel1);
     moqRA.Expect(x => x.GetChannel<TestingContract2>(evnt.sagaId, true)).Returns(channel3);

     eb.BeginNewSagaEvent(evnt.sagaId, evnt);
     eb.CloseSaga(evnt.sagaId);

 }

And now we can look at the different options. The InspectorBehavior is just a helper class to wite the filter and/or inspector to the endpont. (The Authorization Manager is setup at the service level (i.e. for all endpoints))

public class InspectorBehavior : IEndpointBehavior
 {
     
     public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
     {
     }

     public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
     {
         throw new NotImplementedException();
     }

     public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
     {
      var inspector = new TestInspector();
      endpointDispatcher.DispatchRuntime.MessageInspectors.Add(inspector);
      endpointDispatcher.ContractFilter = new TestFilter(endpointDispatcher.ContractFilter);
         
     }

     public void Validate(ServiceEndpoint endpoint)
     {
     }

The first thing I tried was the "ContractFilter". It is actually very simple to use. You inherit from MessageFilter and there are two "Match" method you need to override. One that accepts a buffer and one that accepts a (WCF) Message. WCF calls the Match method which accepts a Message.

WCF's Message class is interesting in the sense that it has a one-time touch feature. i.e. only one piece of code can read/copy it and the next piece of code which will try to do the same will fail

So the match method you can do something like the following:

public override bool Match(Message message)
 {
     var buffer = message.CreateBufferedCopy(Int32.MaxValue);
     message = buffer.CreateMessage();
     var r = buffer.CreateMessage().GetReaderAtBodyContents();
     .
     .
     .
 }

Which basically means get a buffer of the message, create one copy to preserve and the get another copy for internal use and work with that to parse and verify the actual message. Unfortunetly, this doesn't really work - the message parameter is not passed as ref so the original message is lost on the first line of the method and that's it. Note that you can access the header part of the message without problem, however that's not a good fit for what I am trying to do.

The next thing I looked at the MessageInspector. Again implementing it is rather simple, you just need to implement the IDispatchMessageInspector interface. This interface has two methods BeforeSendReply and AfterReceiveRequest. We'll look at the AfterReceiveRequest method. Again we try the message copy trick:

public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
 {
     var buffer = request.CreateBufferedCopy(Int32.MaxValue);
     request = buffer.CreateMessage();
     var temp = buffer.CreateMessage().GetReaderAtBodyContents();
     .
     .
     .
 }

This time it works since we get the request parameter as ref. At first it seemed to me that while you can inspect and alter the message as your heart wishes there is no way to say that the message is bad. One option is to alter the message to a faulty message and let the application handle it - but that means too much coupling between infrastructure and application. Another, better, option is to throw an exception.

So using the MessageInspector is a usable option. It is very good if you want to alter the incoming message but throwing an exception when the message is bad is not very clean

Which brings us to our third option Authorization Manager which, surprisingly  turned out to be the best option

public class TestAuthorizer :ServiceAuthorizationManager
 {
     public override bool CheckAccess(OperationContext operationContext, ref Message message)
     {
         var autorized= base.CheckAccess(operationContext, ref message);
         var buffer = message.CreateBufferedCopy(Int32.MaxValue);
         message = buffer.CreateMessage();
         var testMessage = buffer.CreateMessage();
         .
         .
         .
         return autorized;
     }
     
 }

Like the message inspector it receives the message as ref and like the filter it allows a single yes/no answer to decide if a message should continue or be discarded. Additionally it notifies the client that the message was rejected if that is what you choose to do (in the WebHttpBinding I used that means a 400 bad request return code)

Ok, so we've seen some of the options for implementing the Service Firewall and briefly went over thier different behaviors. The next part in this series will take a look at some of the actual implementation I did



 
Tags: .NET | SOA | SOA Patterns | WCF

One of the SOA patterns I already described is the Service Firewall. The idea behind the service firewall is to have an intermidiator between the actual service and callers and inspect in an applicative level incoming and outgoing messages.


Anyway, while I documented the pattern as a security one, I am actually going to implement it for another purpose - a saga filter.
In our implementation of EventBroker I made the design decision to have services expose regular WCF contract. i.e. services can communicate with each other directly and not just via eventing. This design decision is there to allow both interaction with non WCF services and to allow flexibility for multiple message exchange pattern (where events are not the best choice).
Another design decision we have is that we have two types of services. Servers and Channels. Servers handle multiple sessions and are (relatively) heavy to write. Channels on the other hand are light-weight services that  are stateful and dedicated for a specific session. Naturally there are a lot of instances of channels to allow supporting multiple sessions (and there are infrastructure bits to allow allocations and propagate liveliness etc. but that's another story). Channels have several benefits like increasing the systems capabilities to cope with failure (if a channel is down only the session it supported fails). One of the benefits of Channels is simple coding model. The Channel is dedicated to a session (typically a saga) and thus it doesn't have to handle all the routing of messages to sagas etc. that Servers have to cope with. This is where the Service Firewall comes to play.
In order to keep channels' code simple "someone" has to make sure the channel doesn't get messages that are not related to the saga it is part of. Otherwise the Channel will have to know about its current active saga and filter messages by itself - which kind of misses the point.
Making sure other services will not send messages while not in saga etc. will only take us so far (you know - latencies and stuff). A service firewall will let us intercept the messages before they reach the service and only allow the messages related to an active saga to pass through (while maintaining the benefits of direct contracts)

WCF has a rich extensibility model (see figure from MSDN below). This series will show how you can use some of these extension points to implement a service firewall and achieve the goal depicted above.  I hope you'd find it interesting




 
Tags: .NET | SOA | SOA Patterns | WCF

January 5, 2009
@ 08:02 PM
We are going to use some of our test code in production. Yes you read it right test code in production. Here are the details
In our system, among other things, we support visual search in video calls. i.e. an end user calls the system, points the camera at something she is interested, and (hopefully :) ) gets relevant information. Basically the system is made of several resources (image extraction, identification etc.) that collaborate via an event broker. We have a blogjecting watchdog that makes sure everything is up and running and we have applicative recovery service to handle failures.
The watchdog makes sure resources/services are up, resources report their liveliness and wellness so we know more about the resources than the fact that they are up. However, we still need a way to make sure that resource instances  can collaborate to provide the service.

Enter our automated acceptance tests. Part of our development effort included building a test runner for automated tests scenarios, e.g. load tests, verifying algorithms correctness etc. One of these tests is the smoke test (run after each successful build) which includes a sunny-day scenario of a video call- as described above. What we're going to do now is build on the test runner and the sunny day scenario a "keep-alive" tester that will periodically make test calls to the system (depending on the current load etc.) and make sure that everything is still working correctly.


So there you have it, an unexpected benefit of automated acceptance tests, who would have thunk it :)



 
Tags: .NET | SOA | Software Architecture | TDD | WCF

October 18, 2008
@ 10:58 PM
As I mentioned in a couple of previous posts (like "Using REST along with other architectural ), I've been spending the last few weeks writing an Event system over WCF (probably also explains posts on  WCF gotchas like this;) ). Being a communication infrastructure it is still a long way from being completed, but it seems to be stabilizing and I think it turned out nicely so I thought I'd share a few details.

Let's start with the simple part - the usage.
The eventing is built on the idea of a bus (i.e. no centralized components) and the resources/services that want to use eventing have to use a library which I call EventBroker.  There are two modes for using the EventBroker. one is "regular" events which are contexless. This means that consecutive events can reach different services, and there is no context that flows from event to event:

bool raisedEvent = eb.RaiseEvent<SampleEvent>(new SampleEvent());
The second type of events are Sagas, which represent long running interactions. Sagas does have a "best effort" guarantee to reach the same recipients over consecutive calls. Also you can also End sagas (sucessful termination), Force End Saga (successful termination by a service that didn't initiate the saga) and Abot Saga (unsuccessful termination): Here is how you raise a saga event.
var evnt = new SampleEvent { data = somevalue};
var SagaId = Guid.NewGuid();
eb.RaiseSagaEvent<SampleEvent>(SagaId, evnt);
if you use the same Saga Id, the events are handled as part of the same saga, if you use a Saga Id that wasn't previously defined it will initialize a new saga.
The eventbroker translates events to the relevant contract and dispatches the events over to the different subscribers. Which brings us to to the next part which I  guess,   is also a little more interesting. How subscriptions are defined.

The first thing to do is to define the event itself.
    public class SampleClassEvent :ImEvent
{
public string DataMember1 {set;get;}
public int DataMember2 { set; get;}
}
There aren't any real constraints on the event, except that it has to "implement" the ImEvent interface. Which is really an empty interface but it marks the event as one for the event broker.
Then you have to define an interface for handling the event. The event broker, builds on the idea of convention rather than configuration (an idea popularized by the rails framework) so it is easier to generate the interface (something I do with a resharper template)
    [ServiceContract]
public interface IHandleSampleClass
{
[OperationContract]
int SampleClass(SampleClassEvent eventOccured);

}
The convention is that the interface will have a IHandle prefix followed by the name of the event. It will hold a single operation named like the event (without the Event suffix) and will recieve a single parameter which is the event data. Currently  events do return a value (int) but I am thinking about changing it to void and have everything marked as OneWay for added performance

Now, when we create a service which needs to handle events it will do that by specifing which events it handles. E.g.
    [ServiceContract]
public interface ImSampelResource : ImContract, IHandleSampleClass, IHandleSomeOtherThing
 {
}
So each contract declares all its subscriptions (by a list of IHandleXXX). It should also include the ImContract interface which holds all the service operation used by the eventbroker (e.g. ending sagas etc.).
Services that want to raise events should inherit from a ControlEdge class (base class Edge component that delegates control events to the event broker)

There's still the question of how does the event broker knows where to find other services. There are several ways this can be done (e.g. a service repository) but since we have  blogjecting watchdogs in place anyway, we use them to propagate liveliness (and location ) of services.

This sums up this post. It is basically just a little context for several planned posts where I hope to talk about some of the challenges, alternatives and design decisions that led me to the current design. Meanwhile, I'd also be happy to hear any comments, ideas or reactions you may have
 
Tags: .NET | Design | OO | SOA | SOA Patterns | Software Architecture | WCF | xsights

I wrote in the past how WCF defaults limit scalability but this thing (which had cost me two days of head scratching) is even worse.
Consider the following scenario:
 You have a WCF service/resource. when you get a message/request your codes needs to send another message to another service.
Sounds common enough now doesn't it? and it is - unless you happen to use a service with   WebHttpBinding (e.g. if you try to develop a RESTful WCF service or want to use POX services).  When you use WebHttpBinding and try to make a call within a call you are likely to find yourself starring at a ProtocolException with a 405 error - Method not allowed. Turns out WCF finds itself confused by the Operation Context (OperationalContextScope) of the incoming request so if you want things to work properly you need to create a new one for the request
var webBinding = new WebHttpBinding();
var channel = new ChannelFactory(webBinding, controlUri);
channel.Endpoint.Behaviors.Add(new WebHttpBehavior());
var proxy = channel.CreateChannel();
using( new OperationContextScope((IContextChannel) proxy))
{
proxy.Dostuff()
}

I already spent the time figuring this bugger out- I hope this post will save you the trouble


 
Tags: .NET | REST | WCF