What kind of integration patters do you implement in BizTalk?

I like to divide BizTalk based integrations into three different patterns.

1. Point-to-point integration
2. Broker-based integration
3. ESB-based integration

I am sure someone could come up with fancier names but I will in this post try and dig into each of them, explain what I mean by them. I will also try and highlight some issues with each of the patterns.

Point-to-point integration

This is where a lot of us started. The idea is that the sending system has information and knowledge about the receiving system.

This usually means that the messages are exported in a message format that is tailored to the format of the particular receiving system’s needs. It also means that we usually get one integration process for each receiving system there is.

The example in the figure on the right shows a sending system that sends information about invoices to two different receiving systems (“System A” and “System B”). Using a point-to-point pattern in this scenario we end up with two different types of messages that are exported. Each of the exported messages that are suited and tailored to the format of the receiving system.

Issues

The main problem with this kind of point-to-point integration, where the sending system has detailed knowledge about the receiving systems, is that this knowledge translates into a coupling between the systems. When we tie the export message format from the sending system to the format of the receiving, all changes in the receiving system will also cause changes in the sending system.

Suppose there is a sudden need to change the format of the receiving system – as we use that format as our export format we now also have to change it there.

Another problem is the lack of agility. In this invoice scenario all other system that also has the need of invoice based information has to get this by going all the way back to the sending system and develop a whole new specific integration – separate to the existing ones.

Broker-based integration

In the broker-based scenario the integration platform is used as a broker of messages.

This means that only one canonical format is exported from the sending system. The broker then handles the routing to the different systems and the transformation to the message formats that the receiving systems expects.

The main advantage between this approach – where the sending system do not know anything about the receiving systems – and the point-to-point pattern is agility.

If there now is a need for invoice information in a third, new system, we do not have to change the export from the sending system (as long as we have all the information need that is) or develop a whole new integration. All we have to do is to route the invoices so that they also are sent to the third system and transformed into the format that the system expects. A new BizTalk map and port and we are done!

Issues

In my point of view this approach to integration has a lot of advantages over the point-to-point integrations previously discussed. And in a lot of simpler, stabile scenarios it works just fine and is the way to go.

But in some scenarios it kind of breaks down and becomes hard to work with. The problems is related to configuration and how we define the routing information in BizTalk. In BizTalk we can either create an orchestration and “hardcode” the process that defines which systems the messages should be sent to. We can also create a “messaging” scenario where we configure this routing information in the different BizTalk port artifacts by setting up filters.

Regardless if we choose a “messaging” or orchestration based solution the routing information becomes hard to update as the solution grow in size. We either get very complicated orchestrations or loads of ports to maintain.

Furthermore the whole process is very coupled to the one canonical schema which makes versioning and updates hard. If the canonical schema needs to be updated and we still need to be able to send information using the old schema (so we have a “version 1” and “version 2” of the schema side-by-side) all artifacts needs to be more or less duplicated and the complexity of the solutions grows fast.

This makes business agility hard and any changes takes long time to develop and deploy correctly.

I guess these issues has more to do with how BizTalk works than the integration pattern itself – but this is a BizTalk blog!

ESB-based integration

ESB-based integration in BizTalk is today achieved using ESB Toolkit from Microsoft.

One of the ideas of ESB-based integration is that the integration platform should not have hardcoded routing information. These routing rules should either be looked up at run time or travel with the actual payload of the message (called “dynamic routing”).

This in combination with having generic on- and off-ramps instead of loads of separate ports to maintain promises to create more agile and configurable solutions (called “Loosely coupled service composition” and “Endpoint run-time discovery and virtualization”).

The figure again shows the invoice scenario used previously. In this case we export a “Invoice 1.0”-format from the sending system. The format is not directly bound to a schema in the port (as we are used to in BizTalk) but it is basically possible to send any message format to the on-ramp.

The on-ramp then identifies what kind of message it received (for example using the XML-namespace) and looks up the routing rules for that specific message (the message “itinerary”) in a repository.

In this scenario there could be rules to route the message to a orchestration or directly to a send port. All the configuration for this port is however configured as part of the itinerary and applied at run-time on the generic off-ramp port. And as itineraries are simple XML documents they are super light to update and deploy in the repository!

So if we now wanted to update the “Invoice”-format to “version 2.0” all we would have to do is to create a new XML itinerary for the “Invoice 2.0” message type and possibly create new orchestration to handle the new message type. No new ports, no new bindings etc, etc. The configuration and deployment would be a lot simpler than before! We would end up with a lot fewer artifacts to maintain.

And the itinerary for “Invoice 1.0” would still be applied to all incoming Invoice 1.0 messages. Thus we have achieved what used to be so hard with a lot less development power!

Issues

With agility and dynamic capabilities comes complexity in debugging when something bad happens … I also feel that we give up a lot of validating and control capabilities that we had in the more schema-coupled patterns previously discussed.

I am new to this ESB-based model and I would love to hear your experiences, problems, stories etc!

I am however sure that this is the way to go and I would not be surprised if ESB like patterns will play an ever bigger role in future BizTalk versions!

Does BizTalk have man-boobs?

I’ve just finished watching this webcast from QCon 2008 with Martin Fowler and Jim Webber. It’s basically their view on how integration middleware is used today and how we plan and implement our SOA and ESB project.

Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration – fair enough. I totally agree that trying to convert your complete enterprise architecture into a SOA architecture is a guaranteed failure. This is also something we heard for a while now from others as well.

I do also agree that BizTalk is a huge platform and that it isn’t perfect in all aspects. IMHO it does however give us some important advantages compared to a custom coded message bus and services. I’ll try and list a few of them below.

1. Fixed architecture
   We don’t have invent the wheel every time. BizTalk is a product with an architecture that one have to learn and use. There are times when this is a pain (did I hear low latency and BizTalk persistence points?) but it’s also a huge kick start to all projects once you learnt it. Once you figured out how you use the products you’ll actually have something up and running in no time.

   Isn’t an early delivery that we can test something good? I’m sure I can deliver a BizTalk based integration faster that some can using custom code when starting from scratch.
    

2. Drag-and-drop
   There is a learning curve to BizTalk and all it’s tools but once one gotten over this one can move really fast, even without a deep understanding of .NET and software development (there are of course both pros and cons to this). I’ve seen projects with 50+ integration processes (to me that’s a big, complex project) where we actually used people fresh out of school, spent two weeks to teach them basic BizTalk and had them deliver critical parts of the projects. I’d like to see that happen custom coded ESB project with thousands lines of code …

   You probably get a nice design and implementation if you can hire 10 top developers and a couple of architects, but that isn’t always possible.
    

3. Tools
   Does a custom code, lean approach, really scale in this scenario? Do you take the time to pause and build that management and configuration tool that you don’t get with a custom code project? I don’t say that we got the perfect view and control of our processes and messages in BizTalk but at least we got some control. At least I got the BizTalk Administration Console to let me see how my different application are doing, what messages and process etc that got suspended. At least I got the BAM framework where I can configure a tracking and monitoring in no time (usually …) etc, etc. 
    
4. It works
   Say your implementing a process that receives purchase orders and that these orders might contain orders for a couple of millions dollars. Do you want to be developer that tells you boss that you think you might have lost a message due to a exception in your custom code? Of course you have 90% test coverage and continuous integration but you never tested for this exception case … I don’t want to be that developer/architect.

I don’t say don’t test. I’m very pro testing and I really feel that agile is the right approach. I’m just saying I need something tested and safe to build this super critical solutions on. Something that I know works and that I can be really productive on and start solving business cases from minute one.

What do you think? Does BizTalk have man-bobs and is that only a bad thing? And does Martin Fowler really have leather pants on?

Speaking at KNUG – Karlstad .NET User Group

Yesterday I presented the MasterData Management using BizTalk 2006 R2 talk (I’ll soon have a post out with the presentation in English) I recently held at Developer Summit at the local .NET user group in Karlstad (KNUG).

KNUG is a new .NET user group that I actually helped start a couple of months ago. This meeting was the second meeting for the group. The meeting was attended by about 20 persons and we had two presentations on the agenda. Besides my own Thomas Heder showed the group some LINQ and how he and his colleagues uses LINQPad to develop and test there queries.

We also discussed future subjects, possible speakers and moving information on the group over to a Community Server driven site.

Does anyone have any experience on Community Server and how the feature set matches those need for running a user group (managing users, blogs, email lists, calendar etc)?

A loosely coupled Scatter-and-Gather implementation In BizTalk 2006

Why loose coupling?

There was a question in Microsoft BizTalk forums the other day about how one could implement a Scatter and Gather pattern in a more loosely coupled fashion.

Most examples on the implementation of this pattern in BizTalk use the Call Shape functionality available in BizTalk orchestrations. This however creates a hard coupling between the "Scatter orchestration" and it’s "partner orchestrations". The downside of that is that when one adds or removes a partner the whole solution has to be recompiled and redeployed.

If one however could use the publish-subscribe architecture in the MessageBox to route messages between the "Gather orchestration" and it’s partners, it’d be possible to add partners without having to worry about the rest of the solution. This post shows and example on how to implement a solution like that.

The BizTalk process in steps

NOTE: Notice the difference between the PartnersRequest, the PartnersResponse, the PartnerRequest and the PartnerResponse messages. The names are unfortunately very similar.      

The PartnersRequest and PartnersResponse messages are used for communication between the Scatter and the Gather orchestrations. It’s also a PartnersRequest message that activates the process.

PartnerResponse and PartnerRequest are used for communicating between the Scatter and the Gather orchestration and all the partner orchestrations.

1. Request and scatter
   A PartnersRequest message is received. This message is an empty message and is only used for activating the process in this example scenario. The PartnersRequest message is consumed by the Scatter orchestration. The Scatter orchestration creates one PartnerRequest message. The orchestration also generates a unique key called a RequestID and start a correlation combining that that id and the PartnersRequest MessageType. Finally it post the PartnerRequest message to message box, writes the generated RequestID to the request messages context (RequestID is a MessageContextPropertyBase based context property) and dehydrates itself.
   
2. Partners
   All the enlisted Partner orchestrations pick up the PartnerRequest message from the message box. These orchestration then communicates with their specific data source (could be a service, database, file or whatever), receives a response. Finally these orchestrations transform the response they received and creates a PartnerResponse message that’s posted to the message box. Notice that the RequestID that was generated by the Scatter orchestration is also part of the context of the newly created PartnerResponse message.
3. Gather
   The PartnerResponse messages are routed to the Gather orchestration. This orchestration uses a Singleton pattern based on the RequestID which all PartnerResponse messages carried with them in their context. This means that it’ll receive all the PartnerResponse messages containing that same RequestID into the same orchestration instance (ergo all the Partners that were activated by the request message being sent from one Scatterer). For each message instance it receives it add it’s price to a total price variable. When the Gather orchestration has received all the PartnerResponse messages (the orchestration knows how many Partners responses it should expect from one Scatterer orchestration and we can timeout if we don’t get all expected with a timeframe) the total price we calculated is written to a PartnersResponse message.
4. Response
   This message is routed back the Scatter orchestration by using the correlation it initialized in the start. It’s finally this orchestration that send the final outgoing message (a PartnersResponse message).

Example solution

An example of the implementation can be downloaded from here.

The solution contains five different schemas.

1. PartnersResponse
   Used for initializing the process.
2. PartnersRequest
   Send from the Gather orchestration to the Scatter orchestration. It’s also the final result and outgoing message from the process.
3. PartnerRequest
   Picked up and activates all enlisted Partner orchestrations.
4. PartnerRequest
   Send from the Partner orchestration containing the result from the Partner Service and send to the Gather orchestration.
5. LooselyCoupledScatterGatherExampleProperties
   Property schema for storing the RequestID and to correlate all the PartnerResponses as well as the final PartnersResponse back to the Scatter orchestration.

Five orchestrations

1. Scatterer orchestration
   The "main orchestration" that receives a request message from outside and "scatters" party requests to all the party orchestrations.
2. Gatherer orchestration
   Gatherer orchestration that gathers all the responses from the partners and transforms these to a reply that is being routed back to the Scatterer and back out.
3. Partner1, Partner2 and Partner3 orchestration
   Partner orchestrations that communicates to different services and receives price information.

Setting up and testing the example solution – it’s easy!

When the solution is built and deployed one needs to setup and bind two ports; one outgoing port and one incoming port (this could also be a Request-Response port by changing the port type in the Scatterer orchestration). That’s it!

Enlist and start everything by dropping a PartnersRequest test message (you’ll find one among the zipped files) in the incoming folder. A PartnersResposne message should then be published in the outgoing folder containing a calculated price from all the Partner orchestrations.

Test message and a binding file are part of the zipped solution.

What would be different in "real life" solution?

I’ve made some major simplifications in this example to make it easy for setting it up and test the concept. These would very different in a "real" solution.

Partner Services

The Partner orchestrations are very simple. They actually don’t communicate with outside world at all. All the do is setting a hard coded price and post a response. In a real solution these would not be part of the same solution as the Scatter and Gather orchestration (otherwise we would be force to redeploy when adding a Partner orchestration to the dll).

The Partner orchestrations would also communicate with some sort of outbound source like a web service or database for example. This would however complicate the setup therefore I’ve skipped that part in the example.

Managing partners

One of the benefits with a loosely coupled implementation is the possibility to add and remove Partner orchestrations without having redeploy the rest of the solution. Using this implementation the Gatherer orchestration needs to know how many Partner responses it should wait for before timing out. This requires that value being set in a config file or something similar. In this example the number of Partners are hard coded into the Gather orchestration (it’s set to 3 Partners) to simplify the setup.

Final thoughts

Knowing how to create loosely coupled solutions like this is good knowledge to have. It’s my own and others belief that this architecture makes it possible to create more robust and separated solutions that one can update without having to do a lot a work and disturb the current processes. It’s however not the best solution performancewise as it adds a lot of extra hits on the MessageBox database and generates more work for the MessageAgent.

There are also a few things to watch out for:

Eternal loops

It easy to end up in a situation where you’re subscribing to the same message as you posting to the MessageBox. That’ll create and endless loop and cause a lot disturbance before you’ll find it. Think through and document you subscriptions!

Correlations for promoting values

When doing a direct post in BizTalk most properties are not promoted. To force you properties to be promoted you’ll have to initialize a correlation on the property as you send it. I can’t say I like this. There should be a other way of saying that one wants it promoted.

A couple of other useful articles as we on the subject:

• Is there a pub/sub system underneath BizTalk?
• Okay, so where do subscriptions come from?
• Starting BizTalk Server 2004 orchestrations using the MessageAgent.

Download the example orchestration and let me know how you used it and what your solution looks like!

Speaking at Developer Summit 2008!

Whoho! I’ll be speaking at Developer Summit 2008. Developer Summit is a conference hosted by Cornerstone, it’s being held in Stockholm between the 9th and 11th of April 2008. 

I’ll be doing a talk on Master Data Management (MDM) for the Enterprise using BizTalk 2006. Basically I’ll present some basic theory behind the MDM concept and how it relates to SOA. Then I’ll relate all that to a customer case I recently worked on solving a MDM requirement. Finally I’ll be showing a short demo where I publish some Master Data and update all subscribers to it. In the same demo I’ll also demonstrate on how to monitor the process using Business Activity Monitoring.

Sound interesting? Something specific you think should be in the presentation? Let me know!

Hope to see you there!

SOA isn’t response request

I’ve often been told that one of the biggest mistakes people make when implementing a service oriented architecture is that they don’t re-architect their current architecture to become service oriented. I’ve never really understood what they meant by that until I read this article. SOA is not about adding a service based call to expose your current procedure calls as a service – it’s so much more. It’s about enabling ease of change and to create a more agile architecture.

Each of these assumptions exist in a Remote Procedure Call.  They are forms of coupling, pure and simple.  They fly in the face of SOA.

What do you think? Does it make sense in your world?

What’s an ESB architecture!?

I’ve just finished  reading a paper written by David Pallmann (PDF) from Neudesic. Belive it or not (or even better check it out), it’s a short and understandable description of what an ESB is! 

Neudesic sells a product called Cyclone which is something they call a ESB architecture and framework software and even if describing this product is the main purpose of the paper the first part deals with describing the basics of an ESB and try’s to “assemble a synthesis of popular ESB definitions”.

I’ve chosen a couple of parts of the paper that really appealed to me but I really recommend the full length version (only 8 pages!).

Defining the ESB

• An ESB is a backbone for connecting and integrating an enterprise’s applications and services.
• An ESB provides the necessary infrastructure to create a service oriented architecture.
• An ESB is a convergence of EAI, MOM, and SOA concepts.
• An ESB is based on open standards such as XML, SOAP, and WS-*.
• An ESB provides intelligent routing, such as publish-subscribe, message brokering, and failover routing.
• An ESB provides mediation, overcoming data, communication, and security differences between endpoints.
• An ESB integrates with legacy systems using standardsbased adapters.
• An ESB provides logical centralized management but is
  physically decentralized.
• An ESB is able to apply EAI concepts such as rules and
  orchestrations.
• An ESB is able to monitor and throttle activity as per a Service Level Agreement (SLA).

I never thought of an ESB as an convergence of EAI (Enterprise Application Integration), MOM (Message Oriented Middleware) and SOA (Service Oriented Architecture) concepts. But, ok, so what does the writer really mean by these concepts?  

SOA makes loosely-coupled, decentralized solutions possible that are enterprise-ready and based on interoperable standards. EAI allows integration of any combination of systems, with sophisticated message brokering, message translation, business process orchestration, and rules engine processing. MOM provides intelligent routing such as publish-subscribe topical messaging and strong managerial controls over routing, auditing, activity monitoring, and throttling.

And why do we need a bus architecture on top of SOA, MOM and EAI?

Point-to-point architecture
works all right on a small scale, but its problems become
apparent when used at the enterprise level. If each system has to know the connection details of every other system, then each new system added increases the problem of configuration and management exponentially. This was the impetus that led us to hub-and-spoke architectures, which most EAI products use. This architecture was a vast improvement over point-to-point architectures, and each system needed to communicate with only the hub. In addition, the hub could provide excellent management features since it was a party to all communication. It only took time to reveal some shortcomings with the hub-and-spoke approach, and today it is often associated with concerns about scalability, single point of failure, and vendor lock-in.

…

Fortunately,  there is a sound compromise to be found in the bus architecture, which provides the benefits of logical centralization but is physically decentralized. The bus architecture in earlier days was often used in message bus systems based on proprietary technologies, but an ESB implements this architecture using WS-* standards.

 So using all three of them will make an ESB? Well, not really …

If combining disciplines was all there was to an ESB, we’d simply call it “consulting”. To properly leverage these disciplines they need to be combined in the right way through an architecture that lets their strengths shine and overcome their inherent weaknesses. Each discipline has some weak areas that the others help to resolve: SOA needs better enterprise manageability; EAI needs to become decentralized; MOM needs to get away from proprietary technologies. Combining these disciplines properly in an ESB overcomes these weaknesses.

I’d be very intreseted in comments of other artciles, papers and so on relating to the ESB concept! 

Weekly sum up

I’ll start trying to post a weekly sum up of BizTalk, .NET topics I’ve come across during the week. We’ll see how goes as I’m not the most constant blogger …

However I worked with mainly three things this week.

Error handling and General Exception

I’ve tested some ideas relating to an error handling pattern we like to implement in a one of our major integration projects.  We’ve looked at how the new BizTalk 2006 Failed Message Routing fits in with our project. The idea is to have and Orchestration listening to the Error Message, log these and then put the failing message parts to file. We’ll then have a system that picks up the failed messages, launches them in an editor and makes it possible to resend the message after it’s been edited.

Problem arises when we get to handling exceptions in Orchestrations. Orchestrations doesn’t have the possible to use Failed Message Routing as ports does (I guess that ok, we might want handle different errors in different ways and so on.). However that means that we have to catch an error and create our own Error Message to submit to the message data box for further routing to the Orchestration handling errors (the one that logs and puts the message to file.) So far so good as long as we don’t get an General Exception! Usually when getting an General Exception, and we don’t catch it, the message gets suspended and we get a some error details giving us an idea of what went wrong. However as soon as one catches a General Exception we loose all information about the error … We’ve still haven’t found a good way of handling these errors. We just can’t afford loosing that error information!

General Exceptions are still a mystery to me. I guess that are there so BizTalk has a chance in catching exception that aren’t raised from .NET based code. But what kind of errors within an orchestration raises these kind of exceptions? I know a failure in a mapping does, what else? I’d really like this scenario to better documented. The error handling patter is based on a chapter in  the Pro BizTalk 2006 book (which is a excellent book in my opinion - buy it!) but the General Exception and how to deal with it isn’t discussed there either. Any ideas, book, articles are highly appreciated!

Validation

One important thing to think of in a BizTalk solution is to not let anything either in or out that doesn’t validate (ok, there are exception to this, but generally speaking)! BizTalk has great built in support for this in the XML Validator Pipeline Component (you might also want to have a look at this code). However when working with this it’s important to understand what one is validating against and it suddenly even more important to understand every details in the schemas (external system owners usually like some technical explanation when your telling them that their messages don’t validate in your pipeline ;)). Things we run into this week is the elementFormDefault attribute. The following reading help me understanding what the attribute does:

• http://www.w3.org/TR/xmlschema-0/#NS
• http://www.hanselman.com/blog/XmlValidating…XmlReaders.aspx
• http://geekswithblogs.net/dmillard…12935.aspx
• http://blogs.msdn.com/ebattalio/archive/2006/03/03/543154.aspx

Another thing we ran into was white space handling and  the xml:space attribute. Apparently if one likes to have a node with a space in it BizTalk removes this if the element doesn’t have the xml:space attribute. so <node> </node> will come out as <node></node> – sound familiar?

But if the schema doesn’t declare that the node will have a xml:space attribute the validation will fail! To get this working one has to get the schema declaring the the xml:space attribute and then reference it like this example. The schema from w3c (if you don’t write your own like in the article) is located here.

End-to-end tracing in a SOA

I’ll make this short even if it should be the longest part in this post. Basically we’re trying to archive the following:

Our solution send loads of different messages types between five different BizTalk servers. The client likes to be able to have full text search within these messages and to also be able to see all the messages  in each interchange within a server.

So say for example that we receive and Order message in a flat file format. This is transformed in to a Xml message that is then routed down to two other different BizTalk servers. It should then be possible for the client to start an application, click on the Order message type, enter for example the order number (an element i in the message) and see the the full content of the different files within the interchange (in this example that would be the flat file and the Xml file) where that Order number is found.

Basically it’s does what the HAT does (with full text search and a custom GUI). In the first phase of the project we’ll have it work per server, but in the final solution this should perform over all servers … I’ll come back to this in a later post. In the mean time listen to this episode of Hanselminutes on end-to-end tracking. This will be a challenge in every service orientated architecture …

Finally

I’ve sold my iPod Nano and ordered the Creative Vision Zen:M. Now I’ll catch up on all those dnrTV Webcasts! I’ll also make sure to watch everyone of the BizTalk 2006 Webcasts during my commute to work.

Let me know if you found some of this information useful and I’ll try harder to post some like this every Friday.

ARCast – Patterns and Anti-Patterns for SOA

I’m a loyal ARcast listener and I think I’ve listened to most of the shows that ever been produced (at least those that are available in the archive). I also listen to a couple of other podcasts that are focusing on .NET and Microsoft techniques (for example .NET Rocks!) and in my opinion ARcast is by far the most rewarding (I guess this is both a matter of taste and depending on what kind of work one does.)! However these two shows on Patterns and anti-patterns for SOA (part 1 and part 2) really stands out!

They really got me thinking of a couple of anti-patterns the project I’m currently working on are stuck in. And Ron Jacob delivers (as always) the message with a lot of humor and enthusiasm. Recommended!