0001;Test row, id 0001, category 10;10
0002;Test row, id 0002, category 10;10
0003;Test row, id 0003, category 10;10
0004;Test row, id 0004, category 20;20
0005;Test row, id 0005, category 20;20
0006;Test row, id 0006, category 20;20
0007;Test row, id 0007, category 20;20
0008;Test row, id 0008, category 10;10
0009;Test row, id 0009, category 10;10
0010;Test row, id 0010, category 30;30

Notice how the the ten rows belong to three different categories (10,20 and 30). These kind of export are in my experience quite common batch export from legacy systems and they usually aren’t ten rows (in my last project the sizes ranged from 5 MB to 25 MB) …

The problem

The problem is that the receiving system expects the data to be in separate groups, grouped by the categories the rows belong to. The expected message might look something like the below for category 10 (notice how all rows within the group are from category 10)

<ns1:Group numberOfRows="5" xmlns:ns1="http://Blah/Group">
  <Row>
    <Id>0001</Id>
    <Text>Test row, id 0001, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0002</Id>
    <Text>Test row, id 0002, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0003</Id>
    <Text>Test row, id 0003, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0008</Id>
    <Text>Test row, id 0008, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0009</Id>
    <Text>Test row, id 0009, category 10</Text>
    <Category>10</Category>
  </Row>
</ns1:Group>

The problem is now that we need to find a efficient way of first grouping the incoming flat file based message and then to debatch it using those groups. Our ultimate goal is to have separate messages that groups all rows that belongs to the same category and then send these messages to the receiving system. How would you solve this?

I’ve seen loads of different solution involving orchestrations, databases etc, but the main problem they all had in common is that they’ve loaded up to much of the message in memory and finally hit an OutOfMemoryException.

The solution

The way to solve this is to use pure messaging as one of the new features in BizTalk 2006 is the new large messages transformation engine.

Large message transformation. In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat.

So the idea is the to read the incoming flat file, use the Flat File Disassembler to transform the message to it’s XML representation (step 1,2 and in the figure below) and the to use XSLT to transform in to groups (step 4 and 5). We will then use the XML Disassembler to split those groups into separate messages containing all the rows within a category (step 6 and 7).

Step 1, 2 and 3 are straight forward and pure configuration. Step 4 and 5 will require some custom XSLT and I’ll describe that in more detail in the section below.  Step 6 and 7 will be discussed in the last section of the post.

Grouping

Let’s start by looking at a way to group the message. I will use some custom XSLT and a technique called the Muenchian method. A segment from the XML representation of the flat file message could look something like this.

<Rows xmlns="http://Blah/Incoming_FF">
    <Row xmlns="">
        <ID>0001</ID>
        <Text>Test row, id 0001, category 10</Text>
        <Category>10</Category>
    </Row>
    <Row xmlns="">
        <ID>0002</ID>
        <Text>Test row, id 0002, category 10</Text>
        <Category>10</Category>
    </Row>
…
[message cut for readability]

The XSLT will use could look something like the below. It’s kind of straight forward and I’ve tried commenting the important parts of in the actual script. Basically it will use keys to fins the unique categories and then (again using keys) selecting those rows within the category to loop and write to a group.

<?xml version="1.0" encoding="utf-8"?>

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ns1="http://GroupAndDebatch.Schemas.Incoming_FF"
                xmlns:ns2="http://GroupAndDebatch.Schemas.Grouped"
                >
    <!–Defining the key we’re gonna use–>
    <xsl:key name="rows-by-category" match="Row" use="Category" />

    <xsl:template match="/ns1:Rows">
        <ns2:Groups>

        <!–Looping the unique categories to get a group for–>
        <xsl:for-each select="Row[count(. | key(’rows-by-category’, Category)[1]) = 1]">

            <!–Creating a new group and set the numberOfRows–>
            <Group numberOfRows="{count(key(’rows-by-category’, Category))}">

            <!–Loop all the rows within the specific category we’re on–>
            <xsl:for-each select="key(’rows-by-category’, Category)">
                <Row>
                    <ID>
                        <xsl:value-of select="ID"/>
                    </ID>
                    <Text>
                        <xsl:value-of select="Text"/>
                    </Text>
                    <Category>
                        <xsl:value-of select="Category"/>
                    </Category>
                </Row>
            </xsl:for-each>
            </Group>
        </xsl:for-each>
        </ns2:Groups>
    </xsl:template>

</xsl:stylesheet>

You have found all the XSLT and XML related features in Visual Studio - right?

Ok, so the above XSLT will give us a XML structure that looks some like this.

<?xml version="1.0" encoding="utf-8"?>
<ns2:Groups xmlns:ns2="http://Blah/Groups" xmlns:ns1="http://Blah/Group">
    <ns1:Group numberOfRows="5">
        <Row>
            <ID>0001</ID>
            <Text>Test row, id 0001, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0002</ID>
            <Text>Test row, id 0002, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0003</ID>
            <Text>Test row, id 0003, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0008</ID>
            <Text>Test row, id 0008, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0009</ID>
            <Text>Test row, id 0009, category 10</Text>
            <Category>10</Category>
        </Row>
    </ns1:Group>
    <ns1:Group numberOfRows="4">
        <Row>
            <ID>0004</ID>
            <Text>Test row, id 0004, category 20</Text>
            <Category>20</Category>
        </Row>
…
[message cut for readability]

Finally! This we can debatch!

Debatching

Debatch the Groups message above is also rather straight forward and I won’t spend much time on in this post. The best way to learn more about it is to have a look ate the EnvelopeProcessing sample in the BizTalk SDK.

And the end result of the debatching are single messages within a unique category, just as the receiving system expects! Problem solved.

Issue #1 - slow transformations

The first time I’ve put a solution like this in test and started testing with some real sized messages (> 1 MB) I really panicked, the mapping took forever. And I really mean forever, I sat there waiting for 2-3 hours (!) for a single file getting transformed. When I had tested the same XML based file in Visual Studio the transformation took about 10 seconds so I knew that wasn’t it. With some digging here I found the TransformThreshold parameter.

TransformThreshold decides how big a message can be in memory before BizTalk start buffering it to disk. The default value is 1 MB and one really has to be careful when changing this. Make sure you thought hard about your solution and situation before changing the value - how much traffic do you receive and how much of that can you afford reading in to memory?

In my case I received a couple of big files spread out over a night so setting parameter with a large amount wasn’t really a problem and that really solved the problem. The mapping finished in below 10 minutes as I now allow a much bigger message to be read into memory and executed in memory before switching over to the large message transformation engine and start buffering to disk (which is always much slower).

Problem #2 - forced to temp storage

Looking at the model of the data flow again you probably see that I’m using the XML Disassembler to split the grouped files (step 5 to step 6).

The only way I’ve found this to work is actually to write the Grouped XML message to file and the to read that file in to BizTalk again and in that receive pipeline debatch the message. Not the most elegant solution, but there really isn’t a another out-of-the-box way of debatching messages (the XML Assembler can’t do it) and I don’t want to use an orchestration to execute the a pipeline as I want to keep the solution pure messaging for simplicity and performance reasons.

Finishing up

Have you solved similar cases differently? I’d be very interested in your experience! I also have a sample solution of this - just send me an email and make sure you’ll get it.

One problem for us was the we had our solution files in one place on the file system and our build files in a totally different place. The solution was to add the build file for the current solution as a Solution Folder (as shown in the figure below) and then set MSBuild to use $(ItemDir) as its Initial Directory. That will kick of MSBuild from the directory that the current selected Solution Folder points to and in our case that’s were the XXX.Build.Article.proj file exists.

Joel is also one of the founders of the FogCreek Software company. FogCreek has a couple of different products but the most known is probably FogBugz which is a really cool project management/bug tracking/wiki/support tool for software development and management. At first it might not look like a very impressing tool but have a look at this presentation by Joel and I think you’ll start thinking different (I know we did and we’re now using it for both planing and managing big integrations projects).

Anyway, if you missed the story behind stackoverflow have a look here and don’t miss the podcasts that Joel and Jeff record, some of them are really cool. I know I’ve learnt a lot by just listening in to their conversations.

We not have quite a few online BizTalk communities such as the Microsoft forums, Google Groups, BizTalk Gurus, facebook (search for BizTalk related groups) and at LinkedIn (a couple of open BizTalk related groups) - is there room for another one at stackoverflow? Do we need one?

The post recently got commented by Jeff Lynch (one of the codebetter.com bloggers) asking me why I just didn’t create a schema without any XML namespace in it to represent the outgoing schema (see figure below) and then map to that in the send port.

Say for example that we have incoming messages like the one below with namespaces.

<ns0:BlahRoot xmlns:ns0="http://Sample.BlahIncomingSchema">
  <BlahNode>Test Value</BlahNode>
</ns0:BlahRoot>

We’ve then defined a schema without namespace and map to that and get the following result.

<BlahRoot>
  <BlahNode>Test Value</BlahNode>
</BlahRoot>

This method is of course much cleaner then any previous and it’s also more conceptually correct as the schema actually represents the contact between BizTalk and the receiving system (the contract is a message without namespace in it, not one with that we then remove).

The only problem with this kind of approach is that as BizTalk recognized the message type using a combination between root node and XML namespace we can’t have another schema with the BlahRoot root node without a defined XML namespace. Even if those two schemas would look totally different in structure and be two different message types BizTalk would be able to see the difference (to BizTalk they both be #BlahRoot message types).

Thanks Jeff for pointing this out to me.

Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration - fair enough. I totally agree that trying to convert your complete enterprise architecture into a SOA architecture is a guaranteed failure. This is also something we heard for a while now from others as well.

I do also agree that BizTalk is a huge platform and that it isn’t perfect in all aspects. IMHO it does however give us some important advantages compared to a custom coded message bus and services. I’ll try and list a few of them below.

1. Fixed architecture
   We don’t have invent the wheel every time. BizTalk is a product with an architecture that one have to learn and use. There are times when this is a pain (did I hear low latency and BizTalk persistence points?) but it’s also a huge kick start to all projects once you learnt it. Once you figured out how you use the products you’ll actually have something up and running in no time.

   Isn’t an early delivery that we can test something good? I’m sure I can deliver a BizTalk based integration faster that some can using custom code when starting from scratch.
    

2. Drag-and-drop
   There is a learning curve to BizTalk and all it’s tools but once one gotten over this one can move really fast, even without a deep understanding of .NET and software development (there are of course both pros and cons to this). I’ve seen projects with 50+ integration processes (to me that’s a big, complex project) where we actually used people fresh out of school, spent two weeks to teach them basic BizTalk and had them deliver critical parts of the projects. I’d like to see that happen custom coded ESB project with thousands lines of code …

   You probably get a nice design and implementation if you can hire 10 top developers and a couple of architects, but that isn’t always possible.
    

3. Tools
   Does a custom code, lean approach, really scale in this scenario? Do you take the time to pause and build that management and configuration tool that you don’t get with a custom code project? I don’t say that we got the perfect view and control of our processes and messages in BizTalk but at least we got some control. At least I got the BizTalk Administration Console to let me see how my different application are doing, what messages and process etc that got suspended. At least I got the BAM framework where I can configure a tracking and monitoring in no time (usually …) etc, etc. 
    
4. It works
   Say your implementing a process that receives purchase orders and that these orders might contain orders for a couple of millions dollars. Do you want to be developer that tells you boss that you think you might have lost a message due to a exception in your custom code? Of course you have 90% test coverage and continuous integration but you never tested for this exception case … I don’t want to be that developer/architect.

I don’t say don’t test. I’m very pro testing and I really feel that agile is the right approach. I’m just saying I need something tested and safe to build this super critical solutions on. Something that I know works and that I can be really productive on and start solving business cases from minute one.

What do you think? Does BizTalk have man-bobs and is that only a bad thing? And does Martin Fowler really have leather pants on?

KNUG is a new .NET user group that I actually helped start a couple of months ago. This meeting was the second meeting for the group. The meeting was attended by about 20 persons and we had two presentations on the agenda. Besides my own Thomas Heder showed the group some LINQ and how he and his colleagues uses LINQPad to develop and test there queries.

We also discussed future subjects, possible speakers and moving information on the group over to a Community Server driven site.

Does anyone have any experience on Community Server and how the feature set matches those need for running a user group (managing users, blogs, email lists, calendar etc)?

Describing the scenario

In this case we have a SOAP receive port that receives a request and returns a response. The request is routed to a orchestration that calls three different send ports. These ports then sends new requests to back-end systems and returns responses (communication with back-ends systems are also SOAP based). The three responses are used to build up the final response that is then returned to original receive port as a final response.

Our goal is to track the duration between the request and response on each of the ports. The idea is also to find a solution and tracking model that doesn’t have to change if we add or remove ports or add similar processes to track.

Defining and deploying the tracking model

We’ll start by defining our tracking model in Excel. Our activity contains of the following items:

• InterchangeId (data item)
  As we won’t correlate all the tracking point into one single row (that would break the goal of having one model for all processes, the model would then have to be specific for one process and it’s specific ports) the interchange id will then tell us what different rows belong together and that describes one process.
• ReceivePortName (data item)
  The name of the receive port.
• Request (milestone item)
  The time the request was either sent or received (depending on if we track a port that received/sent the request using a receive port or send port).
• Response (milestone item)
  The time the response was either sent or received (depending on if we track a port that received/sent it’s response on a receive port or send port).
• SendPortName (data item)
  The name of the send port.

After we described the model it’s time to export it to an XML representation and then to use the BM tool to deploy it and generate the BAM database infrastructure. You’ll find some nice info on this here.

Using the Tracking Profile Editor to bind the model

Next step is to bind the model to the implementation using the Tracking Profile Editor. The figure below shows the different properties that were used. Notice that none of the items was bound to the actual orchestration context. All properties are general properties that we track on the ports. This is important as that gives us the possibility to just add and remove ports to change the tracking.

The next figure shows how the tracking of the request milestone event actually happens on either the RP1 port or on any of the three different send ports! If we developed a new process using other ports we could just add it here, no new model required.

What about the continuation then?

Our final problem is that unless we somehow correlate our request tracking point with our receive tracking point the receive we’ll end up with each tracking point spread over several different rows. In the example below I’ve marked the request for the RP01 port and the response event on the same port.

The reason for this is of course that BAM doesn’t have a context for the two tracking points and doesn’t know that actually belongs together. This differs from tracking in a orchestration were we always are in a context (the context of the orchestration), it’s then easy for BAM to understand that we like to view all the tracking point as one row – when tracking on ports it’s different. Continuation helps us tell BAM that we like have a context and correlate these two points.

In our case ServiceID is the prefect candidate for correlating the two points. A request and a response will have the same service id. In an other situation we could just as well have used a value from inside the message (say for example an invoice id).

The result is one single row for the request response for each port. So in our case a complete process (a complete interchange) is shown on four rows (one row for each of the ports). In the example below the first rows shows us the complete duration (and the other tracking data) between the request response to the client. The other rows show the duration for the send ports communication with the back-ends systems.

This model might not be optimal in an other scenario where your process are more fixed and you can then create a tracking model that is more specific to you actual process. But this solution meets our design goal as we’re now able to just add and remove port using the tracking profiler to track new port in completely new processes without having to go back and change the tracking model.

  NOTE: When configuring BAM to track a port the MessageType is actually promoted. This causes some problems in combination with the SOAP based ports that have been published using the Web Services Publishing Wizard. Saravana writes about this here and all his articles on this subject is a must read when working with SOAP ports. The problem however comes down to that the Web Services Publishing Wizard generates code that puts the wrong DocumentSpecName in the message context and that causes the XmlDisassembler to fail (it tricks the XmlDisassembler to look for a MessageType that doesn’t exists).

This usually isn’t a problem (unless you like to use a map on a port) but as BAM will force the port to promote the MessageType based on the DocumentSpecName we’ll have to fix this. Saravana has two solutions to the problem and I find the one that replaces the DocumentSpecName with a null value and lets the XmlDisassembler find the MessageType to work well.

I can really recommend Developer Summit as a conference. Everything is super well organized and having the opportunity to listen to celebrities like David Chappell, Jim Webber, Dan North and Christian Weyer in Sweden is really great!

This year I also liked the mix of presentation as some where more general presentation as for example  Benjamin Ling who’s the director of platform at Facebook and who gave an insight to how the platform is managed and developed. I also really enjoyed (besides the obvious ones as for example Christian Weyer’s WCF talk) Frans Hänel’s talk on the history and architecture behind a major price comparison site called prisjakt.nu - very interesting and a great technical mix in otherwise Microsoft focused conference.

If you have used the SOAP adapter before you might know that all you had to do was to turn on Windows based security for the exposed SOAP service and the username was automatically promoted to the context of the incoming BizTalk message. That username could then be used for routing, tracking which user called the service or using the value in plain text when communicating further to other connected systems. However using the WCF adapter this is not true anymore - when using the new WCF Message Security model the username and password is encrypted in the message and once the message is received by BizTalk it’s to late to read it. Basically we have to read the username in the actual service and write it into our own context property (that doesn’t get encrypted).

One way of achieving this is to read the username in the service and then to add it to the WCF Message Headers. All WCF message headers will by default be written to a the BizTalk Message context property called InboundHeaders (in the http://schemas.microsoft.com/BizTalk/2006/1/Adapters/WCF-properties namespace). First we’ll create an EndpointBehavior that will use a MessageInspector to add the username to the message header.  Finally we create BehaviorExtensionElement so we can use a WCF Custom Binding in BizTalk and configure it to add our new behavior.

Creating the new EndpointBehavior

To create the configurable behavior we’ll need the three classes we mentioned above.

1. A class that implements the IDispatchMessageInspector interface to handle to reading and writing to the actual message.
2. A class that implements the IEndpointBehavior interface to define what kind of endpoint we’re creating and what it should do.
3. A class that implements the BehaviorExtensionElement abstract class to  create the behavior and make it configurable.

using System;
using System.Collections.Generic;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Description;
using System.ServiceModel.Configuration;

namespace CustomWCFProperties.Behavior
{
    /// <summary>
    /// PromoteUserNameMessageInspector implements IDispatchMessageInspector and adds the name from the WindowsIdentity to a WCF header called WindowsUserName in the http://CustomWCFProperties.Schema namespace. BeforeSendReply only returns as we’re not interested in handling the response.
    /// </summary>
    public class PromoteUserNameMessageInspector : IDispatchMessageInspector
    {
        #region IDispatchMessageInspector Members

        public object AfterReceiveRequest(ref System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel, System.ServiceModel.InstanceContext instanceContext)
        {
            string windowsUserName = ServiceSecurityContext.Current.WindowsIdentity.Name;
            request.Headers.Add(MessageHeader.CreateHeader("WindowsUserName", "http://CustomWCFProperties.Schema", windowsUserName));
            return null;
        }

        public void BeforeSendReply(ref Message reply, object correlationState)
        {
            return;
        }

        #endregion
    }

    /// <summary>
    /// PromoteUserNameBehavior implements IEndpointBehavior and adds a message inspector to the dispatch behavior. Doesn’t use any binding parameters, doesn’t validate any configuration etc and can’t be used in a client (only in a service).
    /// </summary>
    public class PromoteUserNameBehavior : IEndpointBehavior
    {
        #region IEndpointBehavior Members

        public void AddBindingParameters(ServiceEndpoint endpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
        {
            return;
        }

        public void ApplyClientBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.ClientRuntime clientRuntime)
        {
            throw new Exception("The method or operation is not implemented.");
        }

        public void ApplyDispatchBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.EndpointDispatcher endpointDispatcher)
        {
            endpointDispatcher.DispatchRuntime.MessageInspectors.Add(new PromoteUserNameMessageInspector());
        }

        public void Validate(ServiceEndpoint endpoint)
        {
            return;
        }

        #endregion
    }

    /// <summary>
    /// Defines the behavior.
    /// </summary>
    public class PromoteUserNameBehaviorElement : BehaviorExtensionElement
    {
        protected override object CreateBehavior()
        {
            return new PromoteUserNameBehavior();
        }

        public override Type BehaviorType
        {
            get { return typeof(PromoteUserNameBehavior); }
        }
    }
}

Finally we have to sign the assembly using a strong key and add it to the GAC.

Configure the machine.config

As we need BizTalk and the WCF adapter to pick up the need behavior and make it possible to configure our receive port we need to to add the behavior element to the machine.config. The easiest way of doing this is to use the new WCF Service Configuration Editor tool and point to the machine.config file.

After the dll been added and the machine.config file has been saved the the line below should have been added to the <behaviorExtensions> element (that is if you use the same strong name key as in the sample project I’ve linked here).

<add name="addCustomWCFProperties" type="CustomWCFProperties.Behavior.PromoteUserNameBehaviorElement, AddCustomWCFPropertiesBehavior, Version=1.0.0.0, Culture=neutral, PublicKeyToken=705e34637fdffc54" />

Create the BizTalk Receive Port and Receive Location

Next thing to do is to start the BizTalk WCF Service Publishing Wizard. Choose to publish a service endpoint and make sure you enable metadata and create a receive location. In this example we’ll next choose to "Publish schemas as WCF service" and then define our service by naming service operations and so on.

When you then browse to the URL you choose to publish your service to you’ll see the nice example of how to instance the service you just defined.

If we then send a request message to service (you’ll find a client as part of the attached solution here) and inspect the message and its context properties in BizTalk we’ll see that the username of the calling client is nowhere to be found.

Configure a WCF-Custom binding and adding a Endpoint Behavior

To add the username to the message context we’ll need to add our newly created behavior to our service. We’ll do this by switch the service over to use a WCF-Custom binding to enable configuration. We then need to add the URL in the address field, define the binding type to a wsHttpBinding and to add our addCustomeWCFProperties behavior to the list of endpoint behaviors.

  NOTE: there is a limitation in the BizTalk WCF implementation in that you can’t create the WCF-Custom receive location that uses a HTTP in-process based binding (like the wsHttpBinding used in a WCF-Custom endpoint is) first and then use the WCF Publishing Wizard to only publish a metadata endpoint.

Richard Seroter writes about it here and I found the same thing to be true.

"This error doesn’t have to do with mixing MEX endpoints and “regular” endpoints in the same IIS web site, but rather, creating MEX endpoints for in-process HTTP bindings seems to trigger this. Note that an IIS-hosted MEX endpoint CAN be created for IIS-hosted HTTP endpoints, but not for in-process hosted HTTP endpoints."

If you however choose a different binding that Http or (as in this case) publishes the metadata first and then switches over to a custom binding you’re ok.

If we then post another message to the service and inspect the message we’ll see that the behavior actually added a header and that it’s part of our BizTalk context properties. The adapter is also smart enough to know that this header isn’t part of the original headers and therefore stores in it’s own field within the context properties (you’ll find as part of the InboundHeaders block as well).

One problem remains - the actual value of the user is nested inside a XML node and the property isn’t promoted. 

Extract and promote the value

To extract and promote the value we use an old fashion pipeline component using the following code in the execute method (the complete project is part of the downloadable sample project).

        public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
        {
            StringReader reader = new StringReader(inmsg.Context.Read("WindowsUserName", "http://CustomWCFProperties.Schema").ToString());

            if (reader != null)
            {
                XPathDocument document = new XPathDocument(reader);
                XPathNavigator navigator = document.CreateNavigator();
                string value = navigator.SelectSingleNode("/").Value;
                inmsg.Context.Promote("WindowsUserName", "http://CustomWCFProperties.Schema", value);
            }

            return inmsg;
        }

All the component does is reading the XML node the value exists inside and then it reads the actual value. Finally it writes the value back and promotes it. To be able to promote the value we also have to have a Property Schema deployed with a corresponding property name and namespace (WindowsUser and http://CustomeWCFProperties.Schema in this case).

The end results looks something like this.

The username is extracted and promoted and available for example for tracking or to for example use in a routing scenario.

This technique could of course be used for all kinds of scenarios where you like to add information to the context properties and could potentially replace a lot of the classic scenarios for custom pipelines.

All kind of comments are of course more than welcome!

Download the sample solution  here.

Consider the simplified procedure below, I’ll use is an example for much of the discussion in the rest of the post. Basically the procedure inserts or updates (based on if the optional EmployeeID exists from before or not) employee information to the Employee table of a database named "TestDB".

Use [TestDB]
Go
If Exists(select * From dbo.sysobjects Where id = object_id(N‘AddEmployee‘) And OBJECTPROPERTY(id, N‘IsProcedure‘) = 1) Drop Procedure AddEmployeeGo
Create Procedure AddEmployee
    @EmployeeId Int = -1,
    @LastName Varchar(30),
    @FirstName Varchar(30),
    @MiddleName Varchar(30)
As
Begin
    Declare @currentEmployeeId int
    Declare @currentAction tinyint
    – Check based on the EmployeeId if the employee exists from before    
    If Not Exists(Select EmployeeId From Employee Where EmployeeId = @EmployeeId)
        Begin
            – Inserts new employee        
            Insert Into Employee (LastName,FirstName,MiddleName)
            Values (@LastName, @FirstName, @MiddleName)
            Set @currentEmployeeId = Scope_Identity()
            Set @currentAction = 1
        End
    Else
        Begin
            – Updates employee        
            Update Employee Set LastName = @LastName,
            FirstName = @FirstName WHERE EmployeeId = @EmployeeId
            Set @currentEmployeeId = @EmployeeId
            Set @currentAction = 2
        End      

    Select @currentEmployeeId As [@EmployeeId], @currentAction
    As [@ProcedureAction] For Xml Path (‘Response‘)
End
Go

Builds and version control

I always try to have the complete database version controlled. One solution to this that some people advocate is to have a central development database. I have however always found it to be a lot of hassle with overwriting each others changes and so on and always try to make it possible for each developer to build a local version of the database for development. It should also be possible to always check out the latest version and run it to build all the database objects - no matter what you have from before. This is of course a bit tricky when it comes to the actual tables as you might have data and so on (in my opinion it shouldn’t be the case during development but that’s another post discussing test data for the database etc).

When it however comes to Stored Procedure as in this post it’s easy - drop the old one if it exists and create the new one as in the example.

If Exists(select * From dbo.sysobjectsWhere id = object_id(N‘AddEmployee‘) And OBJECTPROPERTY(id, N‘IsProcedure‘) = 1) Drop Procedure AddEmployee
Go
Create Procedure AddEmployee @EmployeeId Int = -1, …

If found this article by Jeff Atwood over at Coding Horror (and the articles by K. Scott Allen it links to) very good on how to think about version control database objects.

Idempotent procedures

In complex integration solution it’s a common problem with multiple messages - sending systems might send the same message twice for a number of reasons and one will end up with a shaky system if that causes an error in the integration. I’ve also seen solutions to this problem where a check is done against the database (say from a BizTalk orchestration) and based on that result a decision is made to call an update or an insert procedure. In my opinion that just unnecessary complexity and database communication. I believe one should try and handle whether to update or insert inside the procedure as long as that’s possible and effective.

I in the AddEmployee Stored Procedure example might receive a EmployeeID as an input parameter and based on a check inside the procedure if the employee exists from before with that id a post is either updated or inserted.

Response message

I generally always want two things in return from a Store Procedure like the one in the AddEmployee example (In a select based procedure that would of course be very different as we like all the data in return).

1. The id that been added or changed
2. What action that’s been performed (insert or update)

What action that was performed is a good idea to return as it’s quite usual to use this in for example BAM tracking or other logging and so on (we might want know how many employees that was added versus updated for example).

Using the Xml Path syntax for generating the Xml response (thanks again Nick for a great post) from the procedure has made it even simpler to actually skip using the Add Generated Item option in BizTalk. All you have think about is to set name of the child node the request root node to the same name as the procedure (in this case AddEmployee). Also remember that the type should be set up as a Multi-Part Message Type in the orchestrations you use it from and that the Receive Pipeline you choose in the Send Port must use a Xml Disassembler component (XmlReceive pipeline will do fine unless you have your custom pipeline for some reason).