Gobbledygooks by Richard Hallgren

Update 2009-08-11: This project turned out to be far more complicated and bigger than I first expected (ever heard that before?). Due to that and the fact that we wanted to have a company behind that could offer full-time support and stability “BizMon” has been released as a commercial product that you can find here.  

I love to get some help from you to test it and make it as good as possible. Even if it is commercial and cost money we have a free alternative for small environments and we work hard to keep the license price as low as possible.

Update 2009-02-25: In the original post I said I’d post more on the architecture and the code during February 09. I’m however current struggling getting the needed legal rights etc, etc to be able to talk further about the "BizMon"-solution. It was harder than I thought … I’ll get back to posing on the subject as soon as I have that sorted.

Integration of enterprise processes often ends up being very business critical. If a integration fails delivering the messages it was supposed to it usually means the business will be affected in a very negative way (for example losing money or delivering bad service). That of course means that monitoring the status of the integrations soon becomes very important (if you’re not into getting yelled at or potentially loosing your job).

Strangely enough BizTalk Server 2006 R2 in my humble opinion doesn’t come with the right tool to efficiently monitoring big enterprise integration solutions!

What do I mean by monitoring?

Before I get myself deeper into trouble I’d like to define what I mean by monitoring. I think monitoring a BizTalk integration solution could be divided into four categories.

1. Infrastructure (traditional)
   This is the easy one and one that IT-pros and alike are used to monitor. Hardware status, network traffic, disk space, event logs etc all fall under this category. If the storing area for the databases start running low on memory we can be pretty sure it’ll eventually effect the integration somehow.
2. BizTalk infrastructure
   This is where it starts getting a bit trickier. This category includes the status of receive locations, orchestrations, host instances and send ports. If a receive location is down no messages will be picked up (but we can also be sure of not getting any suspended messages).
3. Suspended messages
   As most reader of this blog probably know suspended message is due to some sort of failure in BizTalk. It can be an actually exception in code or something that went wrong while trying to send messages. It’s however and important category to monitor.
4. Heartbeat (monitoring actual successful traffic)
   While the points 1-3 above focuses on errors and that things being inactive this category actually monitors that the integration runs as expected.

   To me this final point is almost the most important one. What I mean is that if everything runs as expected and we’re sending the expected amount of messages in the right pace everything else must be ok – right? It’s however the one that in my experience almost always overlooked!

"What do you mean ‘Not the right tools to monitor’? We have loads of tools in BizTalk 2006 R2!"

OK. So let’s see what tools we have available actually monitor the categories above.

1. Infrastructure (traditional)
   I won’t discuss this kind of monitoring in this post. There are loads of tools (all from the huge expensive enterprise ones to plenty of good open-source alternatives) for this and you’re probably already using one or several of them already.
2. BizTalk infrastructure
   There are a couple of way of achieving this. One of the is to use the Microsoft BizTalk Server Management Pack for Operation Manager. It does however of course require that you have invested in System Center Operation Manager already …

   Another way is to either use the ExplorerOM classes or connecting directly to the BizTalk configuration database and code your own report of some sort.

   The final (and most common way in my experience) is to try and document the correct configuration and settings and then have someone check these manually (if you’re that person I feel for you …).

3. Suspended messages
   Suspended messages are of course very important to monitor and it’s for some reason also the first thing developers think of monitoring when developing BizTalk integration (maybe because of the fact that they’re similar to traditional exceptions in software). There are also here a couple of different ways to solve the problem.

   Microsoft BizTalk Server Management Pack for Operation Manager mentioned above has the functionality to monitor and altering on suspended messages.

   BizTalk Server fires the MSBTS_ServiceInstanceSuspendedEvent WMI event every time a service instance gets suspended. It’s fully possible to write a service that watches for this event and then for example sends some sort of alert. Darren Jefford has an example on how do something like that in this post.

   In BizTalk 2006 Failed Message Routing was introduced. This gives the developer the possibility to subscribe to suspended messages. These can then for example be sent out to file system or written to a database. Microsoft ESB Guidance for BizTalk Server 2006 R2 Exception management component uses this approach. The problem with this approach is however that the message is moved out of BizTalk and one loses all the built in possibilities of resending them etc.

4. Heartbeat (monitoring actual successful traffic)

   As I said before I think this is a very important metric. If you can see that messages travel through BizTalk in a normal rate things much be pretty ok – right? Without doing to much coding and developing you own pipeline components for tracking etc there are two options.

   The first one is of course using the Health and Activity Tracking tool (HAT). This shows a simple view of receives, processed and sent messages. I hate to say it but the HAT tool is bad. It’s slow, it’s hard to use, it’s hard to filter information, it times out, it doesn’t aggregate information, it’s basically almost useless … (Just to make one thing clear: I make my living working with BizTalk and I really enjoy the product but tracking and monitoring is really one of it’s ugly sides. I hate to say it.)

   The other option is to develop a simple BAM tracking profile to monitoring the send and receive port ports of the different processes.

So to repeat what I said earlier: no I don’t think BizTalk comes with the right tool to monitor integration solutions. I do however think that the platform has the capabilities to create something that could close that gap in the product.

What I need!

Much of what’s discussed in this post can be solved using the BizTalk Administrations Console (to manually monitor BizTalk infrastructure status) or in the Health and Activity Tracking tool (to manually monitor traffic). The aim of this post is however to discuss the possibilities to use this information, aggregate it and give the persons responsible for monitoring integration a dashboard that shows the current status of all integrations within the enterprise.

The dashboard monitor application need the following main features.

• In one single screen give an overview of the overall status of all the integrations. By status I mean if there are ports, orchestration or host instances that aren’t running that should be running or if there is any suspended traffic on that particular integration.
• The possibility to show detailed information for a specific integration on what artifacts (ports, host instances etc) that are/aren’t running. How much traffic that’s been sent/received via the integration. When traffic was sent/received and if there’s any suspended messages on the integration.
• The option to filter exclude specific artifacts from monitoring (for example receive locations that’s usually turned off etc).
• Setting up monitoring by for example email and also define what integrations to be included in one specific monitoring (different persons are usually responsible for monitoring different integrations).

Introducing "BizMon"

Based on the needs and "requirements" above I’ve started developing a application. The idea is to release it as open-source as soon as I get to a first stable version (I’d be very interested in help on practical details on how to do so). For now I’ll demonstrate it by showing some screenshots. The application is a web application based on ASP.NET MVC.

Screenshot: "Applications" page

The above image shows a screenshot from the start page of the BizMon-application that shows the aggregated status of the entire BizTalk group it’s connected to. The applications is build to monitor one BizTalk group and the shown page displays all applications within that BizTalk group.

In the example image the two first rows have an OK status. That means that all of the monitored artifacts (receive locations, send ports, orchestrations and host instances) within that application are in a running and OK status.
The yellow line on the YIT.NO.Project-application indicates a warning. That means that all the artifacts are in a OK status but there’re suspended messages within that application. The red line indicates that one or more of the monitored artifacts are in a inactive status.

Each row and application show when the last message on that application was received and/or sent. It also show how many suspended messages exists and when the last message got suspended.

Screenshot: "Application-detail" page

When clicking on a application on the main page previously shown the application-detail page is displayed for that application. This page shows detailed information on each of the artifacts within that application. I also shows suspended messages and the date and time of the last suspended.

It also displays a graph showing how many messages that has been processed by each of the ports. Currently the graph can view data from the last 7 days. In the screenshot above data from the 6th of January is shown and as it’s set to display data for a specific day the data is grouped in hours of that day. It’s also possible to view the aggregated data from all the traced days as show below. When viewing data from all days the graphs is grouped by days.

(The graph only shows data from the 6th of January as this is from test and there was no traffic of the previous days but I’m sure you get the idea …)

Screenshot: "Application-detail" page with inactive artifacts

This final page show details of an application with some inactive artifacts. The small cross highlighted by the arrow in the image show the possibility to filter out a single artifact from monitoring. If an excluded artifacts is failing the overall status of the application will still be OK and no alerts will be sent.

Help!

I’d love to get some input and feedback on all this. What do you think could be useful, what do you think won’t? Do you know of something similar, how do you solve this kind of monitoring?

I’d also like to know any suitable placed to publish the code as an open-source project or is the best thing to just release it here on the blog? What do you think? Use the comments or send me a mail.

What’s next?

I have a few thing on the alerts part of the application left and then I’ll release a first version. I’m hoping that could happened at the end of February 09 (look at the update at the top of the post) . Make sure to let me know what you think!

I’ll publish a follow-up post discussing the technical details and the architecture more in detail shortly.

This is a very specific problem but I’m sure some of you stumbled over it. When disassembling a XML message in a SOAP port BizTalk can’t read the message type. This causes problems when for example trying to handle an envelope message and split it to smaller independent messages in the port. It’s a known problem discussed here and here (you also find information about it in the BizTalk Developer’s Troubleshooting Guide) and the solution is to make a small change in the generated web service class. Below is a small part of he generated class.

//[cut for clarity] ...
            Microsoft.BizTalk.WebServices.ServerProxy.ParamInfo[] outParamInfos = null;
            string bodyTypeAssemblyQualifiedName = "XXX.NO.XI.CustomerPayment.Schemas.r1.CustomerPayments_v01, XXX.NO.XI.CustomerPaym" +
                "ent.Schemas.r1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ac564f277cd4488" +
                "e";
            // BizTalk invocation
            this.Invoke("SaveCustomerPayment", invokeParams, inParamInfos, outParamInfos, 0, bodyTypeAssemblyQualifiedName, inHeaders, inoutHeaders, out inoutHeaderResponses, out outHeaderResponses, null, null, null, out unknownHeaderResponses, true, false);
        }
    }
}

Basically the problem is that the generated code puts the wrong DocumentSpecName property in the message context. I’ll not dicusses the problem in detail here but Saravana Kumar does thorough dissection of the problem in his post on it.

The solution is to update the bodyTypeAssemblyQualifiedName to set a null value. That will cause the XmlDiassasemler to work as we’re used to and expect.

If the value null is passed instead of bodyTypeAssemblyQualifiedName, SOAP adapter won’t add the DocumentSpecName property to the context. Now, when we configure our auto-generated SOAPReceiveLocation to use XmlReceive pipeline, the XmlDisassembler component inside XmlReceive will go through the process of automatic dynamic schema resolution mechanism, pick up the correct schema and promotes all the required properties (distinguished and promoted) defined in the schema and it also promotes the MessageType property.

From: http://www.digitaldeposit.net/saravana/post/2007/08/17/SOAP-Adapter-and-BizTalk-Web-Publishing-Wizard-things-you-need-to-know.aspx

//[cut for clarity] ...
            Microsoft.BizTalk.WebServices.ServerProxy.ParamInfo[] outParamInfos = null;
            string bodyTypeAssemblyQualifiedName = null;
            // BizTalk invocation
            this.Invoke("SaveCustomerPayment", invokeParams, inParamInfos, outParamInfos, 0, bodyTypeAssemblyQualifiedName, inHeaders, inoutHeaders, out inoutHeaderResponses, out outHeaderResponses, null, null, null, out unknownHeaderResponses, true, false);
        }
    }
}

But if you have an automated deployment process you probably use MSBuild to generate your Web Services. Then is soon becomes very annoying to remember to update the .cs-file again and again for every deployment. So how can we script that update?

First we need to find a regular expression to find the right values. With some help from StackOverflow (let’s face it, there are some crazy regular expressions skills out there …) I ended up on the following.

(?<=string\sbodyTypeAssemblyQualifiedName\s=\s)(?s:[^;]*)(?=;)

If you’re not a RegEx ninja the line above does something like this: 

1. After the string “string bodyTypeAssemblyQualifiedName = ”
2. turn on single line (treat “\r\n” as any other character) ( this is what “(?s: )” does)
3. match every character that is not a semicolon
4. until a single semicolon is reached.

Then I used a task from the SDC Task library (you probably already use this if you’re using MSBuild and BizTalk). More specially we use the File.Replace

<Target Name="FixSOAPServiceCode">
    <File.Replace
            Path="$(WebSiteServicePath)CustomerPaymentService\App_Code\CustomerPaymentService.asmx.cs"
            Force="true"
            NewValue="null"
            RegularExpression="(?&lt;=string\sbodyTypeAssemblyQualifiedName\s=\s)(?s:[^;]*)(?=;)">
    </File.Replace>
</Target>

Now this task is part of the build script and called right after the tasks that generates the web service. This saves me a lot of manual work and potential errors!

I’ll start by saying that I really (like in “really, really!“) like BizUnit! BizUnit in combination with MSBuild, NUnit and CruiseControl.NET has really changed the way work and how I feel about work in general and BizTalk development in particular.

If you haven’t started looking into what for example MSBuild can do you for you and your BizTalk build process you’re missing out on something great. The time spent on setting up a automatic build process is time well spent – take my word for it!

But build processes isn’t was this post is supposed to be about. This post is about one of the few limitations of BizUnit and the possibilities to work around one of those in particular.

BizUnit is a test framework that is intended to test Biztalk solutions. BizUnit was created by Kevin.B.Smith and can be found on this CodePlex space . BizUnit has quite a significant number of steps that have nothing to do with Biztalk per se and can be used for any integration project testing.

If you have used BizUnit before and need an introduction before reading further, start here, here or here – they’re all excellent articles.

As stated BizTalk has quite a significant number of steps but to my knowledge it’s missing a step to change and update file from within the test script. This step and a couple of other are added in separate fork-project to BizUnit called BizUnitExtensions.

This project [BizUnitExtension] aims to provide some more test step libraries, tools and utilities to enhance the reach of BizUnit. Here you can find some enhancements/extensions to the steps in the base libraries , new steps, support applications, tutorials and other documentation to help you understand and use it….This project is currently owned and contributed to by Santosh Benjamin and Gar Mac Críostaand. Our colleagues have also contributed steps and suggestions. We welcome more participation and contributions.

Amongst other steps (some for Oracle DBs etc) BizUnitExtensions adds a XmlPokeStep!

XmlPokeStep: This step is modelled on the lines of the NAnt XmlPoke task The XmlPokeStep is used to update data in an XML file with values from the context This will enable the user to write tests which can use the output of one step to modify the input of another step.

A cool thing about BizUnitExtensions is that it really just extends BizUnit. You’ll continue to run on the BizUnit dll:s when you use steps form BizUnit and just use the BizUnitExtensions code when you actually use some of steps from that library.

The example below shows how we first use an ordinary BizUnit task to validate and read a value from a file. We then use BizUnitExtension to gain some new powers and update the file with that value we just read.

<!–Ordinary BizUnit step to validate a file –>
<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
    <Timeout>5000</Timeout>
    <Directory>..\..\ReceiveRequest</Directory>
    <SearchPattern>*Request.xml</SearchPattern>
    <DeleteFile>false</DeleteFile>

    <ContextLoaderStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlContextLoader">
        <XPath contextKey="messageID">/*[local-name()=’SystemRequest’]/ID</XPath>
    </ContextLoaderStep>

</TestStep>

<!–Use BizUnitExtensions to poke the value and change it –>
<TestStep assemblyPath="BizUnitExtensions.dll" typeName="BizUnit.Extensions.XmlPokeStep">
    <InputFileName>..\..\SystemResponse.xml</InputFileName>
    <XPathExpressions>
        <Expression>
            <XPath>/*[local-name()=’SystemResponse’]/ID</XPath>
            <NewValue takeFromCtx="messageID"></NewValue>
        </Expression>
    </XPathExpressions>
</TestStep>

This of course means that all you need to extend you current test steps and gain some cool new abilities is to add another assembly to you test project!

Using BizUnitExtensions in a “real” scenario

An scenario when this can be useful is the following example were we need to test some message correlation.

1. A message request is received via a web service.
2. The message is sent to a queue via an orchestration in BizTalk. To be able to correlate the response a message id is added to the message request sent to the back-end system. 
3. A message response is sent from the back-end system using a second queue. The response message contains the same message id as the incoming request contained.

4. BizTalk correlates the message back to the web service using the message id.

So how can we now test this? The steps should be something like the below.

Notice that we read and write to the file system in the example. Once deployed to test these send ports and receive location will be reconfigured to use the queuing adapter. But for testing the scenario the file system works just fine a simplifies things IMHO.

1. Send a request message using BizUnit and the HttpRequestResponseStep. Make sure it runs concurrently with the other steps and then wait for a response (using the runConcurrently-attribute on the step). 
2. Configure the send port so the orchestration that added the generated message id writes the message to a folder. Use the FileValidateStep and a nested XmlContextLoader to read the message from the folder and write the message id the context. 
3. Use the context and the XmlPokeStep from BizUnitExtensions to update a response message template with the message id from the request message (this is of course needed so we can correlate the response message back to the right orchestration). 
4. Copy the update response message template using the FileCreateStep to the folder that is monitored by the the receive location used for reading responses.

<TestCase testName="TestCorrelationTest">
    <TestSetup>
        <!–Clean up!–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
            <Directory>..\..\ReceiveRequest</Directory>
            <SearchPattern>*.*</SearchPattern>
        </TestStep>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileDeleteMultipleStep">
            <Directory>..\..\SendResponse</Directory>
            <SearchPattern>*.*</SearchPattern>
        </TestStep>
    </TestSetup>

    <TestExecution>
        <!–Post a request message on the SOAP port. Run it Concurrently–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.HttpRequestResponseStep" runConcurrently="true">
            <SourcePath>..\..\WebRequest.xml</SourcePath>
            <DestinationUrl>http://localhost:8090/BizTalkWebService/WebService1.asmx?op=WebMethod1</DestinationUrl>
            <RequestTimeout>15000</RequestTimeout>
        </TestStep>

        <!–Read the system request message and read the generaed id to the context–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
            <Timeout>5000</Timeout>
            <Directory>..\..\ReceiveRequest</Directory>
            <SearchPattern>*Request.xml</SearchPattern>
            <DeleteFile>false</DeleteFile>
            <ContextLoaderStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlContextLoader">
                <XPath contextKey="messageID">/*[local-name()=’SystemRequest’]/ID</XPath>
            </ContextLoaderStep>
        </TestStep>

        <!–If we have the file in source control it might be read-only -> remove that attribute–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.ExecuteCommandStep">
            <ProcessName>attrib</ProcessName>
            <ProcessParams>SystemResponse.xml -r</ProcessParams>
            <WorkingDirectory>..\..\</WorkingDirectory>
        </TestStep>

        <!–Update our response template (using BizUnitExtensions) and add the message id that we read into the the context–>
        <TestStep assemblyPath="BizUnitExtensions.dll" typeName="BizUnit.Extensions.XmlPokeStep">
            <InputFileName>..\..\SystemResponse.xml</InputFileName>
            <XPathExpressions>
                <Expression>
                    <XPath>/*[local-name()=’SystemResponse’]/ID</XPath>
                    <NewValue takeFromCtx="messageID"></NewValue>
                </Expression>
            </XPathExpressions>
        </TestStep>

        <!–Wait a moment so we don’t copy the file until we’re done updating it–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.DelayStep">
            <Delay>1000</Delay>
        </TestStep>

        <!–Copy the file to the folder that monitored by the receive location for opicking up system responses–>
        <TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileCreateStep">
            <SourcePath>..\..\SystemResponse.xml</SourcePath>
            <CreationPath>..\..\SendResponse\SystemResponse.xml</CreationPath>
        </TestStep>
    </TestExecution>
</TestCase>

This might look messy at first but I think it’s really cool I also think it worth thinking about on how you should run this at during development otherwise? You would then have to build some small stub to return a response message with the right id … I prefer this method!

BizUnitExtension makes me like BizUnit even more! Thanks to Kevin. B. Smith, Santosh Benjamin and Gar Mac Críostaand for spending so much time on this and sharing it with us mere mortals!

Update: Gar’s blog can be found here.

Lately I’ve been using custom Xslt more and more instead of the BizTalk mapping tool. I still use the mapping tool in easy scenarios when I just need to do some straight mapping or maybe even when I need to concatenate some fields, but as soon as I need to to some looping, grouping, calculations etc I’ve made a promise to myself to use custom Xslt!

I find custom Xslt so much easier in more complex scenarios and once one get past the template matching and understands how and when to use recursion (No you can’t reassign a variable in Xslt and you’re not supposed to!) I find it to be a dream compared to the mapping tool. I also find the code so much easier to maintain compared to the result from the mapping tool. I mean someone would have to pay me good money to even start figuring out what this map is doing. And the scary thing is that if you worked with BizTalk for a while you probably know that maps like this isn’t that rare! I’ve even seen worse!

Don’t get me wrong, Xslt definitely has some major limitations.

Some of the acute limitations of XSLT 1.0 I can think of off the top of my head are:

• The lack of real string comparison
• No support for dates
• No simple mechanism for grouping
• No querying into RTF’s

And it doesn’t take long before one runs up against one of these and suddenly you wish you were back in mapping tool were we just could add scripting functoid and some code or a external assembly. But then you remember … (Sorry, I know it’s painful just to watch it).

There has to be a better way of doing this and combining the best out of the two worlds!

I started looking into to how BizTalk actually solves combining Xslt and the possibility to use external assemblies. After a couple of searches I found Yossi’s nice article that explained it to me (from 2005! I’m behind on this one!) and it even turns out that there an example in the BizTalk SDK.

Ok, so now I had what I need. I started a new class library project and began writing some date parsing methods, some padding methods and so on.

It somehow however felt wrong from the start and I got this grinding feeling that I must be reinventing the wheel (I mean these are well know limitations of Xslt and must have been solved before). Even worse I also felt that I was creating a stupid single point of failure as I started using the component from all different maps in my projects and I have actually seen how much pain a bug in similar shared dll:s could cause. Basically a small bug in the component could halt all the process using the library! Finally I realized that this kind of library would be under constant development as we ran into more and more areas of limitations in the our Xslt:s and that would just increase the risk of errors and mistakes.

After some further investigation I found EXSLT which looked like a solution to my problems! A stable, tested library of Xslt extensions that we could take dependency on as it’s unlikely to have any bugs and that should include the functionality we’re missing in standard Xslt!

How I used EXSLT in BizTalk

These days it’s the Xml Mvp crowd over at the Mvp.Xml project who develops and maintains the .NET implementation of EXSLT. So I downloaded the latest binaries (version 2.3). Put the the Mvp.Xml.dll in the GAC. Wrote a short custom extension Xml snippet that looked like this (using what I’ve learnt from Yossi’s article).

<?xml version="1.0" encoding="utf-8"?>
<ExtensionObjects>
    <ExtensionObject
     Namespace="http://exslt.org/dates-and-times"
     AssemblyName="Mvp.Xml,
     Version=2.3.0.0, Culture=neutral,
     PublicKeyToken=6ead800d778c9b9f"
     ClassName="Mvp.Xml.Exslt.ExsltDatesAndTimes"/>
</ExtensionObjects>

All you define is the Xml namespace you like to use in your Xslt to reference the dll, the full assembly name and finally the name of the class in Mvp.Xml.Exslt you want to use (make sure you also download the source to Xml.Mvp, it helps when looking up in what classes and namespaces different methods are placed).

That means you need one ExtensionObjects block for each class you want you use which really isn’t a problem as the methods are nicely structured based on there functionality.

Then we can use this in a Xslt like this:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:S1="http://ExtendedMapping.Schema1"
                xmlns:S2="http://ExtendedMapping.Schema2"
                xmlns:exslt="http://exslt.org/dates-and-times"
                version="1.0"> 

    <xsl:template match="/">
        <S2:Root>
            <Field>
                <xsl:value-of select="exslt:dateTime()"/>
            </Field>
        </S2:Root>
    </xsl:template>
</xsl:stylesheet>

Which gives us the below output. Notice the current time and date! Cool!

<S2:Root xmlns:S1="http://ExtendedMapping.Schema1" xmlns:S2="http://ExtendedMapping.Schema2" xmlns:exslt="http://exslt.org/dates-and-times">
  <Field>2008-09-05T20:45:13+02:00</Field>
</S2:Root>

All you then have to do in you map is to reference the Xslt and the extension Xml.

Just as final teaser I’ll paste a few methods from the EXSLT documentation

Some string methods:

• str:align()
• str:concat()
• str:decode-uri()
• str:encode-uri()
• str:padding()
• str:replace()
• str:split()
• str:tokenize()

Some date and time methods:

• date:add()
• date:add-duration()
• date:date()
• date:date-time()
• date:day-abbreviation()
• date:day-in-month()
• date:day-in-week()
• date:day-in-year()
• date:day-name()
• date:day-of-week-in-month()
• date:difference()
• date:duration()
• date:format-date()
• date:hour-in-day()
• date:leap-year()
• date:minute-in-hour()
• date:month-abbreviation()
• date:month-in-year()
• date:month-name()
• date:parse-date()
• date:second-in-minute()
• date:seconds()
• date:sum()
• date:time()
• date:week-in-month()
• date:week-in-year()
• date:year()

As if this was enough (!) the Mvp Xml project added a couple of there own methods! What about string lowercase and string uppercase – all in Xslt! And about 30 new date-time related methods extra to the standard ones already in EXSLT!

Check out the full documentation here!

Let me know how it works out for you.

I’ve seen people struggle both on the forums and while doing consulting when in it comes to finding an good way of grouping and transforming content in file before debatching it. Say for example we have a text file like the example below.

0001;Test row, id 0001, category 10;10
0002;Test row, id 0002, category 10;10
0003;Test row, id 0003, category 10;10
0004;Test row, id 0004, category 20;20
0005;Test row, id 0005, category 20;20
0006;Test row, id 0006, category 20;20
0007;Test row, id 0007, category 20;20
0008;Test row, id 0008, category 10;10
0009;Test row, id 0009, category 10;10
0010;Test row, id 0010, category 30;30

Notice how the the ten rows belong to three different categories (10,20 and 30). These kind of export are in my experience quite common batch export from legacy systems and they usually aren’t ten rows (in my last project the sizes ranged from 5 MB to 25 MB) …

The problem

The problem is that the receiving system expects the data to be in separate groups, grouped by the categories the rows belong to. The expected message might look something like the below for category 10 (notice how all rows within the group are from category 10)

<ns1:Group numberOfRows="5" xmlns:ns1="http://Blah/Group">
  <Row>
    <Id>0001</Id>
    <Text>Test row, id 0001, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0002</Id>
    <Text>Test row, id 0002, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0003</Id>
    <Text>Test row, id 0003, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0008</Id>
    <Text>Test row, id 0008, category 10</Text>
    <Category>10</Category>
  </Row>
  <Row>
    <Id>0009</Id>
    <Text>Test row, id 0009, category 10</Text>
    <Category>10</Category>
  </Row>
</ns1:Group>

The problem is now that we need to find a efficient way of first grouping the incoming flat file based message and then to debatch it using those groups. Our ultimate goal is to have separate messages that groups all rows that belongs to the same category and then send these messages to the receiving system. How would you solve this?

I’ve seen loads of different solution involving orchestrations, databases etc, but the main problem they all had in common is that they’ve loaded up to much of the message in memory and finally hit an OutOfMemoryException.

The solution

The way to solve this is to use pure messaging as one of the new features in BizTalk 2006 is the new large messages transformation engine.

Large message transformation. In previous versions of BizTalk Server, mapping of documents always occurred in-memory. While in-memory mapping provides the best performance, it can quickly consume resources when large documents are mapped. In BizTalk Server 2006, large messages will be mapped by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat.

So the idea is the to read the incoming flat file, use the Flat File Disassembler to transform the message to it’s XML representation (step 1,2 and in the figure below) and the to use XSLT to transform in to groups (step 4 and 5). We will then use the XML Disassembler to split those groups into separate messages containing all the rows within a category (step 6 and 7).

Step 1, 2 and 3 are straight forward and pure configuration. Step 4 and 5 will require some custom XSLT and I’ll describe that in more detail in the section below.  Step 6 and 7 will be discussed in the last section of the post.

Grouping

Let’s start by looking at a way to group the message. I will use some custom XSLT and a technique called the Muenchian method. A segment from the XML representation of the flat file message could look something like this.

<Rows xmlns="http://Blah/Incoming_FF">
    <Row xmlns="">
        <ID>0001</ID>
        <Text>Test row, id 0001, category 10</Text>
        <Category>10</Category>
    </Row>
    <Row xmlns="">
        <ID>0002</ID>
        <Text>Test row, id 0002, category 10</Text>
        <Category>10</Category>
    </Row>
...
[message cut for readability]

The XSLT will use could look something like the below. It’s kind of straight forward and I’ve tried commenting the important parts of in the actual script. Basically it will use keys to fins the unique categories and then (again using keys) selecting those rows within the category to loop and write to a group.

<?xml version="1.0" encoding="utf-8"?>

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ns1="http://GroupAndDebatch.Schemas.Incoming_FF"
                xmlns:ns2="http://GroupAndDebatch.Schemas.Grouped"
                >
    <!--Defining the key we're gonna use-->
    <xsl:key name="rows-by-category" match="Row" use="Category" />

    <xsl:template match="/ns1:Rows">
        <ns2:Groups>

        <!--Looping the unique categories to get a group for-->
        <xsl:for-each select="Row[count(. | key('rows-by-category', Category)[1]) = 1]">

            <!--Creating a new group and set the numberOfRows-->
            <Group numberOfRows="{count(key('rows-by-category', Category))}">

            <!--Loop all the rows within the specific category we're on-->
            <xsl:for-each select="key('rows-by-category', Category)">
                <Row>
                    <ID>
                        <xsl:value-of select="ID"/>
                    </ID>
                    <Text>
                        <xsl:value-of select="Text"/>
                    </Text>
                    <Category>
                        <xsl:value-of select="Category"/>
                    </Category>
                </Row>
            </xsl:for-each>
            </Group>
        </xsl:for-each>
        </ns2:Groups>
    </xsl:template>

</xsl:stylesheet>

You have found all the XSLT and XML related features in Visual Studio – right?

Ok, so the above XSLT will give us a XML structure that looks some like this.

<?xml version="1.0" encoding="utf-8"?>
<ns2:Groups xmlns:ns2="http://Blah/Groups" xmlns:ns1="http://Blah/Group">
    <ns1:Group numberOfRows="5">
        <Row>
            <ID>0001</ID>
            <Text>Test row, id 0001, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0002</ID>
            <Text>Test row, id 0002, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0003</ID>
            <Text>Test row, id 0003, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0008</ID>
            <Text>Test row, id 0008, category 10</Text>
            <Category>10</Category>
        </Row>
        <Row>
            <ID>0009</ID>
            <Text>Test row, id 0009, category 10</Text>
            <Category>10</Category>
        </Row>
    </ns1:Group>
    <ns1:Group numberOfRows="4">
        <Row>
            <ID>0004</ID>
            <Text>Test row, id 0004, category 20</Text>
            <Category>20</Category>
        </Row>
...
[message cut for readability]

Finally! This we can debatch!

Debatching

Debatch the Groups message above is also rather straight forward and I won’t spend much time on in this post. The best way to learn more about it is to have a look ate the EnvelopeProcessing sample in the BizTalk SDK.

And the end result of the debatching are single messages within a unique category, just as the receiving system expects! Problem solved.

Issue #1 – slow transformations

The first time I’ve put a solution like this in test and started testing with some real sized messages (> 1 MB) I really panicked, the mapping took forever. And I really mean forever, I sat there waiting for 2-3 hours (!) for a single file getting transformed. When I had tested the same XML based file in Visual Studio the transformation took about 10 seconds so I knew that wasn’t it. With some digging here I found the TransformThreshold parameter.

TransformThreshold decides how big a message can be in memory before BizTalk start buffering it to disk. The default value is 1 MB and one really has to be careful when changing this. Make sure you thought hard about your solution and situation before changing the value – how much traffic do you receive and how much of that can you afford reading in to memory?

In my case I received a couple of big files spread out over a night so setting parameter with a large amount wasn’t really a problem and that really solved the problem. The mapping finished in below 10 minutes as I now allow a much bigger message to be read into memory and executed in memory before switching over to the large message transformation engine and start buffering to disk (which is always much slower).

Problem #2 – forced to temp storage

Looking at the model of the data flow again you probably see that I’m using the XML Disassembler to split the grouped files (step 5 to step 6).

The only way I’ve found this to work is actually to write the Grouped XML message to file and the to read that file in to BizTalk again and in that receive pipeline debatch the message. Not the most elegant solution, but there really isn’t a another out-of-the-box way of debatching messages (the XML Assembler can’t do it) and I don’t want to use an orchestration to execute the a pipeline as I want to keep the solution pure messaging for simplicity and performance reasons.

Finishing up

Have you solved similar cases differently? I’d be very interested in your experience! I also have a sample solution of this – just send me an email and make sure you’ll get it.

Update

Also don’t miss this issue (pdf) of BizTalk Hotrod magazine. There is an article on “Muenchian Grouping and Sorting using Xslt” describing exactly the problem discussed above.

I have a old post on removing XML namespace from outgoing messages using XSLT in a map on the send port. Removing XML namespace is usually a late requirement that shows up during integration tests with for example legacy systems that has problems reading XML and only finds XML namespaces messy and confusing and wants it removed.

The post recently got commented by Jeff Lynch (one of the codebetter.com bloggers) asking me why I just didn’t create a schema without any XML namespace in it to represent the outgoing schema (see figure below) and then map to that in the send port.

Say for example that we have incoming messages like the one below with namespaces.

<ns0:BlahRoot xmlns:ns0="http://Sample.BlahIncomingSchema">
  <BlahNode>Test Value</BlahNode>
</ns0:BlahRoot>

We’ve then defined a schema without namespace and map to that and get the following result.

<BlahRoot>
  <BlahNode>Test Value</BlahNode>
</BlahRoot>

This method is of course much cleaner then any previous and it’s also more conceptually correct as the schema actually represents the contact between BizTalk and the receiving system (the contract is a message without namespace in it, not one with that we then remove).

The only problem with this kind of approach is that as BizTalk recognized the message type using a combination between root node and XML namespace we can’t have another schema with the BlahRoot root node without a defined XML namespace. Even if those two schemas would look totally different in structure and be two different message types BizTalk would be able to see the difference (to BizTalk they both be #BlahRoot message types).

Thanks Jeff for pointing this out to me.

I’ve just finished watching this webcast from QCon 2008 with Martin Fowler and Jim Webber. It’s basically their view on how integration middleware is used today and how we plan and implement our SOA and ESB project.

Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration – fair enough. I totally agree that trying to convert your complete enterprise architecture into a SOA architecture is a guaranteed failure. This is also something we heard for a while now from others as well.

I do also agree that BizTalk is a huge platform and that it isn’t perfect in all aspects. IMHO it does however give us some important advantages compared to a custom coded message bus and services. I’ll try and list a few of them below.

1. Fixed architecture
   We don’t have invent the wheel every time. BizTalk is a product with an architecture that one have to learn and use. There are times when this is a pain (did I hear low latency and BizTalk persistence points?) but it’s also a huge kick start to all projects once you learnt it. Once you figured out how you use the products you’ll actually have something up and running in no time.

   Isn’t an early delivery that we can test something good? I’m sure I can deliver a BizTalk based integration faster that some can using custom code when starting from scratch.
    

2. Drag-and-drop
   There is a learning curve to BizTalk and all it’s tools but once one gotten over this one can move really fast, even without a deep understanding of .NET and software development (there are of course both pros and cons to this). I’ve seen projects with 50+ integration processes (to me that’s a big, complex project) where we actually used people fresh out of school, spent two weeks to teach them basic BizTalk and had them deliver critical parts of the projects. I’d like to see that happen custom coded ESB project with thousands lines of code …

   You probably get a nice design and implementation if you can hire 10 top developers and a couple of architects, but that isn’t always possible.
    

3. Tools
   Does a custom code, lean approach, really scale in this scenario? Do you take the time to pause and build that management and configuration tool that you don’t get with a custom code project? I don’t say that we got the perfect view and control of our processes and messages in BizTalk but at least we got some control. At least I got the BizTalk Administration Console to let me see how my different application are doing, what messages and process etc that got suspended. At least I got the BAM framework where I can configure a tracking and monitoring in no time (usually …) etc, etc. 
    
4. It works
   Say your implementing a process that receives purchase orders and that these orders might contain orders for a couple of millions dollars. Do you want to be developer that tells you boss that you think you might have lost a message due to a exception in your custom code? Of course you have 90% test coverage and continuous integration but you never tested for this exception case … I don’t want to be that developer/architect.

I don’t say don’t test. I’m very pro testing and I really feel that agile is the right approach. I’m just saying I need something tested and safe to build this super critical solutions on. Something that I know works and that I can be really productive on and start solving business cases from minute one.

What do you think? Does BizTalk have man-bobs and is that only a bad thing? And does Martin Fowler really have leather pants on?

Yesterday I presented the MasterData Management using BizTalk 2006 R2 talk (I’ll soon have a post out with the presentation in English) I recently held at Developer Summit at the local .NET user group in Karlstad (KNUG).

KNUG is a new .NET user group that I actually helped start a couple of months ago. This meeting was the second meeting for the group. The meeting was attended by about 20 persons and we had two presentations on the agenda. Besides my own Thomas Heder showed the group some LINQ and how he and his colleagues uses LINQPad to develop and test there queries.

We also discussed future subjects, possible speakers and moving information on the group over to a Community Server driven site.

Does anyone have any experience on Community Server and how the feature set matches those need for running a user group (managing users, blogs, email lists, calendar etc)?

This post will try and explain how BAM tracking can be used in SOAP based request response scenario in BizTalk 2006. It important to notice that some of the issues discussed in the post are specific to the SOAP adapter and are non-issues if the scenario would for example use the the WCF adapter or similar.

Describing the scenario

In this case we have a SOAP receive port that receives a request and returns a response. The request is routed to a orchestration that calls three different send ports. These ports then sends new requests to back-end systems and returns responses (communication with back-ends systems are also SOAP based). The three responses are used to build up the final response that is then returned to original receive port as a final response.

Our goal is to track the duration between the request and response on each of the ports. The idea is also to find a solution and tracking model that doesn’t have to change if we add or remove ports or add similar processes to track.

Defining and deploying the tracking model

We’ll start by defining our tracking model in Excel. Our activity contains of the following items:

• InterchangeId (data item)
  As we won’t correlate all the tracking point into one single row (that would break the goal of having one model for all processes, the model would then have to be specific for one process and it’s specific ports) the interchange id will then tell us what different rows belong together and that describes one process.
• ReceivePortName (data item)
  The name of the receive port.
• Request (milestone item)
  The time the request was either sent or received (depending on if we track a port that received/sent the request using a receive port or send port).
• Response (milestone item)
  The time the response was either sent or received (depending on if we track a port that received/sent it’s response on a receive port or send port).
• SendPortName (data item)
  The name of the send port.

After we described the model it’s time to export it to an XML representation and then to use the BM tool to deploy it and generate the BAM database infrastructure. You’ll find some nice info on this here.

Using the Tracking Profile Editor to bind the model

Next step is to bind the model to the implementation using the Tracking Profile Editor. The figure below shows the different properties that were used. Notice that none of the items was bound to the actual orchestration context. All properties are general properties that we track on the ports. This is important as that gives us the possibility to just add and remove ports to change the tracking.

The next figure shows how the tracking of the request milestone event actually happens on either the RP1 port or on any of the three different send ports! If we developed a new process using other ports we could just add it here, no new model required.

What about the continuation then?

Our final problem is that unless we somehow correlate our request tracking point with our receive tracking point the receive we’ll end up with each tracking point spread over several different rows. In the example below I’ve marked the request for the RP01 port and the response event on the same port.

The reason for this is of course that BAM doesn’t have a context for the two tracking points and doesn’t know that actually belongs together. This differs from tracking in a orchestration were we always are in a context (the context of the orchestration), it’s then easy for BAM to understand that we like to view all the tracking point as one row – when tracking on ports it’s different. Continuation helps us tell BAM that we like have a context and correlate these two points.

In our case ServiceID is the prefect candidate for correlating the two points. A request and a response will have the same service id. In an other situation we could just as well have used a value from inside the message (say for example an invoice id).

The result is one single row for the request response for each port. So in our case a complete process (a complete interchange) is shown on four rows (one row for each of the ports). In the example below the first rows shows us the complete duration (and the other tracking data) between the request response to the client. The other rows show the duration for the send ports communication with the back-ends systems.

This model might not be optimal in an other scenario where your process are more fixed and you can then create a tracking model that is more specific to you actual process. But this solution meets our design goal as we’re now able to just add and remove port using the tracking profiler to track new port in completely new processes without having to go back and change the tracking model.

  NOTE: When configuring BAM to track a port the MessageType is actually promoted. This causes some problems in combination with the SOAP based ports that have been published using the Web Services Publishing Wizard. Saravana writes about this here and all his articles on this subject is a must read when working with SOAP ports. The problem however comes down to that the Web Services Publishing Wizard generates code that puts the wrong DocumentSpecName in the message context and that causes the XmlDisassembler to fail (it tricks the XmlDisassembler to look for a MessageType that doesn’t exists).

This usually isn’t a problem (unless you like to use a map on a port) but as BAM will force the port to promote the MessageType based on the DocumentSpecName we’ll have to fix this. Saravana has two solutions to the problem and I find the one that replaces the DocumentSpecName with a null value and lets the XmlDisassembler find the MessageType to work well.

I recently spoke at the leading developer conference here in Sweden called Developer Summit. The talk was called Masterdata Management using BizTalk 2006. The slides can be found here.

I can really recommend Developer Summit as a conference. Everything is super well organized and having the opportunity to listen to celebrities like David Chappell, Jim Webber, Dan North and Christian Weyer in Sweden is really great!

This year I also liked the mix of presentation as some where more general presentation as for example  Benjamin Ling who’s the director of platform at Facebook and who gave an insight to how the platform is managed and developed. I also really enjoyed (besides the obvious ones as for example Christian Weyer’s WCF talk) Frans Hänel’s talk on the history and architecture behind a major price comparison site called prisjakt.nu – very interesting and a great technical mix in otherwise Microsoft focused conference.