14
Oct
Posted by Richard » Add Comment »

Presenting on “Efficient system integration documentation” at BizTalk User Group Sweden

I’ve talked a lot about efficient documentation previously on this blog, both when in comes to splitting the documentation up in different layers as here, but also in relation to tools BizTalk Documenter and BizTalk Web Documenter for automatically generation technical documentation as here.

Last week I however had the pleasure to present on the BizTalk User Group Sweden meeting and got to talk all documentation for a whole hour Smile. You can find the PPT from the meeting here.

I ended the presentation by presenting the ten commandments for efficient system integrations documentation:

I.        Thou shall not manually document anything that can be automatized

II.      Thou shall keep it simple & make it look nice

III.    Thou shall use a wiki based platform

IV.     Thou shall use pictures whenever appropriate

V.       Thou shall have well defined guidelines for your documentation

VI.     Thou shall have a well defined target audience for your documentation

VII.   Thou shall document continuously in your project

VIII. Thou shall have a common vocabulary and common icons defined

IX.     Thou shall test your documentation with target audience

X.       Thou as the developer of an integration should document it

Follow on twitter
image
30
Sep
Posted by Richard » 1 Comment »

Reborn as BizTalk Web Documenter!

About a year ago I published a project that helps automating documentation for BizTalk Server – much like BizTalk Documenter but with the difference that this tools instead generates a dynamic web site. I’ve called in Config Explorer, a name I knew was sh*t from the start but the best I could come up with at the time.

After a while however, when enough people told me that they also thought the name was crap I decided to change it to BizTalk Web Documenter. I’ve also moved the whole project to GitHub.

I’ll try and spend a bit more time on the project than I have the last year and will update the documentation, wiki and roadmap as well as implement a few new features.

I hope you’ll help me – fork the project today!

2
Sep
Posted by Richard » 4 Comments »

Using BizTalk Web Documenter as a new way of documenting your BizTalk solutions

2012-09-30: I’ve renamed the whole project and moved it to GitHub. Hope to see you there!

As most BizTalk developers/architects you have probably been in a situation were documentation of your integration solution has been discussed.  And you probably also been just as happy as me when you first found BizTalk Documenter. BizTalk Documenter is a great tool for extracting all that configuration data that is so vital to BizTalk and create a nice technical documentation of it.

documentation levelsThere are of course several levels to documentation in BizTalk Server based solution were the technical information is at the lowest level. Above that you would then have you integration process descriptions and then possible another level above that where you fit the integration processes into bigger business processes, and so on – but let’s stick to the technical documentation for now.

BizTalk Documenter does a nice job of automating the creation of technical documentation, instead of basically having to all that configuration job a second time when documenting. Doing technical documentation by hand usually also breaks down after a while as solution setup and configuration changes a lot and the work is really time consuming. And as documentation that isn’t up to date frankly is completely worthless people soon looses interest in both reading and maintaining the technical documentation.

Using BizTalk Documenter however has a few problems …

BizTalk Config Explorer Web Documenter

As we started trying to find ways to solve the issues above we had a few ideas we wanted the new solution to handle better.

config explorer

Check out an an example of a Config Explorer yourself.

Getting started

Config Explorer is a open source tool published on CodePlex GitHub – a good place to start to understanding the workflow of creating documentation and to download the tool itself.

Contribute!

This release is a 1.0 release and there will are many features we haven’t had time for, but a few of them we’ve planned for future releases. Unfortunately there will however also be bugs and obvious things we’ve missed. Please use the CodePlex GutHub Issue Tracker to let us know what you like us to add and what possible issues you’ve found.

I’ll of course also check for any comments here Winking smile.

3
Jun
Posted by Richard » Add Comment »

Update to BAM Service Generator

I have previously written about how to generate a types set of WCF services to achieve end-to-end BAM based tracking – making BAM logging possible from outside potential firewalls and/or from non .NET based clients. 

In that post I reference a open source tool to generate the services based on the BAM definition files to be able to make types BAM API calls. I called the tool BAM Service Generator and it’s published on CodePlex.

The first version of the tool only made it possible to make simple activity tracking calls, even missing the option to defining a custom activity id. All advanced scenarios like continuation, activity and data reference or even updating a activity wasn’t possible. That has now been fixed and the second version of the tool enables the following operations.

Let me know if you think that something is missing, if something isn’t working as would expect etc.

18
Jan
Posted by Richard » 11 Comments »

End-to-end tracking using BAM services and the BAM Service Generator tool

An integration between systems can often be view upon as a chain of system and services working together to move a message to its final destination.

image

One problem with this loosely-coupled way of dealing with message transfers is that its hard to see if and where something has gone wrong. Usually all we know is that the sending system has send the message, but the final system never received it. The problem could then be within any of the components in between. Tracking messages within BizTalk is one thing but achieving a end-to-end tracking within the whole “chain” is much harder.

One way of solving the end-to-end tracking problem is to use BAM. BAM is optimized for these kind of scenarios were we might have to deal with huge amount of messages and data – several of the potential problems around these issues are solved out-of-the-box (write optimized tables, partitioning of tables, aggregated views, archiving jobs and so on). 

And even though BAM is a product that in theory isn’t tied to BizTalk its still a product that is easier to use in the context of BizTalk than outside of it due tools like the BizTalk Tracking Profile Editor and the built in BAM-interceptor patterns within BizTalk. It is however fully possible to track BAM data and take advantage of the BAM infrastructure even outside of BizTalk using the BAM API.

The BAM API is a .NET based API used for writing tracking data to the BAM infrastructure from any .NET based application. There are however a few issues with this approach.

The application sending the tracking data has to be .NET based and use the loose string based API. So for example writing to the “ApprovedInvoice” activity below would look something like this – lots of untyped strings.

public void Log(ApprovedInvoicesServiceType value) 
{ 
    string approvedInvoicesServiceActivityID = Guid.NewGuid().ToString();
    
    var es = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImportw", 1);

    es.BeginActivity("ApprovedInvoices", approvedInvoicesServiceActivityID);

    es.UpdateActivity("ApprovedInvoices", approvedInvoicesServiceActivityID, 
        "ApprovedDate", "2011-01-18 12:02", 
        "ApprovedBy", "Richard", 
        "Amount", 122.34, 
        "InvoiceId", "Invoic123");
 
    es.EndActivity("ApprovedInvoices", approvedInvoicesServiceActivityID); 
}

One way of getting around problem with the usage of strings is to use the “Generate Typed BAM API tool”. The tool reads the BAM definition file and generates a dll containing strong types that corresponds to the fields in the definition.  By referencing the dll in the sending application we can get a strong typed .NET call. The fact that this still however requires a .NET based application remains.

The obvious way to solve the limitation of a .NET based client – and to get one step closer to a end-to-end tracking scenario – is of course to wrap the call to the API in a service.

BAM Service Generator

BAM Service Generator is very similar to the Generate Typed BAM API tool mentioned above with the difference that instead of generation a .NET dll it generates a WCF service for each activity.  

c:\Tools\bmsrvgen.exe /help 
	
BAM Service Generator Version 1.0 

Generates a WCF service based on a BizTalk BAM definition file.

 -defintionfile: Sets path to BAM definition file.
 -output: Sets path to output folder.
 -namespace: Sets namespace to use.

Example: bmsrvgen.exe -defintionfile:c:\MyFiles\MyActivityDef.xml 
-output:c:\tempservices -nampespace:MyCompanyNamespace

The BAM Service Generator is a command line tool that will read the BAM definition file and generate a compiled .NET 4.0 WCF service.  The service is configured with a default basicHttp endpoint and is ready to go straight into AppFabric or similar hosting.

image

This service-approach makes it possible to take advantage of the BAM infrastructure from all different types of system, and even in cases when they aren’t behind the same firewall! As shown in figure below this could take us one step closer to the end-to-end tracking scenario.

image

“BAM Service Generator” is open-source and can be found here.

23
Apr
Posted by Richard » Add Comment »

Promote properties in a EDI schema using the EDI Disassembler

I’ve doing a lot of EDI related work in BizTalk lately and I have to say that I’ve really enjoyed it! EDI takes a while to get used to (see example below), but once one started to understand it I’ve found it to be a real nice, strict standard – with some cool features built into BizTalk! 

UNB+IATB:1+6XPPC+LHPPC+940101:0950+1'
UNH+1+PAORES:93:1:IA'
MSG+1:45'
IFT+3+XYZCOMPANY AVAILABILITY'
ERC+A7V:1:AMD'
IFT+3+NO MORE FLIGHTS'
ODI'
TVL+240493:1000::1220+FRA+JFK+DL+400+C'
...

There are however some things that doesn’t work as expected …

Promoting values

According to the MSDN documentation the EDI Disassembler by default promotes the following EDI fields: UNB2.1, UNB2.3, UNB3.1, UNB11; UNG1, UNG2.1, UNG3.1; UNH2.1, UNH2.2, UNH2.3.

There are however situation where one would like other values promoted.

I my case I wanted the C002/1001 value in the BGM segment. This is a value identifying the purpose of the document and I needed to route the incoming message based on the value.

The short version is that creating a property schema, promoting the field in the schema and having the EDI Disassembler promoting the value will not work (as with the XML Disassembler). To do this you’ll need to use a custom pipeline component to promote the value. Rikard Alard seem to have come to the same conclusion here.

Promote pipeline component to use

If you don’t want to spend time on writing your own pipeline component to do this yourself you can find a nice “promote component” on CodePlex here by Jan Eliasen.

If you however expect to receive lots and lots of big messages you might want to look into changing the component to use XPathReader and custom stream implementations in the Microsoft.BizTalk.Streaming.dll. You can find more detailed information on how to do that in this MSDN article.

12
Apr
Posted by Richard » 7 Comments »

Use code blocks to extend your BizTalk custom XSLT maps

Update 2010-04-13
Grant Samuels commented and made me aware of the fact that inline scripts might in some cases cause memory leaks. He has some further information here and you’ll find a kb-article here.

I’ve posted a few times before on how powerful I think it is in complex mapping to be able to replace the BizTalk Mapper with a custom XSLT script (here’s how to). The BizTalk Mapper is nice and productive in simpler scenarios but in my experience it break down in more complex ones and maintaining a good overview is hard. I’m however looking forward to the new version of the tool in BizTalk 2010 – but until then I’m using custom XSLT when things gets complicated.

Custom XSLT however lacks a few things once has gotten used to have – such as scripting blocks, clever functoids etc. In some previously post (here and here) I’ve talked about using EXSLT as a way to extend the capabilities of custom XSLT when used in BizTalk.

Bye, bye external libraries – heeeello inline scripts ;)

Another way to achieve much of the same functionality even easier is to use embedded scripting that’s supported by the XslTransform class. Using a script block in XSLT is easy and is also the way the BizTalk Mapper makes it possible to include C# snippets right into your maps.

Have a look at the following XSLT sample:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    xmlns:code="http://richardhallgren.com/Sample/XsltCode"
    exclude-result-prefixes="msxsl code"
    >
    <xsl:output method="xml" indent="yes"/>

    <xsl:template match="@* | node()">
        <Test>
            <UniqueNumber>
                <xsl:value-of select="code:GetUniqueId()" />
            </UniqueNumber>
            <SpecialDateFormat>
                <xsl:value-of select="code:GetInternationalDateFormat('11/16/2003')" />
            </SpecialDateFormat>
            <IncludesBizTalk>
                <xsl:value-of select="code:IncludesSpecialWord('This is a text with BizTalk in it', 'BizTalk')" />
            </IncludesBizTalk>
        </Test>
    </xsl:template>

    <msxsl:script language="CSharp" implements-prefix="code">
        //Gets a unique id based on a guid
        public string GetUniqueId()
        {
            return Guid.NewGuid().ToString();
        }

        //Formats US based dates to standard international
        public string GetInternationalDateFormat(String date)
        {
            return DateTime.Parse(date, new System.Globalization.CultureInfo("en-US")).ToString("yyyy-MM-dd");
        }

        //Use regular expression to look for a pattern in a string
        public bool IncludesSpecialWord(String s, String pattern)
        {
            Regex rx = new Regex(pattern);
            return rx.Match(s).Success;
        }
    </msxsl:script>
</xsl:stylesheet>

All one has to do is to define a code block, reference the xml-namespace used and start coding! Say goodbye to all those external library dlls!

It’s possible to use a few core namespaces without the full .NET namespace path but all namespaces are available as long as they are fully qualified. MSDN has a great page with all the details here.

17
Mar
Posted by Richard » 9 Comments »

Is there a bug in BizTalk 2006 R2 SP1?

Update 2012-03-28
This bug was fixed in the CU2 update for BizTalk 2006 R2 SP 1. Further details can be found here and here.

Update 2010-04-12
Seems like there is a patch coming that should fix all the bugs in SP1 … I’ve been told it should be public within a week or two. I’ll make sure to update the post as I know more. Our problem is still unsolved.

Late Thursday night last week we decided to upgrade one of our largest BizTalk 2006 R2 environment to recently released Service Pack 1. The installation went fine and everything looked good.

… But after a while we started see loads of error messages looking like below.

Unable to cast COM object of type ‘System.__ComObject’ to interface type ‘Microsoft.BizTalk.PipelineOM.IInterceptor’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{24394515-91A3-4CF7-96A6-0891C6FB1360}’ failed due to the following error: Interface not registered (Exception from HRESULT: 0×80040155).

After lots of investigation we found out that we got the errors on ports with the follow criteria:

Reproduce it to prove it!

I’ve setup a really simple sample solution to reproduce the problem. Download it here.

The sample receives a XML file, maps it on the send port to schema made to match the store procedure. It also uses a dead simple tracking definition and profile to track a milestone on the send port.image

Sample solution installation instructions

  1. Create a database called “Test”

  2. Run the two SQL scripts (“TBL_CreateIds.sql” and “SP_CreateAddID.sql”) in the solution to create the necessary table and store procedure

  3. Deploy the BizTalk solution just using simple deploy from Visual Studio

  4. Apply the binding file (“Binding.xml”) found in the solution

  5. Run the BM.exe tool to deploy the BAM tracking defintion.
    Should look something like:

    bm.exe deploy-all -definitionfile:<the path to the solution>\BAM\SimpleTestTrackingDefinition.xml

  6. Start the tracking Profile editor and open the “SimpleTestTrackingProfile.btt“ that you’ll find in the solution and apply the profile

  7. Drop the test file in the receive folder (“InSchema_output.xml”)

The sample solution fails on a environment with SP 1 but works just fine on a “clean” BizTalk 2006 R2 environment.

What about you?

I haven’t had time to test this on a BizTalk 2009 environment but I’ll update the post as soon as I get around to it.

We also currently have a support case with Microsoft on this and I’ll make sure to let you as soon as something comes out of that. But until then I’d be really grateful to hear from you if any of you have the same behavior in your BizTalk 2006 R2 SP1 environment.

26
Feb
Posted by Richard » 5 Comments »

Using XSLT 1.0 to summarize a node-set with comma separated values

Pure XSLT is very powerful but it definitely has its weaknesses (I’ve written about how to extend XSLT using mapping and BizTalk previously here) … One of those are handling numbers that uses a different decimal-separator than a point (“.”).

Take for example the XML below

<Prices>
  <Price>10,1</Price>
  <Price>10,2</Price>
  <Price>10,3</Price>
</Prices>

Just using the XSLT sum-function on these values will give us a “NaN” values. To solve it we’ll have to use recursion and something like in the sample below.

The sample will select the node-set to summarize and send it to the “SummarizePrice” template. It will then add the value for the first Price tag of the by transforming the comma to a point. It will then check if it’s the last value and if not use recursion to call into itself again with the next value. It will  keep adding to the total amount until it reaches the last value of the node set.

<xsl:template match="Prices">
  <xsl:call-template name="SummurizePrice">
    <xsl:with-param name="nodes" select="Price" />
  </xsl:call-template>
</xsl:template>

<xsl:template name="SummurizePrice">
  <xsl:param name="index" select="1" />
  <xsl:param name="nodes" />
  <xsl:param name="totalPrice" select="0" />

  <xsl:variable name="currentPrice" select="translate($nodes[$index], ',', '.')"/>

  <xsl:choose>
    <xsl:when test="$index=count($nodes)">
      <xsl:value-of select="$totalPrice + $currentPrice"/>
    </xsl:when>
    <xsl:otherwise>
      <xsl:call-template name="SummurizePrice">
        <xsl:with-param name="index" select="$index + 1" />
        <xsl:with-param name="totalPrice" select="$totalPrice + $currentPrice" />
        <xsl:with-param name="nodes" select="$nodes" />
      </xsl:call-template>
    </xsl:otherwise>
  </xsl:choose>

</xsl:template>

Simple but a bit messy and nice to have for future cut and paste ;)

22
Feb
Posted by Richard » 6 Comments »

Streaming pipeline and using context ResourceTracker to avoid disposed streams

Recently there’s been a few really good resources on streaming pipeline handling published. You can find some of the here and here.

The Optimizing Pipeline Performance MSDN article has two great examples of how to use some of the Microsoft.BizTalk.Streaming.dl classes. The execute method of first example looks something like below.

public IBaseMessage Execute(IPipelineContext context, IBaseMessage message)
{
    try
    {
        ...
        IBaseMessageContext messageContext = message.Context;
        if (string.IsNullOrEmpty(xPath) && string.IsNullOrEmpty(propertyValue))
        {
            throw new ArgumentException(...);
        }
        IBaseMessagePart bodyPart = message.BodyPart;
        Stream inboundStream = bodyPart.GetOriginalDataStream();
        VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);
        ReadOnlySeekableStream readOnlySeekableStream = new ReadOnlySeekableStream(inboundStream, virtualStream, bufferSize);
        XmlTextReader xmlTextReader = new XmlTextReader(readOnlySeekableStream);
        XPathCollection xPathCollection = new XPathCollection();
        XPathReader xPathReader = new XPathReader(xmlTextReader, xPathCollection);
        xPathCollection.Add(xPath);
        bool ok = false;
        while (xPathReader.ReadUntilMatch())
        {
            if (xPathReader.Match(0) && !ok)
            {
                propertyValue = xPathReader.ReadString();
                messageContext.Promote(propertyName, propertyNamespace, propertyValue);
                ok = true;
            }
        }
        readOnlySeekableStream.Position = 0;
        bodyPart.Data = readOnlySeekableStream;
    }
    catch (Exception ex)
    {
        if (message != null)
        {
            message.SetErrorInfo(ex);
        }
        ...
        throw ex;
    }
    return message;
}

We used this example as a base when developing something very similar in a recent project. At first every thing worked fine but after a while we stared getting an error saying:

Cannot access a disposed object. Object name: DataReader

It took us a while to figure out the real problem here, everything worked fine when sending in simple messages but as soon as we used to code in a pipeline were we also debatched messages we got the “disposed object” problem.

imageIt turns out that when we debatched messages the execute method of the custom pipeline ran multiple times, one time for each sub-messages. This forced the .NET Garbage Collector to run.

The GC found the XmlTextReader that we used to read the stream as unreferenced and decided to destoy it.

The problem is that will also dispose the readOnlySeekable-Stream stream that we connected to our message data object!

It’s then the BizTalk End Point Manager (EPM) that throws the error as it hits a disposed stream object when trying to read the message body and save it to the BizTalkMsgBox!

ResourceTracker to the rescue!

Turns out that the BizTalk message context object has a nice little class connected to it called the ResourceTracker. This object has a “AddResouce”-method that makes it possible to add an object and the context will the hold a reference to this object, this will tell the GC not to dispose it!

So when adding the below before ending the method everything works fine – even when debatching messages!

context.ResourceTracker.AddResource(xmlTextReader);
Next Page »