Talking BizTalk

BizTalk could be described as a pre-built Service-Oriented Architecture or as an Enterprise Service Bus, but it is rather more than either of these two descriptions suggest. In fact it is such a versatile tool that it is very much what you want to make of it. Perhaps it is best considered as an example of a very high level language or a domain specific language. It’s essentially about expressing the implementation of a business workflow, and as such it has strong connections with Business Process Execution Language (BPEL). Of course you could do anything that can be implemented in BizTalk by writing a custom application, so the real question is: does it make it easy to implement such systems? In this article we take an overview of the architectural framework that BizTalk provides, and see how to make use of it in a simple example.

What BizTalk does

BizTalk has two major functions. First, it acts as a ‘conveyor belt’ for data. It can accept data in many different formats, convert it into a standard XML representation, and then convert it into some other format. This provides a degree of interoperability that makes BizTalk valuable as an integration server. Put simply you can use it as a layer between incompatible systems. Secondly it can process the data while it is in the standard XML format using an “orchestration”. An orchestration is a procedural description of what should happen to the data on its way through the system. Earlier versions of BizTalk used XLang, a proprietary language, but the latest version, BizTalk 2006 also supports BPEL. In most cases, however, you aren’t going to be writing XLang or BPEL as a way of creating an application.

The detailed structure of BizTalk can be seen in Figure 1.

Figure 1: The BizTalk architecture

It is essentially a graphical programming tool implemented as extensions to Visual Studio. The split between the BizTalk server and the VS development environment makes it harder to fully appreciate how BizTalk development proceeds. Later in this article you will find a “Hello World” entry-level example to get you started.

As already described, the basic BizTalk algorithm is that a message arrives, is converted to XML, processed, and converted from XML to another format. The term “message” really relates to the style of processing of the data rather than to any messaging infrastructure. In this context a message arriving can be as simple as a file being stored in a specified directory. The “message” will then be read by BizTalk, and moved on through the processing pipelines and out to the destination which could be as simple as another folder. There is a range of “adaptors” which connect the pipelines to message sources or sinks. The standard ones included with BizTalk are File, FTP, HTTP, Base EDI, MSMQ, SMTP, SOAP, SQL, WSE and WebSphere MQ. New in BizTalk 2006 is the SharePoint adaptor, which allows SharePoint documents to be sources and sinks of orchestrations. Others are available from third party suppliers or you could always write your own custom adaptor using the adaptor framework.

An adaptor simply has to know how to receive or send the message using the protocol it is intended to work with. That is, an SMTP adaptor “knows” how to retrieve and send email but doesn’t concern itself with the details of the message. The only other detail of the adaptor is the receive or send “location”, i.e. the actual source or sink for a message such as a folder, email address, URL and so on.

Converting the message to and from XML is the job of the pipelines. These consist of a set of components which perform standard tasks that normalise or unnormalise the message. This includes converting from one character encoding to another and converting “fields” in the message into XML and vice versa.

At the end of the pipeline sits a message port, which can perform mapping of fields before passing the message on to the MessageBox, which is simply a SQL database. From here it is either sent to an orchestration or directly to a send port. The orchestration processes the message and can transform it, generate new messages, re-route it based on content, and so on. Any messages that are passed to send ports are processed in the same way by a pipeline and a send adaptor.

You might think at first that this is a very limited architectural model, but this would be overlooking the power that the adaptor/pipeline model provides. Given the availability of adaptors such as SOAP or SQL, the model includes creating and consuming web services and database processing. Of particular note in BizTalk 2006 is the new ability to work with a SharePoint web site. This is exciting because it provides the opportunity to build systems that use SharePoint as a collaborative user interface to a BizTalk system.

Also notice that the implementation of the architecture is scalable. For example, a new instance of the orchestration is created for each message as it arrives. The intent of BizTalk is to provide a mechanism that makes integration easier, especially if you don’t have access to the source or sink of the messages. Using a transformation model, a single orchestration that describes the workflow or business logic can be applied to a range of message sources and sinks. It is in this sense that BizTalk is a very high level language.

Getting started

Trying out BizTalk is relatively easy, as you can download a trial version of BizTalk 2004 and/or the current Beta of BizTalk 2006. From a developer’s point of view BizTalk 2006 has so many advantages that it is certainly the one to try, even if it is still in beta. Its integration with Visual Studio is much better, and, perhaps more importantly for anyone wanting to gain some experience of BizTalk, it provides simple redeployment, which makes a short development cycle possible.

Before installing BizTalk make sure that you have applied all of the service patches and updates to the operating system you are going to be using. A fresh install for all of the software on a dedicated test machine is also a good idea if you want to avoid configuration problems – machines are cheap compared to your time! Install Visual Studio 2005 and SQL Server 2005 (the copy of SQL 2005 Express that comes with Visual Studio works well). If you want to try out any of the SharePoint aspects of BizTalk you also need to install it first. Once you have Visual Studio installed, and have checked that it is working, install BizTalk 2006 – it should be able to detect any missing software components and download and install them for you. It will also detect which BizTalk modules aren’t worth installing because the system lacks (say) SharePoint. After installation a new configuration tool can be used to set up the databases etc. This is usually just a matter of supplying a suitable username and password with the correct permissions to allow BizTalk to create the tables it needs.

To try out BizTalk development you only need to get the BizTalk Runtime and the Business Rules Engine correctly configured. You can keep going round the configuration loop in an effort to correct any problems by supplying new settings (see Figure 2).

Figure 2: A configured BizTalk

It has to be said that the error messages and problem diagnostics facilities are not good (even in BizTalk 2006), and you could well find yourself wondering why some feature or other doesn’t quite work.

First steps with BizTalk

Although there are some BizTalk tutorials, and even hands-on workshops, they aren’t well thought out for the complete beginner. One of the problems that the writers of tutorials seem to have is thinking small enough. The desire to want to show off the product doing something useful often results in first examples that are simply too big and complicated. This is especially so with BizTalk as there is enough to learn in terms of new tools and new ways of working that any additional complexity in the example is unhelpful. The beginner needs to go through the development cycle as quickly and as painlessly as possible. This is, of course, what the “Hello World” example was invented for, and you can treat the following example as a “Hello BizTalk World”.

The simplest BizTalk application I can think of accepts an input message in the form of an XML file in an input folder, passes it to an orchestration, which in turn transfers it to another folder. As the message is a file we can use the File adaptor, and as it is in XML we can use the default “almost do nothing” pipeline. The example could actually be made simpler by dispensing with the orchestration, but as almost any application you are likely to build will have an orchestration, we’ll keep that feature in, but it will be the simplest possible.

A schema

XML schemas have a number of uses in BizTalk, but an important one is in describing the XML structure of both input and output messages. In this case we need to create a single schema to define the single message that we are going to work with. Start Visual Studio and open a new Empty BizTalk Server Project. If you can’t find this type of project in the New Project Wizard then you probably haven’t installed the BizTalk development components on the machine that VS is running on. Call the project HelloBizTalkWorld and store it in the root of the hard disk for simplicity (see Figure 3).

Figure 3: Creating a BizTalk project

To create a new schema, use the command Project, Add Item and select the schema item. Give the new schema the name Greeting. When the Schema editor opens you will see the schema definition in the right hand window (see Figure 4).

Figure 4: Adding a schema

For most of the time however we can avoid manually editing the XML and use the hierarchy diagram in the window on the left to modify the schema – and this is exactly how it should be!

You can think of defining the schema as creating a record of data fields. Each field has a name and a data type. In this example we will create a record with a single text field – Salutation. In general the schema would define many more fields, and would constrain the data types more precisely.

To add an element node to the schema, right-click on the root node and select Insert Schema Node and then Insert Child Field Element. Give the new element the name Salutation and leave its data type as the default – i.e. a string. It is also usual to rename the <root> element to give the data record a meaningful name, but this is unnecessary in this simple example. Save the schema before moving on to the next step.

You will also need a message to test the application with, and the simplest thing to do it to use Visual Studio to generate an instance from the schema. To do this right-click on the .xsd file in question in the Solution Explorer window and select Generate Instance. If you load the resulting file back into Visual Studio you will see:

<ns0:Root xmlns:ns0=
”http://HelloBizTalkWorld.greeting”>
<Salutation>Salutation_0</Salutation>
</ns0:Root>

Change “Salutation_0” to “Hello BizTalk World” (see Figure 5).

Figure 5: The completed schema

Orchestration

Now we can get started on the orchestration. Use the command Project, Add Item to add an Orchestration file to the project and you can accept the default name. When the orchestration opens you will discover that you now have an Orchestration View window to help you. Building an orchestration is a matter of defining messages, ports and so on and these tasks can be done in almost any order. In this case let’s start by creating the single message that we need to use from the schema that we have already prepared. Right click on Messages in the Orchestration View window and select New Message. This creates a message with the default name Message1. To associate it with the schema, you next move on to its properties, and click on message type. You should be able to select the schema we just created from the drop-down list. Now we have a message object defined by the data structure specified in the schema.

Next we need two ports – one to receive the message and one to send it on its way. Ports are placed on the orchestration diagram in two special areas to the right and left of the main diagram – labelled “port surfaces”. You can add a port by dragging the port icon from the toolbox onto the surface. When you drop the port the configuration wizard opens. In this case we will leave the ports with their default names. Port_1 is configured as a receive port with the dialog box properties set to: Create a new port type, One way, and Internal, limited to this project. On the next dialog page select “I will always be receiving messages on this port”. We could bind the port to a location when we deploy the project but in this case there is no advantage to doing so. If you select Specify Now you can fill in the URI to specify the input file:

C:\HelloBizTalkWorld\Input\*.xml

…and set the Transport to File (see Figure 6).

Figure 6: The input port configuration

Repeat the process to create a second port but this time with the direction set to “I’ll always be sending messages on this port” and with the URI set to:

C:\HelloBizTalkWorld\Output\
	%MessageID%.xm

For both ports we can use the default pipeline, which processes XML in and out.

The ports deal with the transport of the message but now we need a Receive component and a Send component to process the data in the message. To add these you simply drag a Send and a Receive component onto the “flowchart” in the middle of the diagram. The orchestration at this point should look like Figure 7.

Figure 7: The nearly finished orchestration

All that remains is to associate the send and receive components with the message and connect them to the appropriate ports. If you select Receive_1 and click on Message in its properties you can select Message_1 from the drop down list. You have to repeat the process with Send_1. The Receive component also has to be set to “activated” in the properties window. Finally you connect up the ports and components by dragging from one connector to another. When all of this is done you should have an orchestration that looks like Figure 8.

Figure 8: The final orchestration

There should be no exclamation marks visible on any component or property to indicate that there is still something to do.

Build and deploy

Building the project is almost like building any project in Visual Studio, but with the extra requirement that the assembly has to be given a strong name. To do this we first have to generate a key pair. While there are easy ways to do this in other project types, at the moment a BizTalk project still needs us to use the command prompt. To create a key pair file, start the Visual Studio command prompt, change directory to the location of the project and use the command:

sn -k HelloKey.snk

This stores a key file called HelloKey.snk in the project directory. Next return to Visual Studio, right-click on the project file in the Solution Explorer and select Properties. In the project properties window that appears navigate to Assembly, Strong Name, Assembly Key File and enter the location of the key file you have just created (you can browse to it). While you have the project properties open it is worth navigating to the Configuration Properties, Deployment section and making sure that the Redeploy property is set to True. Without this you cannot redeploy the project after making changes.

Now everything is set to build and deploy the project. Use the Build, Build or Build,Deploy command and with luck you should see no error messages on the build and after a brief pause no errors in the deployment phase.

The next phase in the deployment takes us to the BizTalk server. You can either open the BizTalk Explorer in Visual Studio or open the BizTalk Server administrator console from the Start bar. It is probably easier to use the Administrator because you can first check to make sure that an instance of BizTalk is running with the command Platform Settings, Host Instances. If there isn’t, create one and start it running. The deployment by Visual Studio will have added an orchestration, a send port and a receive port, which you will be able to find if you expand the Application, All Artifacts branch of the tree.

You now have to configure these within BizTalk. The main task is to bind the orchestration to a host instance that will be responsible for running it. Right-click on the orchestration, select Properties and in the properties window select Bindings. You should see that it is already bound to the two ports that you created.

You now have to add a binding to the Host BizTalkServerApplication and close the property window. Next right click on the orchestration and select Enlist. You also have to repeat the process to enlist the send port and enable the receive location. Finally when the send port and the orchestration show as “Enlisted” you can start each of them in turn by right-clicking and selecting Start. The receive port doesn’t have to be started; just the receive location enabled.

Now if you move back to Visual Studio and save an instance of the schema that you created earlier into the folder “input” you should see it appear, then disappear and reappear in the “output” folder. This is your first BizTalk application.

Moving on

Of course this example is very simple, but as a developer you should now be in a position to see how to build something more complicated. To demonstrate how to redeploy the project, let’s add a count field to the message and make a decision based on it. First we need to modify the message schema. Simply add another child element node called count and set its type to integer. As we want to make use of this element within the orchestration we need to promote it to a “distinguished field”. To do this, right click on a node and select Show Promotions. Then add the Count field to the Distinguished fields list. Save the schema and generate an instance, which should look something like:

<ns0:Root xmlns:ns0=
”http://HelloBizTalkWorld.greeting”>
<Salutation>Salutation_0</Salutation>
<Count>1</Count>
</ns0:Root>

Now we can modify the orchestration to make a decision based on this new field. Add a Decide component, an additional port and a Send component. Create a new directory called Output2 and configure the new port to write to it in the same way you created the first output port. Associate the Send component with Message_1 and link it to the new Port. Now drag the elements of the orchestration so that they look like Figure 9.

Figure 9: The new orchestration

Finally double click on “rule one” and enter the Boolean expression:

Message_1.Count>100

Now you can build and deploy the project. The documentation says that it might be necessary to restart the host instance to make the changes take effect. In practice it is so often necessary that you might as well do it by default. Notice that it is the host instance that needs to be restarted and not just the orchestration, ports, etc. If you don’t do this you can discover that you are still working with the old version of the project even though you have deployed a new version. If redeployment has worked you should now be able to save XML messages in the input folder and watch as they are moved to either Output or Output2 according to the value in the Count field.

You can carry on experimenting with this example and create your own to discover how BizTalk works. It’s not difficult, but it can be frustrating as there are lots of different small ways in which your project can fail to work as expected, and the help and documentation are still poor. The one big difference between our simple project and something that you are likely to want to put into practice for real is that it only uses a default pipeline. In nearly all cases you are going to have to create a custom pipeline to convert the message to and from XML. Building a custom pipeline isn’t difficult – but it is another story.

---

Dr Mike James has over 20 years of programming experience, both as a developer and lecturer, and has written numerous books on programming and IT-related subjects. His PhD is in computer science.