EAI with BizTalk Server 2004

Applications built with loosely coupled architectures need middle-tier or chestrators to broker communication and authentication among other things between points. MS BizTalk Server is one such EAI facilitator

Sunday, December 18, 2005

Over the past several decades, businesses have invested in software systems; purchased or tailor made for their needs. When these systems were commissioned, they could easily serve the business purpose and assist in various business initiatives and decisions. However, businesses around the globe have now rapidly shaped themselves up around IT. Previously, when such disparate systems were developed for independent business units, very less thought was given to their interoperability with systems in other business units, leave aside external organizations.

Most of these systems follow a tightly coupled model leaving very less of it for interaction with other systems. The process formulated across the organization, were dependent much on manual efficiency. Today, business requirements mandate integration of such applications to speed up the processes and facilitate quick decision-making, thus, assisting in potential growth.

In light of these facts, application-integration projects today demand a decoupled approach that would involve passing of messages between the systems. Organizations also need to reshape their processes frequently based on the user feedbacks, fuelling the need to maintain the lead in the competition. This poses an additional responsibility for the application integration projects that they should easily adapt to the changes in business processes.

MS BizTalk Server 2004 addresses these issues with amazing ease and flexibility. It is designed to work as a middle tier EAI tool to assist developers in building quick, robust solutions for application integration.

In this article, we'll glance through the fundamentals of MS BizTalk Server 2004 and see how it helps solve issues involved in typical EAI projects. You may be a developer looking forward to work with BizTalk Server or a technical decision maker who might be thinking about harnessing BizTalk for the next project, or even a technology enthusiast simply curious about the technology. Therefore, you will explore the basics of how BizTalk Server helps connect disparate applications, creating business value.

How does it help?
BizTalk Server 2004, like its predecessors, leverages the XML standard as a ubiquitous platform for a message-oriented architecture. Each piece of information is communicated over the boundaries of external systems and represented as a serializable XML message, hence, abstracting the complexities that might be caused by following the proprietary application formats.

It implements this message-oriented middleware using the SOA (Service Oriented Architecture) fundamentals. The SOA harnesses the XML technology to create a layer of services that allows applications to seamlessly integrate the functionality in a loosely coupled way. The SOA allows heterogeneous applications to communicate by leveraging the Internet as the ubiquitous transport.

Leveraging this architecture, when the BizTalk server enters the integration arena, the first and the most common challenge that it faces is about the data formats that are supported by the end systems.

To assist in transformations of the format of information from proprietary formats to the XML message, BizTalk provides a flexible channel termed as a 'pipeline.' A pipeline in BizTalk performs several functions when a chunk of information enters or leaves the BizTalk playground. At both these points, the pipelines are responsible for converting the data from the end system format to a simple XML message that is understood by the BizTalk server or vice versa.

Additionally, the pipelines also serve the purpose of validating the users of the system along with other things such as encrypting or decrypting the data as it enters or leaves the system. This leaves us without the complexities of knowing the format in which the information is present in the end systems.

With this done, we focus on how the architecture manages the flow of these 'messages' between the end systems. For any system to communicate with another, it has to send the information by some means to the target system. In the typical object-oriented world, this would probably mean invoking some API on the target system to pass the required information. However, in a more loosely coupled approach, this would mean transporting the information over some transport channel that will be agreed on between the two applications. BizTalk, by default, supports several such transport channels including HTTP/S, FTP, SMTP or even simple flat files.

In the BizTalk paradigm, an 'adapter' is responsible for picking up and writing the information chunks to or from the transport type that is understood by the end systems. Therefore, to complete the picture, as the adapter picks up the information, it is sent over for the pipeline to process. Once processed by the pipeline the BizTalk business processes can consume the resultant 'message'.

The business processes can be orchestrated with help of the BizTalk Server orchestration designer that allows the developers to graphically design the processes as a flow chart, thus, considerably reducing the development time. This also helps in significant cost reduction when changing the processes. Almost for all the instances, the message that arrives from the source system needs to be changed or slightly altered to suit the target system. That's for an obvious reason that the systems being integrated might have different purposes, probably designed by totally different teams. To achieve these transformations in the business processes, BizTalk Server provides the 'mapper' tool that along with the powerful set of functoids allows transformation of messages from one format to another that may be required during various stages in the process. The 'mapper' has been carried over from its predecessors, albeit now more tightly integrated with the Visual Studio environment making it easier to use.

After understanding with a birds eye view of how things really work down under, we can conceptualize the BizTalk Server in two parts. One of these will be responsible for exchanging the information with the end systems, thus, enabling the communication. While the other part will be responsible for formulating the workflows that allows these end systems to actively participate for the business.

At the core of its functioning, BizTalk relies heavily on the MS SQL Server 2000 as a store for all its data pertaining to the current configuration and the information that is required to execute the business processes. As a result, the database server is a prerequisite for installation and configuration of the BizTalk Server. Core to the concept of message handling in BizTalk Server 2004 is the 'message box', it enables the server to provide the reliability and robustness that would be of prime importance in the loose coupled architecture. An SQL Server database with its set of tables and stored procedures forms the core component of the message box. This is supplemented by the BizTalk core engine, which takes up the responsibility of managing all the things together along with the message box. Each message being processed in the BizTalk Server is recorded in the message box. Along with this, it also stores the orchestration states at each stage when a business process is being executed. The combination allows the server to recover from any system errors that might hinder the operations, thus providing a reliable and robust system. The end systems participating in the process might often be disconnected physically and would not be able to resend a message if it is somehow lost during processing. Thus, it becomes imperative for BizTalk as the middle tier to expect the message only once. In case of any errors in processing, the end system might not be able to re-send the messages, thus resulting in loss of data and revenue. The message box architecture helps overcome this by tracking every message that is processed within the boundaries of the BizTalk Server.

The SQL Server is also responsible for storing information about all the artifacts being deployed on the BizTalk server. BizTalk interfaces with this information with a management console that allows easy deployment and maintenance of the deployed processes. However, the most significant feature achieved with the help of the SQL Server is state management. The BizTalk orchestrations being asynchronous in nature and relying on various end systems can be running over days or months or sometimes even a couple of years together, while expecting some information from one of the systems. In such scenario, it becomes expensive for the BizTalk server to maintain the orchestration running in the memory. Hence it serializes the state of the orchestration of the SQL Server-a process commonly known as Dehydration; and removes the orchestration from the memory, just to be reloaded when it is actually needed (obviously called the Hydration process).

BizTalk mapper tool, with its powerful set of functoids, allows easy transformation of messages from one format to another

Along with the robust and reliable message transportation with a decoupled architecture, BizTalk provides processes support for transactions and rollback procedures. It leverages the .NET framework to provide programmable interfaces for its various components, thus extending its capabilities to suit the concerned systems.

Unlike its predecessors, BizTalk 2004 provides a fairly simple set of APIs that can be used to design custom components for interacting with applications. For example, let's consider a line-of-business application that is not capable of providing data in any of the common formats supported by BizTalk. Instead, it has a set of COM APIs that need to be executed for interactions. Here, you can easily leverage the BizTalk Adapter Framework APIs to create a custom adapter that will be responsible for interacting with this LOB application. The adapter will be responsible for interacting with the end system and passing over clean XML messages to the BizTalk process to handle. Thus, the actual business process implementation is isolated from the API level complexities involved with the line-of-business application. Similarly, the BizTalk server provides programmable interfaces for creating custom pipelines and mapper functoids as well.

Defining your own rules
The BizTalk server suite introduces the concept of Rules Engine. It has been observed that the most frequent changes required by the businesses are related to the business rules. For instance, payment system may require approval from the finance manager if the amount exceeds $10,000. As the business progresses, this may be changed to mandating the approval only in case the amount exceeds $25000. Considering these facts, the BizTalk Server has an in-built Rules Engine that helps separate out the business rules from the business processes. Any changes to such rules can be deployed without changes to the actual business process.

These rules in BizTalk are created with the help of 'Facts'. Facts are bits of information that can be collected from various sources like database tables, .NET assemblies or XML files. This information collected at runtime helps the Rules Engine evaluate the rules.

To interact with the Rules Engine, BizTalk also offers a Rules Engine Composer-a graphical and easy-to-use interface to design the rules being used by the BizTalk business processes. The Rules Engine composer provides a simplistic interface for designing the rules, thus, helping even the business user modify the rules and cut down drastically on the maintenance costs. Rules Engine also exposes a set of .NET interfaces that allows the Rules to be used directly from .NET code. Thus, the Rules Engine can also act as a standalone component that can help the .NET applications maintain the rules in easy way without involving the BizTalk Server in picture.

The Rules Engine lets business analysts build business rules using a simple user interface

Users can interact
Now let's take a careful look at the example that we used for demonstrating the rules. Within the process, the payment requires approval from the finance manager. However much the business automates, it still requires a human factor to make certain decisions, BizTalk Server 2004 acknowledges this by introducing the Human Workflow Services that allow the decision makers to be a part of the process by interacting through common user tools such as MS Office applications, Infopath or a Sharepoint portal. The Human Workflow Services framework allows this by defining 'actions' that are triggered by the users or the decision makers by performing activities pre-defined in the process. The fundamentals of the Workflow Services lie in the fact that the processes are just the 'Actions' or activities that are performed in certain predefined order. Hence, a developer can define the actions, whereas the way the actions are executed may depend entirely on the user performing them. That is, the users can dynamically define the workflows as they go on working with the system. The workflows can also communicate with the user by notifications that can be sent by e-mail, Sharepoint portals or even the MSN IM.

Single sign-on
Irrespective of the defined workflow, a BizTalk process, as an integrator, will always be involved in talking with various end systems for communicating the required information. Often such disparate systems that have been developed to take care of the security mechanisms in their own way. Interacting with these applications requires the user to log in using specific credentials. Most often, these credentials are also responsible for defining the things that the user can do with the system. We know this using a common term called 'Authorization'. These credentials, therefore, are used for authentication and authorization of the user for a particular system.

When dealing with these disparate end systems that do not leverage any of the common authentication mechanisms like, say, the Windows Active Directory, it is imperative for BizTalk as a middle tier to use the appropriate credentials that will help it gain access and perform the required functions. Most often the credentials that are to be used depend on the user who is issuing the request to BizTalk. Hence, it is required that the BizTalk Server maintains a store for the credentials that allow mappings between the users of the system and their credentials on the LOB Applications in a secured way.

The Enterprise Single Sign-On, commonly known as SSO that is bundled along with the BizTalk Server, serves the purpose of this secured store. Along with this, the SSO leverages its architecture for allowing storage of configuration settings that can again be mapped with users and their applications. The SSO provides its own .NET interfaces that allow interactions like creating applications and mappings, and retrieving them. The BizTalk adapter framework can further leverage this interface, allowing developers to write adapters that provide secure access to the credentials for the LOB applications.

In effect the MS BizTalk Server 2004 provides a robust, reliable and secured platform for integration of the applications providing the power and flexibility much needed for the business problems today.

Sanket Bakshi, Sr Technical Associate, Mahindra British Telecom

Page(s)   1