David M. Scott's Blog

1. Predefined Schedule
2. Predefined number of transactions
3. Predefined number of characters
4. External release trigger

As you can see from the above image, the External Release Trigger option can be selected in the "Interchange Batch Creation Settings" available in the EDI Properties for a BizTalk Party.  The External Release Trigger option allows your application to have complex logic that determines when a particular batch should be released.  To force a batch to be released, a message (with contents similar to the following) must be submitted to BizTalk via a receive location:

<ControlMessage xmlns="http://SQLControlMessage.IssueSelect">
<PAM_Control>
  <DestinationParty>[PartyID]</DestinationParty>
  <EdiMessageType>[0 for X12|1 for Edifact]</EdiMessageType>
  <ActionType>EdiBatchOverride</ActionType>
  <ActionDateTime>[yyyy-mm-ddThh:mm:ss.sss]</ActionDateTime>
  <UsedOnce>0</UsedOnce>
  <ToBeBatched>1</ToBeBatched>
</PAM_Control>
</ControlMessage>

In addition to releasing the transactions of a batching orchestration, control messages can also be used to start and stop batching orchestrations.  So a control message can be used to start, stop or release a batching orchestration.  The message elements of interest are DestinationParty and ActionType.  Set the DestinationParty value with the Party Id you get from the Party name.  And set the ActionType to EdiBatchActivate to start batching; EdiBatchTerminate to terminate batching; and EdiBatchOverride to release the batching transactions.  More information on implementing an external batch release mechanism can be found on MSDN at http://msdn.microsoft.com/en-us/library/bb246108.aspx.

I recently had a request for a utility to simplify the management of BizTalk Parties used for EDI processing.  The BizTalk Administration application already provides a UI for managing EDI Parties.  But there are a lot of EDI details involved in configuring the Parties, especially when using BizTalk 2006 R2's built in batching orchestrations.  I decided to throw something together that would enable a user (with little or no EDI experience) to manage the Parties for a BizTalk group. 

I developed a simple winform utility that displays all Parties configured for a particular BizTalk group.  The utility enables a user to create, update and/or delete BizTalk Parties and their associated data.  Many BizTalk EDI implementations store additional trading partner data in a custom database.  For this utility, the custom database table stores the trading partner's EDI configuration data in addition to data used to map the partner's EDI Id to an internal identifier.

The utility performs the following when creating a Party:

1. Build a Party binding file using a Party binding file template and XPath (or find/replace).
2. Import the Party binding file using BtsTask.
3. Restart the host instance.
4. Get the new Party's Id by calling Microsoft.BizTalk.Edi.PartnerAgreementManager.Partner.GetPartyIdForPartyName.
5. Build a start batching control file using a template and the Party's Id.
6. Submit the start batching control file to BizTalk.
7. Insert the Party's data to the custom database table.
8. Reload the UI data and confirm the Party creation was successful.

The utility performs the following when deleting a Party:

1. Get the new Party's Id by calling Microsoft.BizTalk.Edi.PartnerAgreementManager.Partner.GetPartyIdForPartyName.
2. Build a stop batching control file using a template and the Party's Id.
3. Submit the stop batching control file to BizTalk.
4. Sleep for a configurable amount of time to ensure the Party's batching orchestration is terminated.
5. Use Microsoft.BizTalk.ExplorerOM to delete the Party.
6. Delete the Party's data from the custom database table.
7. Reload the UI data and confirm the Party deletion was successful.

The utility works fine but I'm not thrilled about having to import binding files to create and update Parties.  However, after spending hours searching the web and BizTalk 2006 SDK for alternatives, I'm left with binding files as my only option.  Below are some snapshots of the utility.

Utility initially loads without any data, requiring the user to select an environment (BizTalk Group) to manage:

The user selects the environment to manage and the corresponding Parties are loaded into a DataGridView:

The user chooses to create or update a Party:

The utility provides a simple UI for handling the most basic management tasks for EDI Parties.  I think it is a great tool to simplify EDI Party management in development and test environments.  However, I would not recommend it for a Production environment.

• A View (Winform, Webform, etc.) references its Presenter (custom .NET class) via an interface.  A View calls methods on its Presenter which typically return Model objects used for data binding.  A Presenter can raise events back to its View as needed.  The Presenter is solely responsible for creating the Model in the form of data transport / data binding objects.
• Presenters implement all of the UI logic and call proxy objects to utilize the application's business logic.
• A View instantiates its Presenter using a factory class.  The View never knows the exact Presenter class (just the interface), enabling easy reuse of the View and Presenter.

• PresenterManager - a singleton class that stores references to all active Presenters.  A Presenter requests a reference to another Presenter by passing a unique keyword.  An IPresenter reference to the Presenter is returned (IPresenter is the base interface all Presenters implement).  The returned Presenter is then cast to the specific interface the calling Presenter wants to interact with.  The PresenterManager uses the appropriate factory class to instantiate the requested Presenter.  This is my recommended approach when a Presenter requires another Presenter be active at the time of request.  The Presenter should make sure the reference isn't null after using it the first time.  Another call to the PresenterManager will be necessary if the particular form is closed later on.
• PresenterEventBus - a singleton class that stores event subscription data used by Presenters to raise events to other Presenters that are interested.  This is my recommended approach for optimal loose coupling, when a Presenter isn't necessarily expecting another Presenter to be subscribing at all times.  This enables robust plug in play functionality.

• The View and Presenter can easily be reused in MVP because of a clear separation of function.  The View handles UI presentation and the Presenter takes care of UI logic. 
• Unit testing the UI logic is a straight forward test of the Presenter's functionality.

• Good unit tests are incredibly valuable to developers when it's a large/complex application and documentation is lacking.  They will greatly assist developers making maintenance/enhancement code changes long after the original development team is gone. 
• However, I think comprehensive unit testing can be wasteful when the application has a short lifespan and the same few developers own it from start to finish.

• Multiple types of UI clients (e.g., Winforms, Webforms, Mobile).  The UI logic in the Presenter stays the same with different Views referencing it.
• Different Views of the same data.  Again, the Presenter stays the same.
• Complex UI forms that can serve more than one particular business context.  The View stays the same with multiple Presenters that implement the same interface.

So the future of BizTalk Server is clearly focused on EAI and B2B.  Business Process Management (BPM) will eventually be handled with Oslo and Dublin (~ mid 2010).  Oslo is composed of M (domain model language), Quadrant (visually design domain models), and the Repository (for storing and relating the domain models).  Dublin is the application server (sitting on top of IIS) that is going to execute the components of the domain models, typically WCF services and Workflow services.

Of course, BizTalk will be able to make use of the domain models (business processes, applications, workflows, services, etc.) offered by Oslo and executed by Dublin.  That is because BizTalk is going to be tightly integrated with Oslo and Dublin in future releases (after 2009).  Even BizTalk 2006 R2 and 2009 have limited integration with Oslo.  But notice that BizTalk is just integrating with Oslo and Dublin because they are the clear choice for implementing business processes in the future.  BizTalk will make light use of Oslo/Dublin's BPM functionality for typical EAI and B2B scenarios.  But for enterprise BPM solutions, it will make more sense to implement them with Oslo and Dublin.  And enterprise BPM solutions should call BizTalk (just like any other service) as needed for enterprise integration scenarios (not supported by WCF). 

So we can expect BizTalk to be around for a long time doing what it does best: enterprise scale integration.  XLang orchestrations will still make sense for EAI and B2B scenarios that require complex messaging patterns.  But BizTalk BPM developers will have to plan on building business process logic with Oslo and calling from BizTalk (or vice-versa).  Now is the time for me to say goodbye to XLang orchestrations for new BPM development (while retaining existing investments in XLang).  I am looking to WCF and Workflow Services to implement business process logic and continuing to use BizTalk for integration.

Below are some great resources on the topic: 

  - Workflows, Services, and Models - A great overview of the future of Microsoft BPM solutions (and much more).

  - Oslo Developer Center

  - BizTalk Server Roadmap

This is the final topic I promised to cover regarding EDI.  I'm going to quickly cover three fundamental steps to setting up BizTalk 2006 R2 for EDI processing.  Although there is much more to implementing a successful production solution, these basic steps will prepare you to start receiving and sending EDI X12 documents.  And there is already a plethora of information on the web providing comprehensive details on the subject. 

1.  Create a Party

EDI documents are properly received and sent by BizTalk via Parties.  Parties are used to represent partner agreements.  A partner agreement is a relationship defined between a trading partner and typically an internal entity (i.e., a particular department within the organization using BizTalk). There should be a Party configured for every trading partner / internal entity relationship that BizTalk is expected to process.  Incoming EDI documents are resolved to a particular Party by comparing the document's envelope context with specific Party properties.  A Party's configuration also determines the header and footer content of an EDI document generated by BizTalk.

2.  Configure a 'Trading Partner as Sender'

When a trading partner sends an EDI document to BizTalk, it is known as the sender. The associated Party must be configured properly for Party resolution to succeed.  The four fields (ISA5, ISA6, ISA7 and ISA8) highlighted in the figure on the left are used to uniquely identify the Party.  If BizTalk cannot resolve the Party using all four sender/receiver qualifiers and identifiers, it will attempt to resolve the Party using just the sender qualifier (ISA5) and identifier (ISA6).  This particular configuration represents the case where 'PartyA' is sending X12 EDI documents to 'MyCompany'.

3.  Configure a 'Trading Partner as Receiver'

A trading partner is known as the receiver when BizTalk sends an EDI document to it.  The four fields (ISA5, ISA6, ISA7 and ISA8) highlighted in the figure on the left are used to generate the unique header and footer content for the particular partner agreement.  You'll notice the ISA6 and ISA8 values have switched from the "Trading Partner as Sender" case because 'MyCompany' is now sending to 'PartyA'. 

This post is about as simple as it gets.  It's meant to quickly give you an understanding of the R2 fundamentals for implementing EDI.  Yes, there is more to it, such as X12 schemas, EDISend/EDIReceive pipelines, acknowledgements (functional and technical), message correlation, batching, etc.  But you need to start somewhere.  And if you'll master using BizTalk Parties for EDI configuration, you'll have the necessary foundation to address the remaining components of a BizTalk 2006 R2 EDI solution.

As always, feel free to contact me for additional information.  Rather than reproducing content that is currently available on the web, I'll probably just refer you to another blog or similar.  But if there's a topic that needs more coverage, I'll be happy to post a follow-up that adequately answers the question. 

Thanks for reading and check back soon for some posts covering WCF.

Well, the custom architecture I mentioned in topic 1 of the Simple EDI Implementation with BizTalk 2006 R2 - Overview post is a great idea.  It is certainly the kind of custom infrastructure I would consider recommending for any large companies with true enterprise implementations of BizTalk.  I mean HAT and BAM are extremely powerful but a custom rebuild should provide better performance and ease of use (since it will be customized for the specific business context).  However, the purpose of the sample solution I built was purely for integration between and internal system and trading partners via EDI.  BizTalk 2006 R2 was the clear choice for implementing the integration.  And it just so happens that the new R2 EDI Reporting features combined with HAT were enough to satisfy my initial tracking and monitoring needs.  So while HAT and BAM satisfied my business requirements for this solution, I am certain there are business cases that are best satisfied with a custom implementation of tracking and business analysis.

So let's talk about what I found to be worthwhile in HAT and BAM for tracking and managing my EDI application.  Well, we all know HAT can be great for accessing messages that have been processed by BizTalk.  The UI isn't pretty or sophisticated, but it will get you the data you need assuming you know how to query for it.  And R2's built in EDI reporting functionality is a simple way to track EDI specific messages (and associated acknowledgments).  So what do you have to do to make sure messages are available via HAT and EDI reporting?

HAT works off of the tracking database.  BizTalk needs to have tracking turned on for exactly one host in order for messages to be copied from the message box to the tracking database.  The default host will automatically have tracking turned on but otherwise you'll need to access the properties of one of your active hosts (has active host instances) and turn tracking on as shown below.

Next, you'll need to designate which ports and orchestrations should track messages.  Tracking is expensive for the BizTalk host instance process and the BizTalk databases.  So be selective about the messages you choose to track and how long you plan to keep them in the tracking database.  I wanted to see exactly what I received from the internal system and exactly what I sent them.  I wanted the same data available for the 4010 850 that is sent to the VAN and the 4010 810 received from the VAN.  I wasn't concerned with any of the messages internal to BizTalk, just my inputs and outputs.  That is the information that will be most helpful in resolving processing problems.  The images below display the input/output messages tracked for this solution. 

R2 EDI Reporting is activated for the Parties you want.  Right clicking the Party and selecting "EDI Properties" opens the Party properties screen shown below.  There you can enable reporting and choose to store the transaction sets processed by BizTalk.

So now HAT and EDI Reporting are set for the solution.  Simply processing messages will ensure the proper messages and data are saved and accessible for viewing.  So let's have the internal system send a flat file PO to BizTalk and let the 4010 850 be generated.  Then we'll send back an unrelated 4010 810 and BizTalk will send a flat file invoice back to the internal system.  After sending the 4010 850, I access the interchange report by clicking on the "EDI Interchange and Correlated ACK Status" report link on the Administration Console's Group Overview.

And the results of just sending the one 4010 850 batch are shown below.

Right click the interchange record and select "Transaction Set Details" to see the individual transactions that were batched in the interchange.

Right click a transaction record and select "View Transaction Set Content" to see the individual transaction's content.  I have removed most of this content for privacy purposes.  But you get the idea of how easy it is to see your transaction's content.

So you can see that the R2 EDI reporting functionality is quite nice.  And there is actually quite a bit more it is doing with correlating acknowledgments and reporting on the status.  Now what about the flat file PO received from the internal system.  What did it look like and when was it received.  We can use HAT to view the specifics of the non-EDI messages that were processed.  The flat file PO message received from the internal system is displayed by running the "Messages received in past day" query available from the Queries menu. 

Then you can right click the message of interest and click "Save All Tracked Messages" to save the entire message received.  So using HAT and R2 EDI Reporting, we are able to track and monitor the purchase order requests processed by BizTalk.  Similarly we can track and monitor the invoices processed.

I hope this simple post gives you an overview of the EDI reporting capabilities now available in R2.  With the addition of HAT functionality, you can literally monitor the full lifecycle of messages involved with a EDI integration solution.  If there is a message failure, BizTalk provides the tools to determine exactly what was received and sent.  And that information will be critical to resolving the issue with your trading partner and/or internal system contact.

There is much more that could be covered in this post but this is all I have time for today.  Please let me know if you have any questions or interest in seeing a future post with more details on a particular topic.  Thanks for reading and take care.

I just completed an ‘enhancement’ for my original usage of the BizTalk 2006 R2 EDI batching process.  I found the initial batching approach I was using (per Microsoft documentation) to be cumbersome (in terms of managing the send ports, receive locations and parties). I also found the data returned by the default R2 EDI reports to be confusing.  Just so we're clear, R2 provides some nice EDI reporting features via the BizTalk Admin Console.  The data is stored in BAM, which makes sense because BAM is also handling acknowledgment correlation when sending EDI messages to a Party. BAM has been used to define some basic relationships between the various EDI messages moving through BizTalk.  Although you can get to the raw message data in HAT, BAM is the logical choice to provide an EDI context for the messages.  Unfortunately I was bothered by the EDI report results I was getting based on my initial implementation of batching.

Basically, I was generating EDI messages and then sending them back to BizTalk via a batch receive port.  The corresponding batching receive location uses the EDIReceive pipeline which contains the BatchMarker component.  Using the BatchMarker component is the easiest way to tell BizTalk to batch a particular message.  So I sent the EDI transaction messages (that BizTalk had just generated) to the batch receive port and they were picked up by the corresponding batch orchestration.  Well a batch orchestration is created for each Party that has batching turned on.  In the Party you set a filter for which messages the orchestration should subscribe to.  But the Party filter actually works indirectly via the BatchMarker component.  The BatchMarker component actually evaluates all of the batch orchestration filters for a match.  If it finds a single match, three important context properties are promoted and set as follows:  EDI.ToBeBatched = True, EDI.DestinationPartyID = %PartyID%, and EDI.EncodingType = X12 (0) or EDIFACT (1).  And technically the batch orchestrations are subscribing to messages based on that filter criteria (where %PartyID% is the Party ID of the batch orchestration).

Well just as the Microsoft documentation promised, my EDI X12 transaction (ST) messages were properly batched and organized in interchanges (ISA) and groups (GS).  However, there was a glaring problem when I went to look at the EDI reports.  I used the EDI Interchange report to query for 4010 850 interchanges sent to a particular trading partner.  I sadly discovered that not only were my batch interchanges returned but also the individual transactions that I sent to the batch receive port.  So when I used the EDISend pipeline to send the messages to the batch receive port, they were sent as complete interchanges (w/ header and footer).  Hence, BAM captured the activity as valid and stored it for later reporting.  But really it's just the batch interchanges that I wanted returned, because those are the messages actually sent to the trading partners.  And of course right clicking on a batch interchange record allows you to select "Transaction Set Details" and view all of the individual transaction messages that were part of the batch.   So basically, the EDI reports were showing me duplicates of each transaction message that was wrapped in an individual interchange envelope.  The truth is BAM is doing just what it's supposed to do, but the simple approach to batching just isn't acceptable if you want clean EDI reports.

So it was clear I could no longer send the individual transaction messages from BizTalk to the batch receive port.  Because that send and receive process was giving BAM unnecessary information about the EDI messages that wasn't meant to be tracked.  So the clear answer was so send the messages directly to the message box and let the batch orchestrations work their magic.  Not only would the new approach clarify reporting, it would also simplify the process of creating a Party with batching.   The following is the original process I was using to create a Party with batching:

1. Create a file directory for the new Party’s interim batch messages (i.e., ..\FileDrop\PO\4010_850\PartnerA)
   • The Party specific folder (i.e., PartnerA) had to be the exact same as the EDI Identifier for the Party.
2. Create the Party and configure batching.
   • A filter must be set for messages received at the Party’s interim batch message directory.

3. Create a batch send port with a filter set for messages with the Party’s name.
4. Associate the send port with the Party.
5. Create a batch receive location to monitor the Party’s interim batch message directory.
6. Add the Party’s information to the custom Party table using SQL Server 2005.

Although the process isn’t too bad and can be automated, I think it will be a maintenance nightmare for implementations with numerous trading partners.  That's because you’ll have the following unnecessary artifacts for each new Party using batching:

• Send Port
• Receive Location
• File Directory

In my opinion, a much cleaner solution would allow you to simply create the Party (w/o a batch filter) and add a new record to the custom Party table.  This should improve the appearance of the BizTalk Admin’s ‘Send Ports’ and ‘Receive Locations’ views.  Not to mention you will no longer have a bunch of Party specific folders in the ‘FileDrop’ directory.  So I just needed my 4010 850 orchestration to send directly to the message box after setting and promoting the following:  EDI.ToBeBatched = True, EDI.DestinationPartyID = %PartyID%, and EDI.EncodingType = 0 (for X12).  Well my custom Party table is used to translate internal trading partner IDs with EDI Party Names.  So a quick database lookup in my map and I have the correct Party Name for each transaction message.  Then there is a nifty static method in the Partner Agreement Manager API (Microsoft.BizTalk.Edi.PartnerAgreementManager.dll) that allows me to get the Party ID.  The Partner class (in the Microsoft.BizTalk.Edi.PartnerAgreementManager namespace) has the static GetPartyIdForPartyName method.  So it's very easy to set the correct property values and send it to the message box.  Oh, but wait, those properties have values but they're not promoted yet.  I then created a correlation set containing the three EDI properties.  I initialized the correlation set with my first Send shape and I 'followed' the correlation set with my subsequent looping send shape.  Now the properties are set and promoted and the batch orchestration corresponding to the Party ID will receive the message.   The code below provides a glimpse of what needs to be done before sending a transaction message to the message box for batching.

Sample Expression Shape Code

transCounter = transCounter + 1; partyID = Microsoft.BizTalk.Edi.PartnerAgreementManager.Partner.GetPartyIdForPartyName(partyName);

Sample Message Assignment Shape Code

msg850.ST.ST01 = "850"; msg850.ST.ST02 = transCounter.ToString(); msg850.SE.SE02 = transCounter.ToString(); msg850.SE.SE01 = ""; //  A whole lot of xpath count() functions msg850(EDI.DestinationPartyId) = partyID; msg850(EDI.ToBeBatched) = true; msg850(EDI.EncodingType) = 0;

Although that's the gist of how to route messages directly to the batch orchestrations, there are still some other issues to address to make it work.  The batch orchestration is going to expect valid EDI transaction messages.  That means the ST and SE child elements must contain valid data.  You can use dummy data for the SE01 element if you don't mind a lot EDI warnings in the application event log.  But the other elements should contain valid data in order for the batch orchestration to do its job.  Let me know if you have any problems with this and I'll post a follow-up for clarification.

BTW, if you haven't had a chance yet, check out the EDI Solution Architecture documentation on MSDN.  It is a great resource for digging deeper into the R2 EDI internals.

I've decided to share some information about basic EDI development using BizTalk 2006 R2.  I haven't had the pleasure of using the Covast EDI adapter so I can't make a comparison.  But from my personal experience, I can say R2 really makes the fundamentals of EDI straightforward to implement.  Of course there are some features I'd like to see added (some coming with 2009) and I'll get to those in a later post.  But really, the majority of complexity in a simple EDI integration is going to be in the message transformation (mapping).  An implementation could require custom adapters and/or pipelines (beyond the typical flat file pipeline), but that isn't the core of the EDI solution.

Again, the generic solution I am presenting is meant to be a simple EDI implementation, so it makes sense that mapping would be the most complex task.  Below is a high level view of the simple system I'll be discussing over the next few posts.  It is a simple purchase order request to invoice response.  And as you can see from the diagram, an internal system sends a flat file purchase order request to BizTalk.  The purchase order is transformed in to a 4010 850 EDI document and sent to the Value Added Network (VAN) where it is received by the trading partner.  The trading partner sends a 4010 810 back to the VAN and it's received by BizTalk.  The 4010 810 is transformed to an invoice flat file and forwarded to the internal system.

So this is the solution I'll be discussing over the next few posts.  I plan to cover the following topics:

1. The initial vision for the solution and what I ended up building.  I had hoped to utilize WCF, WF and SharePoint to build a much more user friendly solution.  The idea was to turn off BizTalk Tracking and build a simple application for viewing messages, managing partners (Parties), and resubmitting failed messages.  So with BizTalk Tracking turned off, messages would be sent via a WS-HTTP WCF adapter to a WCF service for processing.  WF workflow services (using the WF Rules Engine) were being considered for implementing the business mapping rules that would typically end up in a BizTalk map.  A custom SQL Server database would be used to manage message, exception, and partner data.  SharePoint (WSS 3.0 or MOSS 2007) with custom web parts would provide a user friendly user interface.  All that sounds great and is even moving toward to OSLO, but is it really practical?  Well that is the question I had to answer and I'll discuss it in an upcoming post.
2. The basic concepts of EDI you have to know to be successful.  BizTalk 2006 R2 makes this incredibly easy by using a Party to represent each partner agreement (typically between the internal system's EDI ID and the trading partner's EDI ID).  But of course internal systems can have multiple EDI ID's which can really complicate Party management.   In the end, you just need to be able recognize and generate valid headers and footers for the EDI documents that BizTalk processes.  An upcoming post will cover this topic in more detail.
3. Batching EDI documents with BizTalk 2006 R2 is really easy.  But is the implementation clean and does it work the way you would expect.  I found the initial batching approach I was using to be cumbersome (in terms of ports and parties) and it was returning confusing data using the default R2 EDI reports.  I spent a good day trying to get a better understanding of how BizTalk 2006 R2 Batching really works.  I still have more to learn.  But I've come up with a more manageable approach to ports and parties associated with batching.  And the final results in the EDI reports are exactly what you would expect.  I'll discuss this in an upcoming post.

Thanks so much for taking the time to check out my blog.  I look forward to sharing more in the near future.