Modis Whitepaper | Are You Connecting?

Effective design of asynchronous systems for the enterprise – use of the Hohpe and Woolf Pattern Language.

Are your system integration projects delivering on time? Or are you experiencing repeated delays in delivery of your Enterprise Service Bus (ESB) and Service Oriented Architecture (SOA) enabled projects?

ESB and SOA depend critically on the concepts of messaging and queuing. In particular successful system integration projects depend upon a deep knowledge of asynchronous messaging and queuing concepts and being able to represent these in design documentation. If your designs do not adequately represent these concepts, there is a risk to your project objectives. Effective system integration design requires a technology and vendor independent design framework. The Hohpe and Woolf Pattern Language, provides this framework.

Synchronous systems are well understood. The concept of a synchronous call to a procedure or service, known as a synchronous Remote Procedure Call (RPC) is also well understood, as a similar concept is part of elementary programming. Programs calling sub procedures or services operate like a mechanical mechanism with one cog turning another cog in synchronisation with one another. If one cog stops, then everything else stops.

Messaging and queuing, however, require an understanding of asynchronous concepts. In 1972 NASA launched the two Pioneer spacecraft to the outer solar system. In the spirit of Star Trek, they thought it would be a good idea to put a plaque on the side of each spacecraft with a message to any intergalactic junk collector that might happen to find one of the Pioneers.

This is an example of an asynchronous message. We sent the message on its way, without waiting for a response. After all, we don’t know when the response will come, if ever. If a response does ever arrive, it is likely to be the waste disposal invoice from an environmentally conscious alien intelligence.

The concept of a queue is another important asynchronous concept. If you visit the UK, the old joke is; don’t stand still too long, or a queue will form behind you.

But queues are great if you are ticket seller. You can only sell tickets at a certain pace, yet the numbers of people wanting to buy a ticket in any period of time may vary. The queue enables the ticket seller to work at a steady pace, the queue growing with greater demand and shortening with lesser demand. The queue may even be empty, in which case a new customer will get served straight away.

Queues of messages enable a faster computer system to send messages to a slower computer without having to wait for a response. The receiving system does not even need to be available. The message will wait in the queue. This is known as the slow consumer problem.

What could possibly go wrong when applying asynchronous concepts to business system problems? The concepts are straightforward, the technology exists. What is often missing is the integration design describing the asynchronous solution. Here are two examples of what can go wrong without asynchronous design, the first a messaging architecture problem, the second an integration project management issue.

Messaging architecture issue

On an ERP project where I took over as project manager, an interface was sending SAP Functional Location (FLOC) messages to a data warehouse. During system integration testing of the FLOC interface, FLOCs occasionally went missing in action, and this was thought to be an intermittent fault with the data warehouse. The software developers started debugging, the testers devised ever more extensive tests with more comprehensive test data. The virtual machine hardware was beefed up, but still random FLOCs went missing. The programmers continued their busy work but the defect remained.

Conversations with the programmers triggered a solution architecture review that revealed the awful truth. The interface software was a simple synchronous solution, the goal of which was message translation between sender and receiver. The receiver was simply not fast enough to respond to a volley of FLOCs fired off through the interface pipe by the sending ERP system. The solution was simple, to add a message queue between the sender and receiver. The data warehouse was implemented on a Microsoft platform, and as Microsoft Message Queue is bundled with the operating system this was a low cost fix. Once the message queue was implemented with the data warehouse reading the FLOCs off the queue at its own pace, the system worked faultlessly.

Integration project management issue

On another system integration project, delivery of one of the enterprise interfaces was unaccountably slow and challenging to track from a project management perspective. This was an enterprise interface whose producer application endpoint, message queue and common message specification were to be used by three consumer systems also being built.

The challenge was one of structure - what were the components that needed to be built? What were the design documents required to design the interfaces? The programmers already had their interfacing technology chosen for them, and the designers had described the interface in simple data translation terms.

An enterprise integration pattern language

What was missing in both of these examples was a vendor independent way of describing the interfaces that included the concepts of asynchronous messaging, and the key interface architecture components. For those of us working in commercial software development the chances are that any problems we come across have already been solved elsewhere. The missing piece in this case was the symbolic language described in Hohpe and Woolfs’ classic book Enterprise Integration Patterns.

Messaging architecture solution

In the first example, the FLOCs being sent over the interface were going missing because the data warehouse could not receive the FLOCs as fast as the ERP system could send the FLOCs. What they had built, using the Hohpe and Woolf symbolic pattern language was this:

Modis Insights | FLOC interface as-built
FLOC interface as-built

There was no queue to absorb the FLOCs when they were produced faster than the data warehouse could consume them. By reading the symbolic pattern language, even the project manager could see that what was needed was this:

Modis Insights | FLOC interface - what was needed
FLOC interface – what was needed

Once the need for a queue was identified - the gap could be filled with a message queue built using Microsoft Message queue. The use of the Hohpe and Woolf symbolic pattern language made the documentation at all levels far more usable.

Integration project management solution

In the example of the interface used by three consuming systems, the enterprise integration design is shown in the diagram below. The ERP channel adaptor is connected to the message queue, and the messages are translated into an enterprise standard message. This is then made available to multiple subscribers using the publish-subscribe model. Each system needs the message data in a particular format so it is translated before being queued for use by the application adaptors. The benefit of using this symbolic enterprise integration language is ease of communication both for technical, management and business users, the ability to partition the design into separate documents, and to understand at a high level the tasks that need to be performed to deliver the solution. These tasks could then be estimated, scheduled, assigned and tracked.

Modis Insights | Sending a message to three consumer systems
Sending a message to three consumer systems.

The design also provides an operational advantage during implementation and production. Any of the applications can be off-line without impacting the others, providing operational flexibility and robustness.

The pattern language is widely applicable

But what of the Pioneer spacecraft mentioned earlier, can we represent the message it carries using the Hohpe and Woolf symbolic pattern language?

Modis Insights | Symbolic notation for the pioneer message
Symbolic notation for the pioneer message.

The Pioneer message was produced by a committee led by Carl Sagan who wanted to send a message to whoever it may concern, that would be representative of the whole Earth. Once the message was assembled it was translated into a form that would be usable by any finder. Controversially it also included a return address, a concept also included in the Hohpe and Woolf notation. Finally the channel, or queue, with its single message is the Pioneer spacecraft itself as it travels through space.

Summary

The application of the Hohpe and Woolf enterprise integration pattern language can assist with the development of project work break down structures, design of system architecture, and even with messages to other worlds. Closer to home, can the Hohpe and Woolf pattern language assist with the design of your enterprise integration projects?

Whitepapers

WhitepaperWhat to Look Out For When Selecting a Subscription Billing Vendor

Technology is providing a powerful enabler of new as-a-Service (aaS) consumption and delivery models. The Subscription economy has evolved and is disr...

WhitepaperAre You Making These Five Common Estimation Mistakes?

While ‘non-estimators’ expect estimation to be an exact science, there are a large number of easily-made mistakes, that can have a major impact on bot...

WhitepaperAddressing Public Sector Productivity Decline

It is accepted that productivity in the public sector (and also in the wider economy) has declined over the past 20 years despite a fairly massive inv...

Browse more whitepapers