Creating Middleware using Azure Service Bus
In the time when Microsoft released its Logic Apps in public preview on Azure it seems to be a little out of date to implement own middle-ware. The benefit of own system is creating architecture which could be enhanced, scaled and modified as business processes are changed in time in reliable way. Although our architecture will be close to what Logic Apps are about, but it will be more transparent to users what happens under the hood.
In this article I will introduce architecture and design of simple data flow system. I will describe some important attributes of system which needs to be taken in account. At the end I will explain how to extend the system.
Our organization has several ways how they register new customer. One is old-fashioned direct sales where new customer is inserted into the system via small desktop application used by sales people. The second is online e-commerce system, where customer fill basic or enhanced registration form. For purpose of this article it is enough to know we have more or less (mandatory) information about customer at the beginning of registration process.
We will store customer data into 2 systems. One is application database which allows user to be logged in the online e-commerce system. And the second is Customer Relationship Management (CRM) system.
Although the business requirements are straightforward from technical point of view we can solve it in several ways. And here I would propose distributed system decomposed in smallest loose coupled components. In our case we have 2 components: 1. write customer data into database and 2. write customer data into CRM.
If we will think in therms of distributed system, it is important to consider a few attributes from beginning:
- Eventual consistency – is an attribute that saying when data enters the system it will be reliable processed in certain amount of time. Practically means data will not lost and they will end up on defined targets eventually.
- Idempotency – is an attribute of the system or its component that data is not affected by proceeding 1 identical message twice. It means once message was successfully proceeded by system next process of the same message will be skipped or by-passed. This is useful when solving any failure.
- Extensibility – by this one attribute I mean all like enhancement, scalability, etc.
We can write down much more, but for our example these are enough.
Azure Service Bus
I do not want to rewrite what is well written on Azure documentation. Azure Service Bus has 3 basic options Queue, Topic and Relay. Because our system is one way communication I will remove Relay from our list.
Comparing the simple Queue to Topic, there is only one fundamental difference. Since both Queue and Topic can have many type of senders / publishers, Queue can have only one type of receiver, however scalable in many instances. On other hand Topic can have many type of subscribers, each with own “queue” (message is shadow copying for all defined subscribers). At the basic Topic can emulate Queue by defining only one subscriber.
Both Queue and Topic has implemented mechanism of reliable process of messages and it guarantees that message is locked for defined time when one instance takes it. This instance is then responsible for notifying Service Bus the message was successfully or not proceeded. If there is not such acknowledge within the time period Service Bus treat it as failure and the message is unlocked and return back into queue (any other instance can take it now). Both options have so called dead-letter queue where messages which failed in defined count are stored for manual processing.
In my example architecture Azure Service Bus and its Topic will be base component of the system.
Composition of Topics
At the bottom we need to define 2 types of composition for subsequent and parallel processing. In both it is important to consider all consequences any composition could have and implement proper remedy or recovery procedures, what is out of scope of this article.
This is simple combination of 2 Topics where message is processed by first Topic and the first subscriber is responsible resend message into second Topic after its action is completed.
Parallel processing without merging
This is the simplest architecture component, since it is only Topic with 2 or more subscribers. The important thing here is that processing is split into several independent processing threads, which will end on their own.
Parallel processing with merging
This is previous composition type enhanced by merging mechanism. There is not any option so far in Azure Service Bus how to trigger moment when message was proceeded by all subscribers, we need to design it on our own. The merging mechanism is not simple and will require some kind of state persistent and post processing in second Topic subscriber. The implementation details are out of scope of this article.
Data Flow System Design
Let me define the components which we will build our system from. At the base we will need 2 custom subscribers which will store customer data into CRM respectively into database. These subscribers will process data in parallel and thus we will create one Topic in Azure Service Bus.
At the picture you can see a little more complex system. I have introduced one more Topic at the beginning of data flow. Its only subscriber will be responsible for generating unique ID, e.g. GUID, which will identify message in the whole system.
Data flow positive example
- Any application (publisher) will send new customer data message to NcIdTopic.
- The message is taken by its subscriber, which will generate GUID. This GUID is attached to the message as its new attribute. Subscriber sends updated message to NcTopic.
- Subscriber confirms success to its Topic.
- In the NcTopic message is distributed to each subscriber queue.
- The message is taken by its subscribers independently and each processes message by its own.
- Subscribers confirm success to their Topic.
Data flow negative examples
The failure can happen anywhere in the system, so we need to verify in deep. Here the key role will play Topic, its dead-letter queue and eventually GUID. Here we say that Azure Service Bus functionality is reliable and we will not treat it.
If fails Step 1 our system cannot do much with it. It musts be implemented on application / publisher side.
Close look at the Step 2 reveals there is only sending message operation which could fail. If this happens instead of confirms success subscriber will confirm failure and origin message will appear in the queue again. The Step 2 will be re-executed and new GUID will be generated.
The exactly same could be written for Step 5. However there is one more aspect needs to be consider. The external data source CRM or database could be offline for longer time due to various reasons. And because of that several messages could exceed internal retry counter and could stored into dead-letter queue(s).
So let’s say the CRM is under maintenance and temporarily inaccessible. It means that storing new customer into CRM fails several times and this message is moved into its subscriber’s dead-letter queue. But the second subscriber successfully stored new customer into database and message was proceeded. If we will want to re-process message for CRM system we can do it manually (what is not good design) or we can somehow restart the process by moving message into the subscriber queue again.
The important attribute of Topic is there is only one entry point. There is not way how to resend message into 1 subscriber only. And now is the magic with GUID and idempotency. We can resend the message into Topic without any fear. Each subscriber has implemented idempotency, what basically means as first instruction it will compare message GUID with its internal database and process the message only and only if the message was not process yet. And due to this feature CRM subscriber will execute its action and try to store new customer again, but database subscriber will skip the action and confirm success immediately.
In this article I briefly outlined the data flow system based on Azure Service Bus. It could seem it is a little complex, however this systems is robust and immune to failure. There is a little infrastructure implementation which could be done in some base class and reuse for each subscriber. With a small effort we can implement metrics with some notification system and thus have inside data how our system behaves. This system is easily extensible by creating more subscribers and/or inserting more Topic according to changes in business flow.