Decentralized message broker federation architecture with multiple DHT rings for high survivability

Abstract

A message broker is an essential component of messaging services that connect information providers and consumers, message service clients, to enable the integrated message network. To support large scale message clients with the contracted service level, multiple brokers have to collaborate and form a federated broker cluster in many cases. For example, Kafka is one of the most popular messaging systems that allows multiple brokers co-working together. ZooKeeper, the distributed cluster coordinator, manages cluster nodes and stores metadata of messages for Kafka. Even though ZooKeeper does the coordination job well in a simple way, its half-centralized coordinating methods make the overall system less capable in survivability. In some domains such as military warfare and embedded sensor networks, we may lose the primary coordinator or lose more than a half of the coordinator machines. In these cases, we cannot support the minimum survivability to maintain the message network. To address this limited survivability problem, we propose a decentralized message broker federation architecture with distributed hash table. In our proposed architecture design, the decentralized coordinator supports the DHT exchanges between brokers to manage metadata of distributed message partitions. We built a prototype of a message broker federation based on our proposed decentralized metadata coordinator design to show the feasibility in terms of practical application.