The world is increasingly becoming event-driven and needs to operate in real-time. That’s where a smart event broker comes into its own, providing the foundation for a connected enterprise across all stakeholders.
The adoption of event-driven architecture (EDA) by enterprises is on the increase as the need for real-time communication of information grows among companies across all industries. Real-time communication happens via an event broker, a unique tool for enterprises looking to power time-sensitive applications, processes, and insights at scale. Yet, achieving the full potential of EDA will rest on the chosen event broker – and ensuring this “middleman” is the most suitable for that enterprise. Here, we delve into the needed event broker criteria: Fine-grained routing and filtering, event mesh-ability, and rich application integration.
First, let’s take a step back to put EDA’s use into perspective. More and more enterprises are realizing the benefits of adopting EDA platforms, as eight in 10 of IT leaders claim that their company plans to adopt EDA to two to three new use cases within the next two years, according to an IDC Infobrief. It coincides with digital maturity, as just under half of respondents described their EDA journey as either maturing (“centralized”) or “advanced”.
Across numerous industries, the rise of EDA is been prevalent. From retail, manufacturing, transportation, and logistics to aviation and financial services, more and more enterprises worldwide are recognizing the need to adopt an event-driven strategy to enable application connectivity and modern use cases.
Enter the EDA middleman
Nestled within EDA transformation is the middleman: the event broker. Middleware software designed to route events and data between applications, systems, and devices. But as the use of EDA has grown, so has the number for event brokers, and today’s enterprises have a large variety of brokers to choose from. Selecting the event broker best suited to your business could make or break for real-time event-driven strategies, so it’s more important than ever that application teams weigh up their evaluation criteria thoroughly.
Give your event-driven strategy the best chance at success: pick the right event broker
An event broker is message-oriented middleware that enables the reliable transmission of events between different components of a system, acting as a mediator between publishers and subscribers. It is the cornerstone of event-driven architecture, and all event-driven applications use some form of an event broker to send and receive information.
As a first and critical step, the choice for IT leaders is now notwhether to embrace EDA, but rather which event broker tochoose to underpin their EDA, or which ones to use for which use cases, as often an enterprise might find that they need more than one type of broker, because some are better suited to particular use cases.
Analysts are jumping into the debate, notably David Mooter of Forrester, who recently outlined the choice between a “log stream broker” vs. a “smart broker.” A log stream broker can support high data throughput and tolerate some degree of complexity, as well as message replay, when combined with real-time analytics and event sourcing. A smart broker, on the other hand, can support complex message routing, granular controls on message filtering, global order guarantees, transactional commits, and many other capabilities.
The power of a case in point when selecting the broker
The fundamentals of event brokers, Mooter explains, allow organizations to build their applications as a collection of composable services that are de-coupled in nature. This provides the benefits of agility, scalability and resilience but, depending on the use cases, he argues, choosing between a log stream broker or a smart broker can make all the difference.
Log stream brokers are best epitomized by Apache Kafka. Over the last few years Apache Kafka has taken the world of data streaming by storm, because it’s very good at its intended purpose of aggregating massive amounts of “log” data and streaming it to analytics engines and big data repositories.
One event broker doesn’t equal the other
Unfortunately, the popularity and prevalence of Kafka has led many developers to use Kafka for use cases for which it is not ideally suited – namely operational, “run-the-business” scenarios, which often involve a mix of applications, systems, and devices that need to tap into specific event streams to run effectively.
Log based brokers use rigid, flat topic structures to describe the data they transmit, which puts the onus on the applications to do the work to filter through all the data that’s being streamed. This can be like drinking from a firehose – as applications need to consume, filter and throw away events they don’t need which leads to increased cost and complexity, not to mention security concerns.
The more connected the business becomes, the smarter the broker must be
On the flip side, “smart” brokers do a lot of the thinking, filtering, routing – especially as enterprises work to become more integrated, connected and real-time across diverse Information Technology (IT) and Operational Technology (OT).
These brokers feature rich and flexible topic hierarchies which enable applications to easily publish and subscribe to the very specific subsets of data they are interested in. From an event streaming perspective, smart brokers support a wide range of message exchange patterns beyond publish/subscribe, including request/reply, streaming and replay, as well as different qualities of service, such as best effort and guaranteed delivery.
This makes smart brokers ideal for operational use cases, acting as the “digital nervous system” of a distributed enterprise. Smart brokers go beyond simply analytics – imagine a global bank streaming over 150 billion events a day between low latency trading platforms and market data centers in multiple locations across the globe. Prices could be published in New York, and consumed in London in no time, allowing orders to be placed quickly based on the latest information.
Choosing the right smart broker to run your business
The choice becomes clear – if an organization is looking to address operational “run-the-business” applications and use cases across a distributed enterprise, they need a smart broker, providing them three critical attributes:
1) Introducing the event mesh
Organizations who require real-time event streaming for operational use cases need to look at how their broker can support an “event mesh”. An event mesh is an architecture layer consisting of a network of event brokers interconnected to allow events from one application to be dynamically routed and received by any other application, no matter where these applications are deployed – no-cloud, private cloud, or public cloud. An event mesh makes all data that touches the mesh available on demand in a secure, reliable manner – exactly where and when needed.
It provides the ability to integrate legacy applications, data stores, modern microservices, SaaS, IoT and mobile devices dynamically and all in real-time. If Information Technology and analytics is legacy integration; and Operational Technology is the sensing of devices; then an event mesh is the glue that ties old technology with the new.
Consider a manufacturing organization using IoT edge devices for asset tracking, product quality monitoring, and predictive maintenance. With an event mesh, sensor events in manufacturing processes can be leveraged with real-time streaming analytics to improve quality and detect machine maintenance issues sooner. Operations are always on, and so are customer and employee expectations. An event mesh supporting real-time data delivery ultimately means no waiting. Spikes in demand are deftly handled so systems don’t crash.
2) Refined filtering creates a targeted approach
When dealing with complex business operations, events need to be intelligently routed and filtered to be made available to the right people at the right time, not just served up en masse. A smart event broker allows consumers to subscribe to only receive the subset of events they need in the original sequence. Take the operational example of a retailer using microservices to stream order-related events – but different functions are being dealt with in different locations. Those application users who only want to process new orders do not require information such as shipped orders or returns; they just require information based on new orders.
Put the refined approach into action
It’s also the job of a smart broker to publish information in a way that allows fine-grained filtering of events based on topic taxonomy. Consider an example of an aviation authority that is required to manage incoming and outgoing flights from a particular airport – sounds straightforward, but there will be a multitude of events related to each and every flight. A smart broker can handle the volume of events related to all flights but allow subscribers to drill down by airline, arrival/departure, on time/delayed, and inbound and outbound gate – meaning they can receive detailed information based on the factors relevant to their job function or area of responsibility.
3) Distributed organizations need a modern integration process: let the smart broker connect it all together
Catering to the complex business ecosystems they support, a smart broker easily connects a variety of applications and devices without worrying about protocol translation from one to another. A distributed organization, for example, has many custom applications in many different languages that they need to share information.
Operational use cases will usually involve this complex mix of legacy and modern applications as well as taking on further data streams from IoT devices – all communicating using different languages and protocols. These streams may also need to be accessed by partners or third parties using standards-based protocols such as AMQP, MQTT, or webhooks – hence the need for a smart broker to support integration across this application landscape.
Integration of SaaS, legacy and packaged applications and middleware are also required across an organization. This requires the use of specialized event connectors to event-enable these applications – so the event broker you choose needs to have many of these “off the shelf” connectors as well.
Heineken: the case for rich application integration
Heineken, the multinational brewing company, has business units that are spread across 190 countries and is the perfect use case of how EDA deployments have increased in sophistication. Heineken uses an event-driven integration approach to support event streaming across thousands of business-critical applications touching payments, logistics, inventory management, and more. Only a smart broker enables this consistently fast, reliable, and robust integration process across all key business functions.
The smarter the event broker, the smarter business applications can become
The world is increasingly becoming event-driven – and soon we will only see more and more business operations become data-driven with the need to operate in real-time.
It’s here that the smart broker will come into its own, providing the foundations for a connected enterprise across stakeholders, developers, users, consumers, partners, and end-customers – always and in real-time.
