Apache Kafka has been growing in adoption over the past decade, serving as a powerful tool for leveraging real-time data. However, it has not been well known outside the development department. The challenge is figuring out how to “democratize” this process to deliver enterprise-scale real-time data streaming capabilities that the business can appreciate.
The reimagining of Kafka as a business enabler was the subject of a recent webinar co-hosted by TIBCO Software and RTInsights. “Kafka is like an island, the developers love it, it’s very lush,” said Mark Palmer, senior vice president of analytics for TIBCO. “They’re doing a lot with it, building applications. But from a business user’s perspective, they ask questions like this: ‘How do I connect Kafka to my enterprise data? How can I see and analyze what’s inside Kafka?’ A lot of data is real time, they want to see what’s going on.”
Opening up Kafka to the enterprise may help achieve the trust and acceptance that data-driven enterprises require. “Everyone loves data analytics and the possibilities real-time data offers to our businesses,” said Joe McKendrick, senior executive analyst with RTInsights and co-presenter. “But real-time data streaming may not be quite ready to go to the next level. Business leaders haven’t quite learned to trust or depend on the data they receive from data-driven analytic models to make their decisions. As we see in the industry statistics, everyone aspires to be data-driven, but few business leaders feel they’re anywhere near that goal.” At the same time, millions of developers and data scientists have embraced Kafka, the real-time streaming technology, and are prepared to move forward with it. The challenge is bringing together this disconnect to deliver greater advantages to the business.
While Kafka has been readily deployed at leading tech-intensive platforms such as LinkedIn and Netflix, “The power of Kafka is now available to mainstream enterprises as well,” McKendrick adds. “The ultimate goal is a continuous intelligence environment. That’s where the business value is realized. Decision makers don’t have to worry about where information is coming from, and they’re not hampered by disjointed islands of data analytics. Real-time insights are delivered automatically in a pervasive, almost ambient way.”
Palmer points to three technologies that are helping to build the bridge between the business and Kafka developers, which represent a different way of thinking about Kafka other than it being in the exclusive domain of the application developer:
- Streaming data virtualization. With data scattered across enterprises, data virtualization “puts streaming data right next to your historical data,” said Palmer. “For example you might want to see the historical data—bring the data fabric together with real-time streaming and historical. … Virtualization you get a virtual view and develop very quickly, as opposed to moving data around, getting it into the right format and centralizing it. Kafka can be a participant in that scenario.
- Streaming business intelligence with Kafka. Trusting and knowing the data that is streaming through the enterprise is key to success. “As systems automate, as more intelligence gets built into applications, real-time decisions get more and more important,” says Palmer. “This is part of scaling out the wisdom and providing it to everybody in the organization. The point is how we can democratize Kafka data by connecting easily as it was any other data source.”
- Streaming data science with Kafka. Kafka can play a key role in proliferating artificial intelligence across the organization. “Many organizations have data scientists, and they make great observations, but those observations are not operationalized to the business,” says Palmer. “You have applications that are accessed by hundreds of thousands of users across the enterprise. This is what we’re after—scaling out the wisdom and making it available to everybody in the enterprise. You’re not going to train everybody and send that back to become Ph.Ds.”
