62% of survey respondents say they expect to deploy one to five more applications based on the Apace Flink framework in the coming year.
A global survey of 217 IT professionals that have adopted open source Apache Flink stream processing framework finds machine learning (64%) anomaly detection/system monitoring (27%), business intelligence/reporting (25%), recommendation/decisioning engines (22%) and security/fraud detection (19%) are the top five types of applications IT organizations are planning on deploying in 2018. A full 62 percent of the respondents say they expect to deploy one to five more applications based on the Apace Flink framework.
Conducted by data Artisans, a provider of a distribution of Apache Flink, the survey also finds 70 percent of the respondents say their team or department is growing and hiring in 2018, and that 59 percent expect their budget to increase.
See also: Data streaming at the edge — IBM and Apache Quarks
A quarter of the survey respondents say they are already processing at least 1 billion events per day. Data Artisans CEO Kostas Tzoumas says the Apache Flink framework is gaining momentum because it can be used to process both continuous and static datasets. A total of 46 percent of respondents use Flink only for streaming data, while 47 percent use it for a mixture of streaming and static finite datasets, the survey finds. A total of 46 percent of respondents also said the Flink framework is being used to make high volumes of data available in real-time. Citing data compiled by the research firm MarketsandMarkets, Tzoumas notes the streaming analytics market is expected to worth over $13 billion by 2021.
“There are a lot more applications now where real-time is required,” says Tzoumas.
Tzoumas says the Flink framework is being employed as an alternative to Apache Spark and Storm software. The fundamental difference is that Flink makes it possible to rollback streaming data pipelines when necessary within the context of a stateful application. That eliminates the need to rely on a separate database when processing real-time applications.
Other benefits of Flink cited by survey respondents include making it easier to build distributed applications (46%), improved scalability (37%), improved performance (35%), simplified application design (35%), reduced application complexity (29%) and bringing applications online faster (23%).
Flink Integration a Priority in the Future
Top priorities of focus for the Flink community in the future, says Tzoumas, will be tighter integration between Flink and Java, Scala, and SQL.
Overall, the survey finds satisfaction with Flink is 92 percent; with the top four areas of satisfaction being event time handling (76%); the DataStream API (74%); throughput and latency (72%) and windowing and watermarks (71%).
While the Apache Flink framework is apparently gaining traction it’s also become apparent that competition between various open source and commercial frameworks for processing data in real time has become fierce. IT organizations are increasingly realizing that most emerging digital business and Internet of Things (IoT) applications require data to be processed in real time. That doesn’t necessarily eliminate the need for batch-oriented applications altogether. But there’s a clear shortage of talent and expertise when it comes to processing data in real-time across the enterprise. The challenge and opportunity facing IT organizations now are figuring out how to close that gap as quickly as possible.
