52% of organizations have at least 6 systems running Kafka.
Some 86% of organization are increasing their use of Apache Kafka, according to a“Streaming Data” report from Confluent.
Kafka acts as a messaging broker in real-time and big data systems, often determining where data goes for analysis.
Some 20% of respondents to the Confluent report, which surveyed professionals in a wide range of industries in 47 countries, said that their use of Kafka is increasing sharply.
Respondents to the survey held positions of expertise in areas ranging from developers to tech management and engineers. One in four work for organizations that have more than $1 billion in sales.
Some 52% of organizations said they have at least six systems running Kafka while 21% have over 20. That is in sharp contrast to last year’s report where only 41% of organizations said they had at least six systems running Kafka and only 10% had more than 20.
Kafka helps realize a unified streaming platform. It allows a centralized team to manage massive amounts of data per day and integrates with legacy technology including data warehouses, Hadoop, ETL and messaging middleware. Over 15% of respondents to the Confluent survey are processing more than a billion messages a day.
Kafka’s use in the cloud is also growing. The report found that 34% of organizations use it in virtual private clouds, 52% in public clouds, and 57% on premises. Roughly 32% said they have at least six Kafka applications in the cloud.
Kafka is also creating new business opportunities. Because it makes data available in real-time, companies can create new products or transform existing ones. Roughly 54% of organizations said Kafka enables their business to make faster and more accurate decisions; 47% said that Kafka has allowed them to reduce operating costs; and 40% said it helped them provide a better customer experience.
Organizations are using Kafka in a variety of ways:
- The most common use is data pipelines, with 81% say they are using it for that purpose.
- 66% use Kafka for stream processing
- 60% use it for data integration.
- 50% say they are using it for microservices.
- 75% said they have applications that process data from websites, analytics and sensors connected to their Kafka systems.
- Asynchronous apps make up 57% of Kafka use while data warehouse apps are a close second with 51%.
- Other uses for Kafka include application monitoring at 41%, system monitoring at 30% and recommendation engines at 30%.
The Kafka API is also enjoying increased popularity, with 37% said they are using it, compared with just 12% last year. Approximately 59% of respondents have databases connected to their Kafka clusters but only 36% are using the Kafka Connect API with Hadoop/HDFS,
Kafka Streams API is the new kid on the block but has already been enthusiastically adopted, with 89% of organizations saying they are familiar with it.
40% are using Kafka Streams for ETL, 32% for core business apps, and 25% for asynchronous applications. Of organizations that have microservices, 28% are using the Kafka Streams API to manage them. Developers are the most common users at 85% but architects (48%) and application teams (43%) also use them.
Despite Kafka’s popularity and the high salaries people with Kafka skills command, there’s a shortage of skilled Kafka engineers. Approximately 75% of organizations said they are having trouble finding skilled Kafka engineers. Despite that, 70% of organizations reported that they are completely satisfied with their Kafka systems.
