When selecting URP products, organizations must start with a clear understanding of the project goals, business requirements, and more. Here are some points to consider in the evaluation process.
Unified Real-time Platforms (URPs) are an important, growing category of software for supporting real-time business applications that apply analytics to streaming data to facilitate faster and better decisions.
This research document is the second part of a two-part collection. The first document, What Exactly is a Unified Real-time Platform, describes thirteen essential capabilities of URPs and explains how URPs differ from event stream processing (ESP) platforms, streaming DBMSs, and stream-enabled analytical DBMSs.
The purpose of this second document is to help organizations find the URP that is most appropriate for their needs if they determine that a URP is relevant to their application. We begin by outlining the tradeoffs between using a URP versus building the application on a comparable “do it yourself” (DIY) assembly of separate software products or (cloud) services. We then explain seven criteria that apply to your URP product selection:
- Availability of off-the-shelf URP applications (“solutions”)
- Programming model and ease of application development
- Performance – throughput and latency
- Interaction patterns – event-driven and request/reply
- Process orchestration for situation responses
- End-user interface capabilities
- Commercial acquisition considerations
Do You Need a URP?
URPs are motivated by escalating business demands for smarter operational applications that can leverage up-to-the-second data to make faster and better decisions. Streaming data from sensors, vehicles and other machines, cameras, mobile devices, transaction processing applications, news feeds, and other external web sources is proliferating. Real-time use cases create new revenue opportunities, help counter threats, and mitigate risks. For more discussion of where to use URPs and some examples of URP applications, see Four Kinds of Software to Process Streaming Data in Real Time.
If an organization has identified a business requirement for real-time streaming analytics in an operational application, the most common alternative to a URP is to assemble a DIY collection of multiple piecemeal technologies that approximates the services of a URP. A DIY infrastructure may combine an ESP platform with a high-performance streaming DBMS and application servers or frameworks such as Spring Boot, Quarkus, or Micronaut that enable self-hosting Java backend services that run in containers without application servers.
Compared to URPs, DIY generally has drawbacks, such as
- Expertise, technical debt, time to value: Few organizations have the expertise to prepare DIY infrastructure for highly demanding workloads. DIY systems are complicated to design, develop, tune, and maintain because there are so many moving parts. There tend to be multiple independently configured and managed clusters, so DIY projects generally have longer time-to-solution and incur more technical debt than URPs.
- Risk, latency, cost: Architects struggle to deliver very high volume or predictable very low latency because there are many places where network, memory space, and other technical boundaries are crossed. Each link across boundaries adds code path and latency. By contrast, URPs minimize processing overhead and latency by tightly integrating the components into a single offering. In high-volume cases, URP infrastructure often requires less hardware and, thus, lower cost than DIY architectures.
- Complexity, security: DIY projects that combine software components from multiple vendors tend to be even more difficult than DIY projects that acquire all of the software from one vendor. Complexity is the enemy of security. Hyperscalers, including Amazon AWS, Google, Microsoft Azure, and a few other large vendors, provide all of the pieces for ”one-stop shopping” and have partly integrated their relevant software products through the use of common tooling and metadata management facilities. They have also generally tested their multiple software products together, which helps. For example, Google’s Dataflow (based on Flume with Apache Beam SDK) integrates with other services like BigQuery, Vertex AI, Bigtable, Cloud Functions, and Looker to serve real-time application building. Single-vendor suites partly ameliorate DIY development challenges, although they still generally fall short of well-integrated, off-the-shelf URPs.
Nevertheless, more custom-built, streaming operational applications are currently in production on DIY infrastructures than on URPs. However, we expect that URP adoption will continue to expand, particularly for situations where off-the-shelf URP application solutions are available (see below) and for extreme high-volume/low-latency problems.
It is important to note that URP products are quite diverse in their internal architectures and in their intended purposes. Although all URPs have the essential characteristics described in What Exactly is a Unified Real-time Platform, URPs are best understood as a general architectural pattern, not as a single class of product. URP products compete in many disparate markets, serving disparate vertical and horizontal applications. Large organizations will eventually have multiple different URPs for the same reason that they already have multiple kinds of DBMSs for different applications.
URP Selection Criteria
As with any product category, there is no “best” URP; there are only URPs that are better or worse for any particular project. Figure 1 summarizes the evaluation criteria used in this document.
Before evaluating URP products, analysts and architects must develop a clear understanding of the project goals, business requirements, and constraints because every situation is different. The criteria below are roughly in order of importance for typical projects, but you need to make adjustments to reflect your unique situation.
1. Solutions or Infrastructure
URP products can be categorized as either general-purpose infrastructure software or domain-specific software solutions (which are built on embedded general-purpose, domain-independent URP infrastructure).
A solution is a set of features and functions, an application template, or a full (tailorable) commercial off-the-shelf (COTS) application or SaaS offering that is focused on a particular vertical or horizontal domain. URP solutions are available for various aspects of customer relationship management (CRM); supply chain management; (IoT) asset management; transportation operations (trucks, planes, airlines, maritime shipping); capital markets trading; AIOps; and other application areas. For example, URP solutions related to CRM include Evam Marketing, Joulica Customer Experience Analytics, Scuba Analytics’ Collaborative Decision Intelligence Platform, Snowplow Behavioral Data Platform (BDP), Unscrambl Qbo, and ZineOne Customer Engagement Hub.
Infrastructure products offer general-purpose URP capabilities suitable for use in many industries and applications. They are technically a subset of solutions because the user company or a third-party partner must build the application from scratch, whereas solutions should need less customization. URP infrastructure is offered by Gigaspaces, Gridgain, Hazelcast, KX, NStream, Pathway, Radicalbit, Scaleout, Timeplus, Vantiq, Volt Active Data, and other vendors.
Some vendors primarily sell solutions, although their underlying URP platform could technically be used for other applications. Other vendors sell platform infrastructure into multiple vertical or horizontal markets but may also sell URP solutions or partial solutions into one or two particular domains.
Wherever practical, you should buy a solution if you can find one that suits your situation. The age-old advice applies: buy before build. URP solutions generally require less work, have faster time-to-production, and have lower technical risk. Solution vendors have staff expertise in their respective vertical or horizontal domains and often have adapters to external applications, databases, or industry-specific data formats and protocols.
However, there may be no URP solution that fits your requirements. Also, in some cases, it actually takes more work to tailor a solution template or COTS application to your needs than it would take to implement the application from scratch on a URP infrastructure product (and, in turn, a URP infrastructure product will generally involve less time, expertise, and risk than a DIY infrastructure if your application really needs high scalability, low latency, or complicated analytics). Note that some applications have such extreme scale or latency requirements that an otherwise suitable COTS URP solution does not work, so URP infrastructure is the most practical approach.
2. Programming model and development tools
Most organizations give ease-of-application-development and analytics very heavy weight in their selection process because developer productivity is crucial to project cost and time-to-production. It is hard to find good developers for these kinds of applications, so many URPs offer tools at multiple levels of abstraction. For example, Cogility, Evam, KX, and Vantiq, among others, have invested heavily in multiple levels of tools. Some URPs support graphical tools or domain-specific languages (DSL) so that rules or similar parts of the application can be developed by business analysts or other less technical builders. Many URPs support Python or integration with Tensorflow or other libraries for implementing analytic logic that may include inferencing AI (predictive ML inference) or retrieval-augmented generation (RAG) generative AI-based logic. Most high-value use cases leverage real-time inferencing using ML models to continuously make autonomous decisions using real-time features. A few URP applications support continuous ML model training as the system runs. We see the same paradigm already being applied to fine-tuning language models. Many URPs provide SQL interfaces for implementing transformations and query retrievals.
Certain URPs predominantly support programming languages such as Java, whereas others favor rapidly growing languages like Python. The reason for the focus on richer support of a particular language may vary – it is sometimes driven by the strategic intention to serve the developers within a given market segment rather than solely being determined by the language used to build the URP itself.
Virtually any URP can be used to implement a digital twin design pattern, which is relevant for numerous URP applications. However, some URP vendors provide special tooling to make this better and easier. Examples of explicit digital twin support are offered by Aveva (PI), NStream, XMPro, and Scaleout, among others.
Most URP projects involve integration with previously running applications and databases, such as SaaS, packaged applications, or legacy applications (sometimes even mainframe
systems). Most URP solution vendors have multiple off-the-shelf adapters for integration, as do some platform infrastructure vendors, including Hazelcast, Gigaspaces, Gridgain, Radicalbit, Vantiq, and Vitria, among others.
Some products have introduced the ability to create vector embeddings on real-time streaming data to support retrieval augmented generation (RAG) pipelines and AI assistants or chatbots.
3. Performance
All URPs (solutions and infrastructure) provide good scalability and low latency because that is intrinsic to the URP value proposition. If a URP product can meet your foreseeable requirements for message rates (e.g., 50,000 events per second), the number of entities being tracked (e.g., 1,000 trucks, 1,000 cell towers, or 100,000 customers), and latency (e.g., 99% of responses in less than 400 milliseconds) then it doesn’t matter much if there are faster or more-scalable URPs on the market.
However, some applications, particularly in telcos (e.g., network monitoring), financial services (e.g., fraud detection), and certain asset management (“IoT”) scenarios, are dealing with extreme volume (millions or tens of millions of events per second) and/or require a single digit or low double-digit millisecond latency. These require URP products that are purpose-built with in-memory data management, business logic processing co-located in the same address space with data, one-thread-per-core architecture, or other high-performance design concepts. A related benefit may be that they require fewer cores and less memory than conventional systems. Hazelcast, Gigaspaces, Gridgain, NStream, and Scaleout are among the URPs known to offer extreme performance.
4. Interaction
Some URP applications support externally-triggered, request-driven business transactions (e.g., OLTP), while others focus on event-driven monitoring of big, complex operations to provide situation awareness and sense-and-respond interventions. It would be TL;DR to explain this difference in detail here, but it suffices to note that some request-driven use cases require transaction semantics such as exactly-once processing, concurrency control, isolation, persistence, strong or eventual consistency, or checkpoints for rapid restarts after failure. Some URP vendors, notably Hazelcast, Gigaspaces, Gridgain, and Volt Active Data, among others, support multiple integrity features because transaction processing plays a major role in most of their applications.
5. Process Orchestration
Projects that focus on monitoring operations detect current or predicted threats and opportunities and then use URP features to emit alerts, update end-user dashboards, or trigger automated responses through messages or RESTful calls. A few URP vendors, including XMPro, Vantiq, and Vitria, among others, go even further by also providing internal process managers that orchestrate longer-running multistep workflows, i.e., response sequences consisting of automated activities and/or human steps. URP projects that service transactional requests may also require multistep sequences. If a URP does not supply process orchestration natively but multiple steps are required, developers can use an external business process management (BPM) or choreography tool to manage the actions.
6. End User Interfaces
URPs are primarily server-side, backend platforms that receive event streams (typically through Kafka, Kafka-like, or other messaging subsystems) and service requests (typically through Restful API calls) from other applications. However, many URP vendors, including Cogility, Deephaven Data Labs, Evam, Joulica, NStream, Pathway, Scuba Analytics, Snowplow, Unscrambl, Vitria, XMPro, and ZineOne, among others, also supply front-end, end-user-facing analytical applications or dashboards, or tools to build such front ends. Alternatively, developers can build a front-end application using their preferred programming tools.
7. Commercial Acquisition Considerations
Of course, acquiring URP software or subscribing to a URP Platform-as-a-Service (PasS) involves the same considerations as any other software project. Your URP selection process must consider the viability of the vendor and its support practices, price, and terms and conditions. You may prefer self-managed software (on-premises or in a private cloud) or a vendor-supported PaaS, including the Bring-Your-Own-Cloud (BYOC) model, where the vendor provisions the solution in the customer’s infrastructure. You should look at each vendor’s capabilities to ensure that your choice is available because many pure-play URP vendors are relatively small and don’t offer all the options.
