CDC minimizes the resources required for processes because it only deals with data changes. Such capabilities allow streaming projects to scale more easily.
A perfect storm of industry changes is making change data capture (CDC) all the more important in applications that use events-based and streaming data. The reason: Modern applications need a way to manage data changes and events that trigger an action downstream. Quite often, each change is used by different applications, for different purposes, at different times.
Change data capture is a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access the change events in the order in which they occurred.
A typical CDC pipeline would start with an event producer application. An example might be a shopping app where a user creates or updates an account. The app makes those changes in a database. CDC would then make the changes available to a stream engine. CDC minimizes the resources required for processes because it only deals with data changes. Such capabilities allow streaming projects to scale more easily.
The growing need for change data capture
Major industry changes are driving the need for CDC. To start, many more operations and business processes must transition from batch and reactive modes to real-time. Financial institutions can no longer afford to analyze transactions once a month or week looking for fraud. They must react to real-time data about user behavior and transactions in progress and stop the fraud from happening in the moment. Similarly, a modern retailer would use dynamic input such as an in-store beacon alert that a loyalty program customer is in the cosmetic section to instantaneously deliver in-person or text promotional offers. A CDC system could serve as the intermediary between the events and instances that invoke a change and the analytics engine that processes them to derive insights and take actions.
A second transformation underway is the need for rapid application development and constant updates to meet user expectations and demands. Batch-based, monolithic applications of old are not up to the task. They cannot be easily modified. New features take forever to roll out. And any small change requires updating the entire application. Organizations do not have the luxury to take workhorse production applications offline for any period to revise them.
The way to address this issue is to break apart the monolithic applications of old and build new applications based on a cloud-native architecture. Such applications are composed of containers and microservices to form loosely-coupled, distributed applications. Using such an architecture, businesses get a highly dynamic system composed of independent processes that work together to provide business value.
As such, many organizations are moving to cloud-native application architectures based on microservices and containers. The services and data provided by the core systems are encapsulated and presented to other elements of an application as a microservice via an API. In that way, new front-end applications can easily be developed, and new analytics algorithms and models can be used to extract insight into the data. The core application and services it provides remain untouched. Still, CDC can play a role by sitting between the core application and the downstream processes. Again, a CDC would capture the data changes and make them available to a stream engine.
In such a use case, a CDC helps organizations modernize applications and efficiently scale them. For example, industries such as banking, insurance, government, retail, healthcare, and travel have relied on monolithic applications for decades. Organizations in these industries face competition from startups that can build applications from scratch using modern development techniques and architectures. Additionally, they are under intense pressure from their users, who demand access to data and applications. They are used to an always-on, instant-response world, where everything is accessible from the platform of the user’s choice, be it desktop, mobile, phone, or in-person.
Using the microservices approach mentioned above, organizations can make fast changes to existing front-end applications and create new applications. With change data capture software consuming fewer resources as it only addresses data changes, these applications can retain their performance and scale without a massive infrastructure build-up.
CDCs fit well with one additional industry trend. The composable enterprise, where code that enables a single process or step is reused in different applications, is on the rise. Building on the flexibility of cloud-native development, here specific processes or business functions are standardized and can be reused in numerous applications. For instance, a bank or online retailer might have multiple standard processes that are triggered when a customer creates or changes a user account. A security group might use a process that automatically triggers a text to confirm the customer indeed made a password. Another group might use the knowledge of a home address change to send a promotional email for that new location. Or yet another group might take a change in income to analyze and adjust the customer’s credit limit. Regardless of the reusable process, a CDC would sit between the user account database and the action-taking application. As in other uses, here, the CDC would be used to determine and track the data that has changed so that action can be taken using the changed data.
