Application Modernization and Change Data Capture

CDC can play a critical role in application modernization efforts ensuring the tight messaging and passing of events that exists in monolithic applications are carried over to today’s loosely coupled cloud-native environments.

Many monolithic apps are event-driven. As companies strive to modernize their apps and migrate them to cloud-native architectures, they need a way to make event-driven aspects work across distributed, loosely coupled components. Change Data Capture can help.

Why modernize?

Monolithic apps were fine the days when a company might need to make one or two updates a year. But that is ancient history.

Businesses today need to react faster these days. Business units, clients, and customers are accustomed to and demanding quick fixes to problems, more features in their offerings, and better services. Applications require frequent updates to add new features, support different data sources, and incorporate new technology ranging from new front-end devices to more sophisticated analytics routines to derive insights for business decisions. Additionally, companies must quickly scale to meet new demands.

Monolithic apps typically have problems in all of these areas. As a result, many businesses are migrating their legacy applications to modern development and deployment platforms. (And they are building any new ones on those same platforms.) The expected benefits from a modernization include reduced time to market, increased team autonomy so different groups can work the codebase independently, and dynamic scaling. 

Specifically, there are moving to cloud-native architectures based on microservices and containers (most often Kubernetes) to construct and run loosely-coupled, distributed applications.

Containers offer a way for processes and applications to be bundled and run. They are portable and easy to scale. They also allow large applications to be broken into smaller components and presented to other applications as microservices. And microservices provide the loosely coupled application architecture, which enables deployment in highly distributed patterns. Using such an architecture, businesses get a highly dynamic system composed of independent processes that work together to provide business value.

Issues with event-driven monolithic apps

Many monolithic applications in many industries like banking, retailer, hospitality, logistic, and others are event-driven. Traditional applications with tight coupling between components rely on request-driven messaging processes, where calls for data are made directly from one service to another. In monolithic application environments, these processes are tightly integrated and run on the same system.

However, the distributed nature of microservices-based applications needs a different approach to communication. It needs an event-driven architecture.

Event-driven solutions enable real-time and responsive applications, such as loan approval or payment processing in the financial services industry. Event producers publish events, ordered by time of creation, forming an event stream. The stream can be distributed across the enterprise, and consumers may subscribe to the various event streams and use all or only the events they’re interested in.

Where Change Data Capture comes in

As monoliths are broken up into microservices, the event producer and consumer may be in different microservices. The ability for one microservice to publish events to others is a necessity to carry forward the core programming constructs into a distributed computing architecture.

That’s where Change Data Capture (CDC) comes in. CDC is a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access the change events.

A typical CDC pipeline would start with an event producer application. An example might be a shopping app where a user creates or updates an account. The app makes those changes in a database. CDC would then make the changes available to a stream engine. CDC minimizes the resources required for processes because it only deals with data changes. Such capabilities allow streaming projects to scale more easily.

Debezium is the most widely used open-source change data capture (CDC) project with multiple connectors and features. When an application writes to the database, changes are recorded in log files, and then the database tables are updated.  The good news is Debezium has connectors for different databases, so it does the hard work of understanding the format of all of these log files. Debezium can read the log files and produce a generic abstract event into a messaging system such as Apache Kafka, which contains the data changes.

In summary, CDC can play a critical role in many application modernization efforts. It ensures the tight messaging and passing of events that existed in monolithic applications are carried over to today’s loosely coupled cloud-native applications. This, in turn, allows businesses to modernize their applications to gain the benefits of cloud while retaining the real-time processes in their legacy applications.