Data integration patterns help businesses make better use of their data by addressing the challenges of data silos and disparate systems.
The accumulation of more and more data by businesses inevitably leads to siloed data, slower information gathering, and overloaded IT systems.
One way to solve this is through data integration patterns, which deal with the challenges of data silos and disparate systems. Dell Boomi recently reviewed the many types of integration processes and patterns available to businesses.
Data Integration Processes
- Transformation – The process of converting data to another format. This can include format changes like CSV to XML or unifying structured and unstructured data.
- Aggregation – Businesses pull data from a wide array of sources. Aggregation extracts and processes the data and presents it in a single view, for easier access.
- Routing – Directing data to the correct destination is an important process for many businesses. Using metadata, a data routing process can ensure that happens automatically.
- Orchestration – One step beyond simply aggregating data, data orchestration automates and streamlines multiple processes, ensuring that every process happens at the right time.
Businesses must figure out what process works best for their operation. A support ticketing system may find routing, which uses metadata to sort tickets by importance, invaluable. An automated invoice system will require more complex data orchestration, which ensures that each part of the invoicing system operates at the right time.
Data Integration Patterns
Once a process has been identified, a business needs to figure out how to make the integration happen. Dell offers four integration patterns:
- Point-to-point – The simplest of the four patterns, point-to-point connections create an easy way to speed workflows. However, point-to-point is ineffective and not scalable for businesses that use more than two connection applications, as each integration needs to be configured individually.
- Pub/Sub – Most commonly used in RSS feeds to pull data from subscriptions, publish/subscribe messaging can also be used by businesses to subscribe to any form of data. An account manager may subscribe to only important tickets, for example. Pub/sub is more scalable and reliable than point-to-point, but does require both parties to use the same protocols, and may become unwieldy if used to connect to data outside the organization.
- APIs/web services – Utilizing cloud-based applications, developers can write APIs that allow programs or web services to communicate with them. This is a modern, lightweight way to use data, which provides ease-of-access to users without application-level knowledge.
- ETL – Short for Extract, Transform, Load, this is a proven method for large data volumes. It is used for data migration and transfer projects, as it can handle vast amounts of structured or unstructured data. ETL often uses large data warehouses, like Amazon Redshift and Google BigQuery, for their efficiency.
To see the full white-paper, click this link.
