Data Pipeline

Definition

In enterprise architecture, a data pipeline is the technical mechanism that operationalizes the flow of information described in your data architecture and information value streams. It typically includes extraction from source systems (transactional applications, sensors, third-party feeds), transformation (cleansing, standardizing, enriching, aggregating), and loading into a target destination such as a data warehouse, data lake, reporting layer, or downstream application. Pipelines can run in batch (scheduled, periodic runs) or streaming (continuous, near-real-time) modes, and modern architectures frequently combine both. It's important to distinguish a data pipeline from a business process. A business process describes how work gets done by people and systems to deliver a capability outcome — for example, 'Process a Claim.' A data pipeline is one of the technical enablers underneath that process, moving the claim data from intake systems to fraud-detection models, reserve calculations, or regulatory reporting extracts. Business architects don't design pipelines — that's the domain of data engineers and data architects — but they must understand where pipelines exist, what capabilities and value streams they support, and where duplication or fragility in the pipeline layer creates business risk. Data pipelines are also frequently confused with ETL (Extract, Transform, Load) tools. ETL is one pattern for building a pipeline; the broader concept also encompasses ELT (load-then-transform, common in cloud data warehouses), event streaming architectures, and orchestration layers that sequence and monitor multiple pipelines as a coordinated system.

Origin & Context

The term emerged from software and data engineering practice rather than from a formal architecture framework, gaining prominence alongside big data platforms like Hadoop and later streaming systems such as Kafka. Within enterprise architecture, it is most explicitly addressed in the data architecture domain of TOGAF's Architecture Development Method and in DAMA International's Data Management Body of Knowledge (DMBOK), both of which treat data movement and integration as a governed architectural concern rather than a purely technical implementation detail.

Why It Matters

Business architects care about data pipelines because they are the physical realization of the information flows your capability maps and value streams describe on paper — when pipelines are fragmented or duplicated, the business pays through slower decisions, inconsistent reporting across business units, and costly regulatory findings around data lineage. CIOs and CDOs use capability-to-data mapping to identify redundant pipelines built by different teams solving the same problem, a common source of unnecessary infrastructure spend. Compliance and risk leaders rely on pipeline transparency to demonstrate data lineage for regulations like BCBS 239, GDPR, or Solvency II. Getting this right materially shortens the path from data capture to trusted business decision.

Common Misconceptions

Myth: A data pipeline is just an ETL tool or script.

Reality: ETL is one design pattern for building a pipeline. Modern pipelines also use ELT, event streaming, and orchestration frameworks that coordinate multiple jobs, apply data quality checks, and manage failure recovery — the pipeline is the end-to-end flow, not any single tool.

Myth: Data pipelines are purely a technology concern with no relevance to business architects.

Reality: Pipelines implement the information value streams that connect capabilities. When business architects map capabilities to underlying data flows, they routinely surface duplicate pipelines built by different business units for the same underlying data need — a rationalization opportunity that only becomes visible through the capability lens.

Myth: Once a data pipeline is built, it's a stable, set-and-forget asset.

Reality: Pipelines must evolve as capabilities, processes, and source systems change. Without governance tied to the capability model, organizations accumulate 'pipeline sprawl' — dozens of overlapping, undocumented flows that increase both operational risk and maintenance cost.

Practical Example

At a mid-size insurer, the business architecture team was mapping the Claims Management capability to its supporting data flows as part of an operating model redesign. Working with the data architecture lead, they discovered that both the Property claims unit and the Auto claims unit had independently built separate pipelines pulling from the same policy and customer master systems to feed their respective fraud-detection models. Neither team knew the other's pipeline existed because there was no shared capability model tying data flows to a common Fraud Detection capability. The business architect used the capability map to bring both teams together, proposed a single shared pipeline feeding a common data capability, and worked with the data architecture group to retire the redundant flow — reducing maintenance burden and giving compliance a single, auditable lineage path for fraud-related decisioning instead of two inconsistent ones.

Industry Applications

Financial Services

Regulatory reporting pipelines feed capital adequacy, liquidity, and stress-testing calculations, where data lineage must be traceable back to source systems to satisfy regulators.

Healthcare

Clinical and claims data pipelines feed population health analytics and quality reporting, typically mapped to the Care Management and Quality Reporting capabilities.

Retail

Near-real-time inventory and point-of-sale pipelines feed demand forecasting and dynamic pricing engines, tied directly to the Merchandising and Supply Chain capabilities.

Related Terms

• Data Architecture: the broader architecture domain that governs how pipelines and data assets are designed