SpaceDeck – Data Pipelines Overview

Data Pipelines are flexible, configurable, no-code components that gather data from a mix of System of Record databases and tables. This data is loaded into the extremely high speed GigaSpaces data grid, where it can be accessed on a read/write basis.

Data Pipelines - Main Screen

The Data Pipeline A series of data processing steps, including extraction, transformation, and loading (ETL), that move data from its source to a destination system. Data pipelines are essential for integrating and managing data flows. main screen shows the status of the existing data pipelines. You can see details of a pipeline by clicking on its name in the list.

 

Data Pipelines - Main Screen - Fields

Field Names:

• Group by Status – toggle to show the pipelines grouped status

• Create New + – click this to configure a new pipeline. See SpaceDeck – Data Pipeline – Create New Pipeline for instructions on how to define a new pipeline.

• NAME – name assigned to the pipeline

  The Pipeline name has no naming limitations

• STATUS – one of the following conditions: started/stopped/not starting/restarting/not found

• TYPE – the connection type for the pipeline, for example, IIDR IBM Infosphere Data Replication. This is a solution to efficiently capture and replicate data, and changes made to the data in real-time from various data sources, including mainframes, and streams them to target systems. For example, used to move data from databases to the In-Memory Data Grid. It is used for Continuous Data Capture (CDC) to keep data synchronized across environments.

• ORIGIN – data source type from the System of Record, e.g. Oracle

• TOTAL OPERATIONS (LAST 24 HOURS) – number of operations performed by the pipeline in the last 24 hours

• TOTAL ERRORS (LAST 24 HOURS) – number of error operations performed by the pipeline in the last 24 hours

 

For more information:

SpaceDeck – Data Pipeline – Create New Pipeline

SpaceDeck – Data Pipelines – Details