SpaceDeck – Data Pipeline – Monitoring Pipelines

Data Pipeline Status Screen

The Data Pipeline Status screen shows the status of the existing data pipelines. You can see details of a pipeline by clicking on its name in the list.

The New + button allows you to create a new pipeline. See SpaceDeck – Data Pipeline – Create New Pipeline for instructions to define a new pipeline.

Data Pipeline Status Fields

  • Error – pipeline is not available because of one of the following conditions: not found/connection error/not started/started but not running/loading stopped/suspended

  • Loading – pipeline is in the process of being configured

  • Idle – pipeline is configured and the data integration is available

  • Group by Status – toggle to display the list of pipelines grouped by the pipeline status

  • Name – name assigned to the pipeline

  • Status – one of the following conditions: started/stopped/not starting/restarting/not found

  • Type – the connection type for the pipeline, for example, IIDR

  • Origin – data source type from the System of Record, e.g. DB2

  • Total Operations – number of operations performed by the pipeline

  • Total Errors – number of error operations performed by the pipeline

Data Pipeline Details

Click on the pipeline name to view details of the pipeline. Three tabs are available: Monitoring, Tables pipeline, and Configuration:

View Pipeline Details – Monitoring Tab

The Monitoring tab shows a variety of real-time pipeline metrics.

Monitoring Tab Fields

  • Last 24 hours – select the timeframe for the following statistics

  • Ops/Sec over time – the average number of operations per second

  • Errors over time – the average number of errors per second

View Pipeline Details – Tables pipeline Tab

The Tables pipeline tab shows details of the object types in the pipeline.

Fields in the Tables pipeline Tab

  • Object Type – name of the Object Type in the Space. This is initially the same as the source table name in the System of Record, but can be changed when the pipeline is defined.

  • Origin Name – The name of the source table in the System of Record. This will be the same as the Object Type field, unless you changed the Object Type name when you defined the pipeline.

  • Daily Operations – the number of daily CRUD operations performed on the table

  • Daily Errors – the number of daily CRUD operations with errors performed on the table

  • Last Updated – the date that the pipeline configuration was created or updated

Viewing Object Type Details

Click on the object type name to see details of the object type.

Properties Details

The Property tab appears as follows:

  • Name – name of the object property (column)

  • Type – data type of the object property

Indexes Details

The Indexes tab appears as follows:

  • Name – name of the index

  • Type – index type

  • Properties – the object properties that comprise the index

  • Unique – indicates if unique or non-unique values are allowed for the index

View Pipeline Details – Configuration

The Configuration tab shows pipeline configuration information. The data is read-only when the pipeline is running.

The Pipeline Configuration screen appears as follows:

Basic Pipeline Information

You can fill in some or all of the pipeline configuration items (shown below) from a JSON-format configuration file by clicking on the Load Configuration button.
The configuration file may contains some or all of the required details. After the configuration details are loaded from the configuration file, they can be edited if desired, before saving .

  • Pipeline Name – name assigned to the pipeline

  • Space Name – the name of the GigaSpaces Space object that will receive the pipeline data

  • Connector Type – the data connector type, for example, IIDR.

  • Connector Setting:

    • Data Source Connection – the data source from the System of Record. This is a URL and points to a database such as DB2.

    • CDC Kafka Topic – The name of the Kafka topic for CDC changes

    • SYNC Kafka Topic – The name of the Kafka topic for initial load changes

  • Advanced Setting:

    • Batch Write – size of the single batch write from a DI layer to the space. The value specified here is the number of commands.

    • Checkpoint Interval – interval, in milliseconds, that the data integration layer performs a commit to Kafka and flush to Space

Press Create Pipeline to create the new data pipeline.

Select Tables for the Pipeline

You may then press Select Tables to choose which tables to include in the pipeline. Press Add to add the selected tables to the pipeline.

Edit the Pipeline

Edit Pipeline – Parameters Tab

This tab will be supported in a future release.

 

Edit Pipeline – Fields Tab

In the Pipeline fields section of this screen, the Field (column) names are initially the names of the fields from the database table that are included in the data pipeline. These can be edited to provide different property names (column names) in the GigaSpaces object type (table).
Other fields in this screen will be editable in a future release.