SpaceDeck – Configuration File for Data Pipeline

In the screen to define a new data pipeline, the configuration parameters can be entered manually, or can be defined in a configuration file. The configuration can contain all or part of the configuration information. It is in JSON format.

A sample configuration file, and explanation of the parameters, are shown below.

Sample Data Pipeline Configuration File

{
  "name": "pipeline",
  "spaceName": "ditest1-ts",
  "sorName": "DB2",
  "cdcProvider": "IIDR",
  "jobConfigs":{
    "CDC": {
      "kafkaTopic": "noi_cdc"
    },
    "INITIAL_LOAD": {
      "kafkaTopic":"noi_il"
    }
  },
  "batchWrite": 1000,
  "checkpointInterval": 1000
}

Configuration File Parameters

• name – name you are assigning to the pipeline

• spaceName – Space name that will receive the pipeline data

• sorName – name of the database in the System of Record

• cdcProvider – connector type, e.g. IIDR

• CDC:kafkaTopic – name of the Kafka topic for CDC changes

• INITIAL_LOAD:kafkaTopic – name of the Kafka topic for initial load changes

• batchWrite – size of the single batch write from a DI layer to the space. The value specified here is the number of commands.

• checkpointInterval – interval, in milliseconds, that the data integration layer performs a commit to Kafka and flush to Space