SpaceDeck – Configuration File for Data Pipeline

In the screen to define a new data pipeline, the configuration parameters can be entered manually, or can be defined in a configuration file. The configuration can contain all or part of the configuration information. It is in JSON format.

A sample configuration file, and explanation of the parameters, are shown below.

Sample Data Pipeline Configuration File

  "name": "pipeline",
  "spaceName": "ditest1-ts",
  "sorName": "DB2",
  "cdcProvider": "IIDR",
    "CDC": {
      "kafkaTopic": "noi_cdc"
  "batchWrite": 1000,
  "checkpointInterval": 1000

Configuration File Parameters

  • name – name you are assigning to the pipeline

  • spaceName – Space name that will receive the pipeline data

  • sorName – name of the database in the System of Record

  • cdcProvider – connector type, e.g. IIDR

  • CDC:kafkaTopic – name of the Kafka topic for CDC changes

  • INITIAL_LOAD:kafkaTopic – name of the Kafka topic for initial load changes

  • batchWrite – size of the single batch write from a DI layer to the space. The value specified here is the number of commands.

  • checkpointInterval – interval, in milliseconds, that the data integration layer performs a commit to Kafka and flush to Space