SpaceDeck – Data Sources
Data Sources define the origin of data in the System of Record. A data source is a reference to a particular database or data store.
It is a logical representation of the connection to the actual source.
When we define a data pipeline definition, we will include a reference to a particular data source, and to a particular set of tables and fields in that data source.
Data Source Main Screen
The initial Data Sources screen.
Setting Up a New Data Source
To setup a new Data Source click Create New +: or New + if a data source already exists.
The initial Configuration screen will then be displayed.
Data Sources Fields
-
NAME – The name you define for the data source
The data source name is not case sensitive and has no limits for naming
-
CONNECTOR TYPE – The connector type - select either IIDR
IBM Infosphere Data Replication.
This is a solution to efficiently capture and replicate data, and changes made to the data, on relational source systems. Used to move data from DataBases to the In-Memory Data Grid., BATCHLOAD or None.For using Batch Load, the following configuration parameter needs to be used the helm command during Kubernetes
An open-source container orchestration system for automating software deployment, scaling, and management of containerized applications. installation. global.batchload.enabled=true
Data Source - IIDR
If you select Connector Type IIDR the following setup screen is displayed:
Data Sources - IIDR Setup Fields
-
Date Source Name – The name you define for the data source.
-
Description – A free-format description of the data source.
-
Connector Type – In our example, for an Oracle database, IIDR has been selected.
-
Host – The Host for the data source.
-
Port – The Port assigned for the data source.
-
Data Source Type - Options are ORACLE, MSSQL, DB2ZOS, DB2I
-
Username and Password – The credentials for accessing the data source.
Enter the configuration details and click Save Connection, The new data source will now be listed on the main Data Sources screen.
Data Source - Batch Load
Refer to our batch load page for more details.
If you select Connector Type BATCHLOAD the following setup screen will be displayed:
Data Sources - Batch Load Setup Fields
-
Date Source Name – The name you define for the data source.
-
Description – A free-format description of the data source.
-
Connector Type – In our example, for an Oracle database, BATCHLOAD has been selected.
-
Agent - Select the batch load agent being used (use a different agent for each pipeline).
For using batch load, the following configuration parameters need to be used the helm command during Kubernetes helm umbrella installation.
global.batchload.enabled=true. If an additional agent is required, the this helm command needs to be used:global.batchload-agent.enabled=true. For installing an agent and controlling its name:global.batchload-agent.agent.name=[name of agent] -
Data Source Type - Currently only ORACLE is available.
-
URL - URL of the Data Source.
-
Username and Password – The credentials for accessing the data source.
Enter the configuration details and click Save Connection, The new data source will now be listed on the main Data Sources screen.
Deleting a Data Source
There is an option to delete a data source by clicking the kebab menu (vertical three-dot menu) on the far right of the main Data Sources menu and clicking Delete Data Source:
Next Step – Using the Data Source in a Data Pipeline
After you create the data source, you can define a data pipeline using that data source.