07 May 2024

Data Integration (DI) IIDR Encryption

Objective

To create a secure method of transferring data across two separate segments: In this case, from the customer database to the Smart DIH Smart DIH allows enterprises to develop and deploy digital services in an agile manner, without disturbing core business applications. This is achieved by creating an event-driven, highly performing, efficient and available replica of the data from multiple systems and applications, Space Where GigaSpaces data is stored. It is the logical cache that holds data objects in memory and might also hold them in layered in tiering. Data is hosted from multiple SoRs, consolidated as a unified data model.

Overview

This page describes how to configure IIDR IBM Infosphere Data Replication. This is a solution to efficiently capture and replicate data, and changes made to the data in real-time from various data sources, including mainframes, and streams them to target systems. For example, used to move data from databases to the In-Memory Data Grid. It is used for Continuous Data Capture (CDC) to keep data synchronized across environments. components (Access Server and Agents) for TCP/IP encryption.

This can be enabled between:

Access Server and Agents
Agents and Data Stores

Data Flow

Data Flow Example:

The TLS Transport Layer Security, or TLS, is a widely adopted security protocol designed to facilitate privacy and data security for communications over the Internet. A primary use case of TLS is encrypting the communication between web applications and servers. 1.3 channel is encrypted, but the actual data from the Database is not encrypted.

In the diagram above, there are two segments. On one resides the DB (DB Segment), on the other DIH Digital Integration Hub. An application architecture that decouples digital applications from the systems of record, and aggregates operational data into a low-latency data fabric./Space (APP Segment) resides.
Data that goes from the DB segment's Oracle Agent to APP segment's Kafka Apache Kafka is a distributed event store and stream-processing platform. Apache Kafka is a distributed publish-subscribe messaging system. A message is any kind of information that is sent from a producer (application that sends the messages) to a consumer (application that receives the messages). Producers write their messages or data to Kafka topics. These topics are divided into partitions that function like logs. Each message is written to a partition and has a unique offset, or identifier. Consumers can specify a particular offset point where they can begin to read messages. Agent and Access Server is encrypted in a channel via TLS 1.3 This is the latest version of the Transport Layer Security (TLS) protocol. TLS, which is used by HTTPS and other network protocols for encryption, is the modern version of SSL..
Data is also encrypted via TLS 1.3 between the Kafka Agent and Access Server. The Access Server contains the configuration (Kafka and DB and which tables to bring via APIs).

Setup

Certificates

To enable TLS both Server and Client certificates need to be obtained.

Alternatively, you can create your own self-signed certificate (see below)

IIDR uses Java KeyStore format. Any certificate format can be converted to Jave KeyStore by using the keytool that is provided with the JDK.

Example:

PFX A pfx file is a convenient and secure way to store important cryptographic assets in a password-protected bundle: A digital certificate (e.g., a code signing certificate, SSL/TLS certificate, client authentication certificate, etc.), and. The matching private key file. to JKS A Java keystore (JKS) file is a secure file format used to hold certificate information for Java applications.:

CER A CER file contains a public key, information about the key owner, and the digital signature of the certificate authority (CA) that issued the certificate. The digital signature allows the recipient of the certificate to verify its authenticity and ensure that it has not been tampered with. to JKS:

Ultimately, you should have 2 Java KeyStores:

A server certificate, for example: privatekey.jks
A client certificate, for example: trust.jks

Configuring TLS for Access Server and CHCCLP (CLI Client)

The Access Server uses the same port for both encrypted and unencrypted communication. The clients use STARTTLS over an unencrypted connection to beging using TLS encryption. The first few bytes that are sent in each direction are not encrypted. These bytes do not include any customer data.

Older clients do not send STARTTLS and the connection is not encrypted. Newer clients cannot disable encryption and always send STARTTLS when encryption is enabled in the Access Server.

When the Access Server has TLS encryption enabled, the client must trust the Access Server's certificate.

If the client cannot validate Access Server's certificate, the connection will fail. To avoid downtime, it is best practice to keep a record of the expiration date of all certificates used so they can be replaced before they expire.

Edit the tls.properties

It is recommended to limit access to tls.properties for admin user only:

tls.properties:

Where:

trustStorePath - The path to the trust store that contains trusted root certificates. Backslashes must be escaped with an extra backslash.
trustStorePassword - The password that is required to unlock the trust store
trustStoreType - The type of trust store: JKS or PKCS12
privateKeyStorePath - The path to the private key store that contains the private key and public certificate chain. Backslashes must be escaped with an extra backslash.
privateKeyStorePassword - The password that is required to unlock the private key store
privateKeyStoreType - The type of trust store: JKS or PKCS12
enableTLS - Whether to enable TLS (true or false)
datastoresAlwaysTLS - Specify false to support datastores that negotiate encryption with STARTTLS. Specify true to support datastores that always use TLS.
encodedStorePassword (optional) - Whether the password is in base 64 encoded format (true or false).

From version 11.4.0.4-11107 passwords can be stored in base64.

To use base64 password, add the following:

To get base64 format you can perform:

Configure TLS for Agents

To enable TLS for agents, a new profile should be created which contains the appropriate configuration for your environment.

Kafka agent example:

For DB Agent it’s a similar process, however, once you save the changes, you will have to provide the password of DB user (cdc-user)

Launch the configuration tool:

MAIN MENU

---------

1. List Current Instances

2. Add an Instance

3. Edit an Instance

4. Delete an Instance

5. Manage encryption profiles

6. Exit

Enter your selection: 5

MANAGE ENCRYPTION PROFILES

--------------------------

1. Add encryption profile

2. Edit encryption profile

3. Delete encryption profile

4. Completed management of encryption profiles

Enter your selection:1

Enter an encryption profile name: TLS1.2

Engine-to-engine encryption enablement:

1. Enabled

2. Disabled

3. Required

4. Always

Select the engine-to-engine encryption enablement [Enabled]: 1

Enter the path to the private key store: /data/gs_software/iidr/tls/privatekey.jks

Enter the password that was used to encrypt the private key store:

Private key store type:

1. JKS (Java)

2. JCEKS (Java Cryptography Extension)

3. PKCS12 (Public-Key Cryptography Standards)

Select the private key store type [JKS (Java)]: 1

Enter the path to the trust store [/data/gs_software/iidr/kafka/jre64/jre/lib/security/cacerts]: /data/gs_software/iidr/tls/trust.jks

Enter the password that was used to encrypt the trust store:

Trust store type:

1. JKS (Java)

2. JCEKS (Java Cryptography Extension)

3. PKCS12 (Public-Key Cryptography Standards)

Select the trust store type [JKS (Java)]: 1

MANAGE ENCRYPTION PROFILES

--------------------------

1. Add encryption profile

2. Edit encryption profile

3. Delete encryption profile

4. Completed management of encryption profiles

Enter your selection:4

If not already done, create a new instance and select the encryption profile you generated during the previous steps.

If you already have an instance, then:

Stop the instance
Run dmconfigurets tool, choose Edit an Instance
Select the particular instance (i.e: KAFKA)
Select [5] - Manage Encryption Profiles
Select the encryption profile created in the previous step
Select [7] - Save Changes and Return to the Main Menu
For the case of a DB Agent you will be prompted for the DB password

Saving changes. Please wait...

Re-type database password:
Return to the main menu and select Exit

Creating a Private Keystore and a Trust Store with Self-Signed Certificates

TLS encryption for the communication between a source and target can be enabled by using self-signed certificates.

In this case, there is no central certificate authority so each server must trust the other server's certificates. This procedure uses commands that ship with CDC Change Data Capture. A technology that identifies and captures changes made to data in a database, enabling real-time data integration and synchronization between systems. Primarily used for data that is frequently updated, such as user transactions. Replication. You can also use third-party tools such as openssl. This procedure uses keytool, which is located under installation_directory/jre32/jre/bin.

Each server needs a private key and a self-signed certificate which can generated using the keytool -genkeypair command.

The openssl command-line utility cannot create a PKCS12 trust store that is compatible with CDC Replication. Use keytool instead of openssl. Change the -storetype parameter to PKCS12 to create a PKCS12 trust store.

Example:

Hostname (CN) = ip-10-0-1-137.eu-west-1.compute.internal

IP: 10.0.1.137

If you are using a combination of both self-signed certificates and certificates that are signed by a public certificate authority, then you need to trust the normal public certificate authorities in addition to the self-signed certificates. You can import the normal public certificate authorities into a new Truststore by using the keytool -importkeystore command.

For example:

Configuring the TLS DI Subscription Manager

Create a tls.properties file at …/di-subscription-manager/latest-di-subscription-manager

Comment out the private key lines and use only the truststore lines:

Restart the di-subscription-manager and test it using Swagger.

Testing the TLS Configuration

You can test your tls configuration by running the Access Server and CHCCLP CHCCLP is the command line interface for the CDC Replication technology of IBM® IBM Data Replication. CHCCLP allows users to script CDC Replication functionality, such as connecting to datastores, creating subscriptions, mapping tables, modifying mapping details, and controlling and monitoring replication. client.

Both the Access Server and CHCCLP use the same tls.properties configuration file.

Run the Access Server:

Run the cli client:

Connect to the Access Server:

To confirm that the Access Server is running in TLS mode through the UI console, look for the yellow lock icon located at the bottom of the window:

To verify that the Access Server is running in TLS mode through the UI console, look for the yellow lock icon located beside the Datastores on the left side.

SSL Debugging and Troubleshooting

To debug and troubleshoot SSL, add this java property for each Java process: