SSD - RocksDB Add-On
XAP Native Persistence delivers built-in high speed persistence using HDD or SSD. It delivers low latency write and read performance & fast data recovery. XAP Native Persistence is based on RocksDB that is an embeddable persistent key-value store for fast storage on regualr drives and flash devices. It was developed at Facebook and is now a popular open source project . XAP Native Persistence is part of the MemoryXtend add-on.
If you’re not familiar with MemoryXtend, make sure you read its documentation before proceeding.
The MemoryXtend add-on is available for free during the evaluation period, but is not included in the premium edition license. For information about purchasing this add-on please contact GigaSpaces Customer Support.
- Currently supports Linux only (Windows support will be available in the future)
- Read/Write permissions to mounted devices/partitions
- The number of mounted devices/partitions should match the number of space instances that will be deployed on the machine.
- For creating partitions you can use
fdisklike explained here.
- For creating partitions you can use
Creating a space with the RocksDB add-on can be done via
pu.xml or code. For example:
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os-core="http://www.openspaces.org/schema/core" xmlns:blob-store="http://www.openspaces.org/schema/rocksdb-blob-store" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd http://www.openspaces.org/schema/core http://www.openspaces.org/schema/10.2/core/openspaces-core.xsd http://www.openspaces.org/schema/rocksdb-blob-store http://www.openspaces.org/schema/10.2/rocksdb-blob-store/openspaces-rocksdb-blobstore.xsd"> <blob-store:rocksdb-blob-store id="myBlobStore" paths="[/mnt/db1,/mnt/db2]" mapping-dir="/tmp/mapping"/> <os-core:embedded-space id="space" name="mySpace" > <os-core:blob-store-data-policy blob-store-handler="myBlobStore" persistent="true"/> </os-core:embedded-space> <os-core:giga-space id="gigaSpace" space="space"/> </beans>
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os-core="http://www.openspaces.org/schema/core" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd http://www.openspaces.org/schema/core http://www.openspaces.org/schema/10.2/core/openspaces-core.xsd"> <bean id="blobstoreid" class="com.gigaspaces.blobstore.rocksdb.config.RocksDBBlobStoreDataPolicyFactoryBean"> <property name="paths" value="[/mnt/db1,/mnt/db2]"/> <property name="mappingDir" value="/tmp/mapping"/> </bean> <os-core:embedded-space id="space" name="mySpace"> <os-core:blob-store-data-policy blob-store-handler="blobstoreid" persistent="true"/> </os-core:embedded-space> <os-core:giga-space id="gigaSpace" space="space"/> </beans>
RocksDBBlobStoreConfigurer rocksDbConfigurer = new RocksDBBlobStoreConfigurer(); rocksDbConfigurer.setPaths("[/mnt/db1,/mnt/db2]"); rocksDbConfigurer.setMappingDir("/tmp/mapping"); BlobStoreDataCachePolicy blobStorePolicy = new BlobStoreDataCachePolicy(); blobStorePolicy.setBlobStoreHandler(rocksDbConfigurer.create()); blobStorePolicy.setPersistent(true); EmbeddedSpaceConfigurer spaceConfigurer = new EmbeddedSpaceConfigurer("mySpace"); spaceConfigurer.cachePolicy(blobStorePolicy); GigaSpace gigaSpace = new GigaSpaceConfigurer(spaceConfigurer).gigaSpace();
In addition to the general MemoryXtend configuration options, the RocksDB MemoryXtend add-on supports the following configuration options:
|paths||Comma separated available or new RocksDB folder locations.A path is a mounting point to a flash device.The list used as a search path from left to right.The first one exists will be used.||required|
|mapping-dir||Point to a directory in a file system. This directory contains file which contains a mapping between space name and a RocksDB location.||required|
|Enable in case you have a centralized storage. in this case each space is connected to a predefined RocksDB mounted location.||false||optional|
|options||RocksDB configuration options||optional|
|strategy-type||Merge or Override given options with XAP default RocksDB options.||merge||optional|
|sync||By default, each write returns after the data is pushed into the operating system. The transfer from operating system memory to the underlying persistent storage happens asynchronously. When configuring sync to true each write operation not return until the data being written has been pushed all the way to persistent storage.||false||optional|
RocksDB is created on a given directory path, RocksDB path allocation per a machine is managed via the
/tmp/blobstore/paths/path-per-space.properties file. Each time a new blobstore space is deployed an entry is added to this file listing the data grid instances provisioned on the machine.
This configuration allows each Space instance within a cluster (primary or backup) to use a dedicated storage device (SSD / HDD). With this approach, primary instances using their local storage media to preserve the data, replicating to backup instances as well use their local storage media to preserve the data. Each Space instance once provisioned performs its data recovery (if enabled) from its local storage. This configuration will work well for development and small / medium data grids.
Since each Space instance is using a local storage device, its survivability is low as there is no hardware level high-availability is running, keeping the data safe in another device in case the entire machine running the Space instance completely fails. The central storage option below is leveraging a single storage appliance or multiple appliances which offers much better data survivability and much larger data storage capacity.
The RocksDB Add-on supports storage area network (SAN) which means the disk drive devices are installed in a remote machine but behave as if they’re attached the the local machine. Most storage networks use the iSCSI or Fibre Channel protocol for communication between servers and disk drive devices.
In central storage mode each space is attached to a pre-defined device as explained on these examples:
Single storage array
The following example deployes a 2 partitions space with a single backup (2,1) in the following manner:
/mnt/db1will be mounted to the 1st primary.
/mnt/db2will be mounted to the 2nd primary.
/mnt/db3will be mounted to the 1st backup.
/mnt/db4will be mounted to the 2nd backup.
<blob-store:rocksdb-blob-store id="myBlobStore" paths="[/mnt/db1,/mnt/db2,/mnt/db3,/mnt/db4]"/>
Two storage arrays
It is also possible to define two storage arrays instead of one, which will guarantee that the primary and backup instance of a partition will not be provisioned to the same storage array. The following example deployes a 2 partitions space with a single backup (2,1) in the following manner:
/mnt1/db1will be mounted to the 1st primary.
/mnt1/db2will be mounted to the 2nd primary.
/mnt2/db1will be mounted to the 1st backup.
/mnt2/db2will be mounted to the 2nd backup.
<blob-store:rocksdb-blob-store id="myBlobStore" paths="[/mnt1/db1,/mnt1/db2],[/mnt2/db1,/mnt2/db2]"/>