Scaling the GigaSpaces Environment

Scaling with the GigaSpaces Helm Chart

GigaSpaces supports rolling updates of the Helm chart so that you can scale your system up or down as necessary. Configure the following to ensure that you keep your data and system configuration intact:

  • Readiness probe - To update the chart while maintaining data consistency, you must enable the readiness probe in either the current or the new Helm chart. This ensures that the update process, which first terminates the backup pods, won't begin to update the primary pods before the backup pods have fully recovered and reloaded the data, and can therefore take leadership when the primary pods are terminated.
  • Reuse- values flag - You must mark the reuse-values flag to ensure that all your custom chart configurations (including the license and partition definitions) are not overwritten by the update process. If you don't mark this flag, your system will revert to the default configurations.

For example, run the following command to change the amount of memory for each service (pu) pod from 400 MB to 600 MB.

helm upgrade demo xap --set pu.resources.limits.memory=600Mi,pu.readinessProbe.enabled=true --reuse-values

Vertically Scaling a Kubernetes Pod

GigaSpaces supports on-demand updates of a StatefulSet during runtime. This capability can be used to scale a pod’s resources up or down. For stateful services (such as a Space), each StatefulSet represents a specific partition, so you target the required partition for updating. In a high availability environment, this will affect both the primary and the backup pods.

The pods of the StatefulSet are updated in reverse ordinal order.

Readiness Probe

For stateful services, you must enable the readiness probe in order to maintain data consistency (as mentioned above). The upgrade process restarts the first pod, leaving the second one alive. The Space inside the second pod is a primary Space (it may have originally been a primary Space, or a backup that became the primary). This pod isn't restarted until the first pod is up again and the Space it contains (the backup) has fully recovered and reloaded the data. When the second pod is upgraded, the Space in the first pod becomes the primary. The upgrade process is complete after both pods are up and fully recovered.

To enable the readiness probe, set the readinessProbe.enabled option to true:

helm install insightedge-pu --name mySpace --set manager.name=hello,partitions=2,readinessProbe.enabled=true

Scaling the Pod

You can update a pod using either the REST Manager API, or using the GigaSpaces CLI.

Using the REST Manager API

The following code sample shows how to scale the memory capacity of the mySpace-insightedge-pu-2 Statefulset using the REST Manager API.

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: text/plain' -d '[ \ 
   { \ 
     "op": "replace", \ 
     "path": "/spec/template/spec/containers/0/resources/limits/memory", \ 
     "value": "600Mi" \ 
   } \ 
 ] \ 
 ' 'http://192.168.99.171:30890/v2/kubernetes/statefulsets/mySpace-insightedge-pu-2'

Using the GigaSpaces CLI

The following command shows how to scale resources using the GigaSpaces CLI command. You must specify the name of the service (Processing Unit) and the partition number.

./gs.sh --server=192.168.99.171:30890  pu scale mySpace 2 --memory=600Mi --cpu=500m

Monitoring the Update Process

When the pod is scaled using the REST Manager API or the CLI, you receive the request ID as a response. You can use this request ID to check the status of the request and verify that the update process was accomplished successfully and the pods are up, updated and fully recovered.

Checking the Status via the REST Manager API

You can send a request to check the status of the update process using code like the example here:

curl -X GET --header 'Accept: application/json' 'http://192.168.99.171:30890/v2/requests/1'

Checking the Status via the GigaSpaces CLI

You can use a CLI command to check the status of the update process as follows:

./gs.sh --server=192.168.99.171:30890  request status 1 

Manually Disabling the Patch Role

By default, the GigaSpaces manager Helm chart creates a role-binding with a patch role that is enabled. This is required to perform the update request against the Kubernetes cluster. To disable patch role and therefore disable the update operation, set the value of the PatchSupport.enable option to false:

helm install insightedge-manager --name hello --set PatchSupport.enable=false

Troubleshooting

The update process may fail if the update request has an error. For example, if you try to scale to a memory size that is larger than the available memory of the system. If this happens, the pod may enter broken state; the StatefulSet stops the update process but still waits for the pod to become Ready and Running, even though it’s in broken state.

To find the unhealthy pod, use the following kubectl command and view the status of all the pods:

kubectl get pods

To fix the problem, you need to send a new update request with a valid configuration (for example, scale to a valid memory size), and then delete the unhealthy pod.

To delete the unhealthy pod, use the following Kubernetes command.

kubectl delete pod mySpace-insightedge-pu-1-0

Kubernetes will recreate the pod with the latest and correct configuration.