Connecting to The Grid


This section describes how to connect your Spark application to the Data Grid.

Creating the InsightEdgeConfig

To connect the Spark application to the Data Grid, use import org.insightedge.spark.implicits.all._ to enable the Spark-to-DataGrid specific API and configure SparkConf with the InsightEdgeConfig object:

import org.insightedge.spark.context.InsightEdgeConfig
import org.insightedge.spark.implicits.all._

val gsConfig = InsightEdgeConfig("insightedge-space", Some("insightedge"), Some("127.0.0.1:4174"))
val sparkConf = new SparkConf().setAppName("sample-app").setMaster("spark://127.0.0.1:7077").setInsightEdgeConfig(gsConfig)

In this example we connect to a Space insightedge-space with a lookup group insightedge and a lookup locator 127.0.0.1:4174.

Supported InsightEdgeConfig parameters are listed in the table below:

Property Name Description
spaceName Defines the logical namespace for Data Grid objects. When starting InsightEdge in a demo mode the name of space is insightedge-space
lookupGroups Used to locate the space with multicast discovery. In a demo mode the group is insightedge
lookupLocators Used to locate the space with unicast discovery. In a demo mode the locator is 127.0.0.1:4174

Creating the SparkContext

Create SparkContext as you usually do with a pure Spark application:

val sc = new SparkContext(sparkConf)

Now you are all set and can start using the SparkContext with the extended functionality.

Closing the SparkContext

When you are done interacting with the SparkContext, close it and all connections to Data Grid with:

sc.stopInsightEdgeContext()

Under the hood it will call regular Spark’s sc.stop(), so there is no need to call it manually.

See also:

For details on creating SparkContext please refer to Initializing Spark .