Spark Logging for InsightEdge

Spark uses log4j to make logging calls. For information about the log4j logging framework, refer to the log4j Overview online documentation.


Log4j configures the logging during initialization, using a logging configuration file that is read at startup. This logging configuration file is in the standard java.util.Properties format.

InsightEdge log4j Default Configuration

The default Spark logger configuration file is located under the following directory:


Configuration File Properties

Console Appender

The Console Appender writes log messages to the console.


The RollingFileAppender writes log messages to log files. The Spark log files are:

  • Written to /logs.
  • Rolled based on file size (up to 2GB)

Log File Name Format

The format of the log file name is:


The role is either spark-master or spark-worker. For example:


Spark Log Message Format

In log4j, the log message format is configured in the *.layout.ConversionPattern property. In InsightEdge, the Spark log message format is configured to match the XAP format. Therefore, the ConversionPattern is:

    log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %p [%c] - %m%n
    log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %p [%c] - %m%n

Overriding the Default Configuration can either be modified to suit the requirements of your environment, or deleted if you prefer to use the Spark default log4j configuration.