Here is an overview, what is hidden behind spark interpreter in Apache Zeppelin.<\/p>\n\n\n\n\n\n\n\n
Source code for Apache Zeppelin is available here: source code.<\/a><\/a><\/p>\n\n\n\n First of all we should look inside of Interpreter launcher: source of launchers<\/a>. For spark there is a special launcher SparkInterpreterLauncher.java<\/a>, which extends standard launcher: StandardInterpreterLauncher.java<\/a>. Here we can see configuration for spark command line, like –conf <key>=<value> and usage of proxy users, plus, check of running mode, like yarn client or cluster.<\/p>\n\n\n\n Depending on selected isolation options in interpreter, spark is started as RemoteInterpreterRunningProcess or RemoteInterpreterManagedProcess. This stays in launch function in StandardInterpreterLauncher. Important is that spark starts as a new process. For example, if you use spark option –proxy-user, then zeppelin will create separate spark processes per user. Communication between current process and interpreter process is done using RPC in case of RemoteInterpreterManagedProcess.<\/p>\n\n\n\n Behind the scene interpreter.sh<\/a> is started by RemoteInterpreterManagedProcess. And inside of interpreter.sh script, spark-submit from active spark distribution, with RemoteInterpreterServer<\/a> as a spark class is called. To communicate with RemoteInterpreterServer from zeppelin UI Apache Thrift RPC is used.<\/p>\n\n\n\n This makes a trick, spark started per spark submit in separate JVM together with Thrift RPC server, to enable communication with spark process.<\/p>\n\n\n\n Spark interpreter (spark-interpreter*.jar) is in class path. RemoteInterpreterServer can initialize then spark inside of its process.<\/p>\n\n\n\n Then we need to look, what happened inside of spark interpreter, which is a classical spark application with SparkContext for spark1 and SparkSession for spark2.<\/p>\n\n\n\n