SparkLE/docker

Building docker images

spark: spark over kubernetes docker image hibench-base: HiBench base image modified to be able to submit workloads on a kubernetes cluster

/HiBench/bin/functions/workload_functions.sh:

@@ 220c220 @@
-      SUBMIT_CMD="${SPARK_HOME}/bin/spark-submit ${LIB_JARS} --properties-file ${SPARK_PROP_CONF} --class ${CLS} --master ${SPARK_MASTER} ${YARN_OPTS} ${SPARKBENCH_JAR} $@"
+      SUBMIT_CMD="${SPARK_HOME}/bin/spark-submit ${LIB_JARS} --properties-file ${SPARK_PROP_CONF} --class ${CLS} --master ${SPARK_MASTER} --conf spark.kubernetes.container.image=pl4tinum/hibench-kube:spark-3.0-hadoop-2.7-scala-2.12 --conf spark.kubernetes.authenticate.driver.serviceAccountName=default --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.namespace=iccs-hibench --conf spark.driver.host=hibench-master-nfs-svc --conf spark.driver.port=5000 --conf spark.kubernetes.executor.podTemplateFile=/executor.yaml ${YARN_OPTS} ${SPARKBENCH_JAR} $@"

hibench-spark: Spark over kubernetes with hibench jars