I have followed this manual to run spark on my Ubuntu 16.04. I have downloaded Spark 2.3, Java 9 and Scala 2.12.5.
In my /.bashrc:
export JAVA_HOME=/usr/local/java/jdk-9.0.1 export SPARK_HOME=/usr/bin/spark export PATH=$PATH:SPARK_HOME/bin export PATH=$PATH:SPARK_HOME/sbin export SBT_HOME=/usr/bin/sbt export PATH=$PATH:$SBT_HOME/bin
In my spark-env.sh:
JAVA_HOME=/usr/local/java/jdk-9.0.1 SPARK_MASTER_IP=10.1.20.241 SPARK_WORKER_MEMORY=4g
at the end of file.
but when I run
./spark-shell I get this error:
2018-03-28 17:17:26 WARN Utils:66 - Your hostname, ali resolves to a loopback address: 127.0.1.1; using 192.168.1.52 instead (on interface enp0s3) 2018-03-28 17:17:26 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/bin/spark/jars/hadoop-auth-2.6.5.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2018-03-28 17:17:28 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Failed to initialize compiler: object java.lang.Object in compiler mirror not found. ** Note that as of 2.8 scala does not assume use of the java classpath. ** For the old behavior pass -usejavacp to scala, or if using a Settings ** object programmatically, settings.usejavacp.value = true. Failed to initialize compiler: object java.lang.Object in compiler mirror not found. ** Note that as of 2.8 scala does not assume use of the java classpath. ** For the old behavior pass -usejavacp to scala, or if using a Settings ** object programmatically, settings.usejavacp.value = true. Exception in thread "main" java.lang.NullPointerException