How to access pyspark in anaconda installed in another user?

Question

I have two user user1 and user2 with latter one having root privilege.I have installed spark and hadoop in user2.

Now i have installed Anaconda2 on user1 and when iam trying to run my job on jupyter notebook, it's giving me error no module named pyspark.

So, Can i access pyspark from user1?

Answer 1

Make sure user2 has SPARK_HOME environment variable configured if not, set it. If you have set SPARK_HOME for all users should, accessing SPARK_HOME should not be issue for user2. Once SPARK_HOME is variable is configured, add following script in the first cell of the Jupyter notebook and after this cell is executed you should be able to use sparkContext(sc).

# Configure the necessary Spark environment
import os
import sys

spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "/python")

# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.8.2.1-src.zip'))

# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))