1

I can get Spark on it through the Software Center, but how do I get pyspark?

6

pyspark is available via pypi.

So all you need to install pyspark is pip and execute the following command.

pip install pyspark or sudo pip install pyspark

| improve this answer | |
5

pyspark is a python binding to the spark program written in Scala.

As long as you have Java 6+ and Python 2.6+ you can download pre-built binaries for spark from the download page. Make sure that the java and python programs are on your PATH or that the JAVA_HOME environment variable is set. Follow these steps to get started;

  1. Unzip and move the unzipped directory to a working directory:

    tar -xzf spark-1.4.0-bin-hadoop2.6.tgz

    mv spark-1.4.0-bin-hadoop2.6 /srv/spark-1.4.8

  2. Symlink the version of Spark to a spark directory:

    ln -s /srv/spark-1.4.8 /srv/spark

  3. Edit ~/.bash_profile using your favorite text editor and add Spark to your PATH and set the SPARK_HOME environment variable:

    export SPARK_HOME=/srv/spark

    export PATH=$SPARK_HOME/bin:$PATH

Now you should be able to execute pyspark by running the command pyspark in the terminal.

Some references:

https://spark.apache.org/docs/0.9.0/python-programming-guide.html

https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python

| improve this answer | |
  • You also need git installed. Found out the hard way when installing on a clean virtual image... – bangnab Dec 8 '15 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.