Starting up PySpark for using python with Spark in eclipse [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 8 years ago.

Improve this question

how do i use python for a Spark program in eclipse? I've installed PyDev plugin in eclipse and installed Python on the system but how do i use PySpark.

Answer 1

See the Spark Python programming guide

To run an interactive Python Spark interpreter, build Spark, then run:

./bin/pyspark

from within the Spark installation.

PySpark can also be used from standalone Python scripts by creating a SparkContext in your script and running the script using bin/pyspark.

One can also launch PySpark in IPython, the enhanced Python interpreter - see the programming guide for details.

Answer 2

I started a new Python project in PyDev, then went into Project -> Properties -> PyDev - PYTHONPATH -> External libraries. I added a "source path" entry for

/path/to/spark/spark-0.9.1/python

This allowed PyDev to see all Spark-related code and provide auto complete, etc.

Hope this helps.