StreamingContext couldn't bind to a port used by Java

StackOverflow https://stackoverflow.com/questions/17693097

  •  03-06-2022
  •  | 
  •  

Question

I have started Spark master and workers and can easily run a MapReduce like wordcount on HDFS.

Now I want to run a streaming on textstream and when I want to make a new StreamingContext I have this error:

scala> val ssc = new StreamingContext("spark://master:7077","test", Seconds(2))
13/07/17 11:13:45 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
org.jboss.netty.channel.ChannelException: Failed to bind to: /192.168.2.105:48594
    at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298)
....

I checked the port and it was used by Java. I killed the process and I got out of Spark-shell.

Is there any way I can change the StreamingContext's port to a random free port?

Was it helpful?

Solution

Java is the underlying process for spark (scala runs on the jvm). It is possible that you have multiple copies of spark /spark streaming running. Can you look into that?

Specifically: i get the same result if I have a spark-shell already running.

You can check for other spark processes:

ps -ef | grep spark | -v grep
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top