How to add Spark to Maven project in Eclipse?

StackOverflow https://stackoverflow.com/questions/15211074

  •  18-03-2022
  •  | 
  •  

Question

I would like to start Spark project in Eclipse using Maven. I've installed m2eclipse and I have a working HelloWorld Java application in my Maven project.

I would like to use Spark framework and I'm following directions from the official site. I've added Spark repository to my pom.xml:

<repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
</repository>

And then the dependency:

<dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
</dependency>

But I'm getting an error in Eclipse:

Missing artifact spark:spark:jar:0.9.9.4-SNAPSHOT

How can I resolve this issue? I don't want to download Spark's jar file and place in the local repository.

This is my pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.myproject</groupId>
  <artifactId>Spark1</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Spark1</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
  </repository>

  <dependencies>
<!--     (...) -->

    <dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
    </dependency>

  </dependencies>

</project>
Était-ce utile?

La solution 2

The repository block needs to be wrapped in a repositories block:

<repositories>
    <repository>
        <id>Spark repository</id>
        <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
    </repository>
</repositories>

Autres conseils

Currently no repository is required to add for Spark library loading

You just need to add

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.6.0</version>
</dependency>

And that's it.

Useful tutorials to play with is here

Reason for failure is 0.9.9.4-SNAPSHOT is not available.Below is the list of the snapshots available. Use one among them based on your requirement.

0.9.8-SNAPSHOT/ Sat May 21 21:54:23 UTC 2011
0.9.9-SNAPSHOT/ Mon May 23 10:57:38 UTC 2011
0.9.9.1-SNAPSHOT/ Thu May 26 09:47:03 UTC 2011
0.9.9.3-SNAPSHOT/ Thu Sep 01 07:53:59 UTC 2011

Thanks, Sankara Reddy

I had run into same issue because initially I started with different repository url for spark and then to use earlier version I changed the repository url. Some how it didn't seem to come into effect until I changed the repository id. Try changing the repository id.
Could be bug in maven because running maven from console also couldn't resolve the dependency without updating the id.

please add the repository , tag inside the repositories tag like below 

<repositories>
        <repository>
            <id>Spark repository</id>
            <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
        </repository>
    </repositories>

The last versions (2.1 and later) of Spark only need the dependency defined inside the pom.xml file 

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.1</version>
</dependency>

the repository definition is not longer required

use this latest repository. http://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10/1.6.0

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.6.0</version>
</dependency>

use this, and also make sure you change spark library to version 2.11.x in eclipse project build path

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.spark-scala</groupId>
    <artifactId>spark-scala</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>${project.artifactId}</name>
    <description>Spark in Scala</description>
    <inceptionYear>2010</inceptionYear>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.tools.version>2.10</scala.tools.version>
        <!-- Put the Scala version of the cluster -->
        <scala.version>2.10.4</scala.version>
    </properties>

    <!-- repository to add org.apache.spark -->
    <repositories>
        <repository>
            <id>cloudera-repo-releases</id>
            <url>https://repository.cloudera.com/artifactory/repo/</url>
        </repository>
    </repositories>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <!-- see http://davidb.github.com/scala-maven-plugin -->
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.13</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <!-- "package" command plugin -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>
</project>
Licencié sous: CC-BY-SA avec attribution
Non affilié à StackOverflow
scroll top