storm-deploy submit topology java.lang.NoClassDefFoundError

Question

after running topology in local cluster I created a remote storm cluster (storm-deploy Nathan). i Have removed Storm jars from build path in eclipse before creating a runnable jar with "package dependencies". my topology uses storm-kafka-0.9.0-wip16a-scala292.jar, which i either left in build path and removed from build path before creating runnable jar(just to try solve this issue..). when i use the following command:

./storm jar /home/ubuntu/Virtual/stormTopologia4.jar org.vicomtech.main.StormTopologia

it always replies:

Exception in thread "main" java.lang.NoClassDefFoundError: OpaqueTridentKafkaSpout
  at java.lang.Class.getDeclaredMethods0(Native Method)
  at java.lang.Class.privateGetDeclaredMethods(Class.java:2451)
  at java.lang.Class.getMethod0(Class.java:2694)
  at java.lang.Class.getMethod(Class.java:1622)
  at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
  at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: OpaqueTridentKafkaSpout
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

Since this topology worked fine on AWS on a single instance as a runnable jar, i can´t figure what i´m missing... here is code inside my main method:

        Config conf = new Config();

        OpaqueTridentKafkaSpout tridentSpout = crearSpout(
                kafkadir, "test");


        OpaqueTridentKafkaSpout logUpvSpout = crearSpout(kafkadir,
                "logsUpv");

        OpaqueTridentKafkaSpout logSnortSpout = crearSpout(
                kafkadir, "logsSnort");

        try {
            StormSubmitter.submitTopology(
                    "hackaton",
                    conf,
                    buildTopology( tridentSpout, logUpvSpout,
                            logSnortSpout));
        } catch (AlreadyAliveException | InvalidTopologyException e) {

            e.printStackTrace();
        }



    } catch (IOException e) {
        e.printStackTrace();
    } catch (TwitterException e) {
        e.printStackTrace();
    }

}

private static OpaqueTridentKafkaSpout crearSpout(
        String testKafkaBrokerHost, String topic) {
    KafkaConfig.ZkHosts hosts = new ZkHosts(testKafkaBrokerHost, "/brokers");

    TridentKafkaConfig config = new TridentKafkaConfig(hosts, topic);
    config.forceStartOffsetTime(-2);

    config.scheme = new SchemeAsMultiScheme(new StringScheme());
    return new OpaqueTridentKafkaSpout(config);
}


public static StormTopology buildTopology(OpaqueTridentKafkaSpout tridentSpout,
        OpaqueTridentKafkaSpout logUpvSpout,
        OpaqueTridentKafkaSpout logSnortSpout
        ) throws IOException,
        TwitterException {

    TridentTopology topology = new TridentTopology();



    topology.newStream("tweets2", tridentSpout)
            .each(new Fields("str"), new OnlyEnglishSpanish())
            .each(new Fields("str"), new WholeTweetToMongo())
            .each(new Fields("str"), new TextLangExtracter(),
                    new Fields("text", "lang")).parallelismHint(6)
            .project(new Fields("text", "lang"))
            .partitionBy(new Fields("lang"))
            .each(new Fields("text", "lang"), new Analisis(),
                    new Fields("result")).parallelismHint(6)
            .each(new Fields("result"), new ResultToMongo());


    return topology.build();

}

Is there any way I can make OpaqueTridentKafkaSpout available? thank you in advance

hope it´s not a stupid cuestion, i'm very newbie to this field

Answer 1

We can keep the storm jar in the build path when you generate the jar-with-dependencies, we just have to tell maven not to bundle it, like this (see the "provided" scope, meaning that jar is provided by the runtime environment, hence not necessary to bundle):

<dependency>
    <groupId>storm</groupId>
    <artifactId>storm</artifactId>
    <version>0.9.0-rc2</version>
    <scope>provided</scope>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Kafka spout however must be included in the jar-with-dependencies, so its maven declaration looks like this:

<dependency>
        <groupId>storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.0-wip16a-scala292</version>
</dependency>

To validate stuff, you can always unzip the resulting jar and check manually that the necessary classes are present/absent as they should before deploying to storm.