Huge performance issue using camel routes in karaf

Question

I have a tricky issue with karaf, and having tried all day to fix it, I need your insights. Here is the problem:

I have camel routes (pure java DSL) that get data from 2 sources, process them, and then send the results to a redis - when using as standalone application (with a Main class and a command line "java -jar myjar.jar"), data are processed and saved in less than 20minutes - when using them as a bundle (part of another feature actually) , on the same machine, it takes about 10 hours .

EDIT: I forgot to add: I use camel 2.1.0 and karaf 2.3.2

Now, we are in the process of refactoring our SI to karaf features, so sadly, it's not really possible to just keep the standalone app.

I tried playing with karaf java memory option, using a cluster (I failed :d ) playing with SEDA and threadpool, replacing all direct route by a seda, without success. A dev:create-dump shows a lot of

thread #38 - Split" Id=166 BLOCKED on java.lang.Class@56d1396f owned by "Camel (camelRedisProvisioning)

Could it be an issue with split and parallelProcessing in karaf ? Standalone app shows indeed a LOT more CPU activity.

Here are my camel route

//start with a quartz and a cron tab
from("quartz://provisioning/topOffersStart?cron=" + cronValue.replace(' ',  '+')).multicast()
        .parallelProcessing().to("direct:prodDAO", "direct:thesaurus");

//get from two sources and process
from("direct:prodDAO").bean(ProductsDAO.class)
.setHeader("_type", constant(TopExport.PRODUCT_TOP))
.setHeader("topOffer", constant("topOffer"))
.to("direct:topOffers");

from("direct:thesaurus")
.to(thesaurusUri).unmarshal(csv).bean(ThesaurusConverter.class, "convert")
.setHeader("_type", constant(TopExport.CATEGORY_TOP))
.setHeader("topOffer", constant("topOffer"))
.to("direct:topOffers");


//processing  
from("direct:topOffers").choice()
        .when(isCategory)
            .to("direct:topOffersThesaurus")
        .otherwise()
            .when(isProduct)
                .to("direct:topOffersProducts")
            .otherwise()
                .log(LoggingLevel.ERROR, "${header[_type]} is not valid !")
            .endChoice()
        .endChoice()
    .end();

from("direct:topOffersThesaurus")
//here is where I think the problem comes
        .split(body()).parallelProcessing().streaming()
        .bean(someprocessing)
        .to("direct:toRedis");

from("direct:topOffersProducts")
//here is where I think the problem comes
        .split(body()).parallelProcessing().streaming()
        .bean(someprocessing)
        .to("direct:toRedis");

//save into redis
from("direct:toRedis")
        .setHeader("CamelRedis.Key", simple("provisioning:${header[_topID]}"))
        .setHeader("CamelRedis.Command", constant("SETEX"))
        .setHeader("CamelRedis.Timeout", constant("90000"))//25h
        .setHeader("CamelRedis.Value", simple("${body}"))
.to("spring-redis://?redisTemplate=#provisioningRedisTemplateStringSerializer");

NB: the body sent to direct:topOffers[products|thesaurus] is a list of pojo (the same class)

Thanks to anyone that can help

EDIT: I think I narrowed it down to a deadlock on jaxb. Indeed, in my routes, I make lots of call to a java client calling a web service. When using karaf, thread are block there : java.lang.Thread.State: BLOCKED (on object monitor) at com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:78)

further down the stack trace, we see the unmarshalling method used to transform the xml in object, those 2 line we suspect to me

final JAXBContext context = JAXBContext.newInstance(clazz.getPackage().getName());
final Unmarshaller um = context.createUnmarshaller();

I remove the final, no improvements. Maybe something to do with the jaxb used by karaf ? I do not install any jaxb impl with the bundle

Answer 1

Nailed it ! As seen above, it was indeed linked to a deadlock on the jaxb context in my webservice client. what I did : - refactoring of the old code for that client by removing the final keyword on Marshaller/Unmarshaller object (I think the deadlock came from there, even if it was the exact same code when runing on standalone) - instanciate the context based on the package, and only once. I must admit Classloader issues with OSGI had me banging my head on my desk for a few hours, but thanks to Why can't JAXB find my jaxb.index when running inside Apache Felix? , I manage to fix that

Granted, my threads are now sleeping instead of blocked, but now I process my data in less than 30 min, so that's good enough for me