WSO2 BAM 2.4.0 startup fails - Could not start connection pool

Question

I am running BAM 2.4.0 in a server which have CEP and APIM instance. Also I am running it with +2 port offset. But when I start BAM, console prints:

[2014-03-06 16:34:04,126] ERROR {me.prettyprint.cassandra.connection.HConnectionManager} -  Could not start connection pool for host localhost(127.0.0.1):9160
[2014-03-06 16:34:04,128]  WARN {me.prettyprint.cassandra.connection.CassandraHostRetryService} -  Downed localhost(127.0.0.1):9160 host still appears to be down: Unable to open transport to localhost(127.0.0.1):9160 , java.net.ConnectException: Connection refused
[2014-03-06 16:34:04,141] ERROR {org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent} -  All host pools marked down. Retry burden pushed out to client.
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
    at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)
    at me.prettyprint.cassandra.service.ThriftCluster.addKeyspace(ThriftCluster.java:168)
    at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.createKeyspaceIfNotExist(DataSourceUtils.java:80)
    at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSConfig(DataSourceUtils.java:92)
    at org.wso2.carbon.bam.datasource.utils.DataSourceUtils.getClusterKeyspaceFromRDBMSDataSource(DataSourceUtils.java:96)
    at org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent.initRecordStore(NotificationDispatchComponent.java:72)
    at org.wso2.carbon.bam.notification.task.internal.NotificationDispatchComponent.activate(NotificationDispatchComponent.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:105)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:192)
    at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:280)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:517)
    at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
    at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
    at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1267)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1186)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1081)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5027)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
[2014-03-06 16:34:04,222]  INFO {org.wso2.carbon.core.init.JMXServerManager} -  JMX Service URL  : service:jmx:rmi://localhost:11113/jndi/rmi://localhost:10001/jmxrmi
[2014-03-06 16:34:04,222]  INFO {org.wso2.carbon.bam.toolbox.deployer.internal.ServerStartUpInspector} -  BAM Toolbox is ready to do deployments
[2014-03-06 16:34:04,233]  INFO {org.wso2.carbon.bam.toolbox.deployer.core.BAMToolBoxDeployer} -  /root/emojot/bam/wso2bam-2.4.0/repository/deployment/server/bam-toolbox
[2014-03-06 16:34:04,273]  INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} -  Server           :  WSO2 Business Activity Monitor-2.4.0
[2014-03-06 16:34:04,275]  INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} -  WSO2 Carbon started in 37 sec
[2014-03-06 16:34:07,643]  INFO {org.wso2.carbon.dashboard.common.oauth.GSOAuthModule} -  Using random key for OAuth client-side state encryption
[2014-03-06 16:34:08,907]  INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} -  Mgt Console URL  : https://209.236.121.230:9445/carbon/
[2014-03-06 16:34:08,914]  INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} -  Gadget Server Default Context : http://209.236.121.230:9765/portal
[2014-03-06 16:34:14,107]  WARN {me.prettyprint.cassandra.connection.CassandraHostRetryService} -  Downed localhost(127.0.0.1):9160 host still appears to be down: Unable to open transport to localhost(127.0.0.1):9160 , java.net.ConnectException: Connection refused
[2014-03-06 16:34:24,109]  WARN {me.prettyprint.cassandra.connection.CassandraHostRetryService} -  Downed localhost(127.0.0.1):9160 host still appears to be down: Unable to open transport to localhost(127.0.0.1):9160 , java.net.ConnectException: Connection refused
[2014-03-06 16:34:34,110]  WARN {me.prettyprint.cassandra.connection.CassandraHostRetryService} -  Downed localhost(127.0.0.1):9160 host still appears to be down: Unable to open transport to localhost(127.0.0.1):9160 , java.net.ConnectException: Connection refused

What is wrong here?

Answer 1

If you change BAM offset you have to change the connection URLs for Cassandra stores defined in master-datasources.xml . If the port offset is 1, then those datasources should be modified as;

           <datasource>
            <name>WSO2BAM_CASSANDRA_DATASOURCE</name>
            <description>The datasource used for Cassandra data</description>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:cassandra://localhost:9161/EVENT_KS</url>
                    <username>admin</username>
                    <password>admin</password>
                </configuration>
            </definition>
        </datasource>

        <datasource>
            <name>WSO2BAM_UTIL_DATASOURCE</name>
            <description>The datasource used for BAM utilities, such as message store etc..</description>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:cassandra://localhost:9161/BAM_UTIL_KS</url>
                    <username>admin</username>
                    <password>admin</password>
                </configuration>
            </definition>
        </datasource>