InitialContext.lookup of remote EJB fails after remote server restart

Question

We have a setup in which EJB A is running on server A, and another EJB B is running on server B. EJB A connects to EJB B via IIOP. This setup normally works, but if server B is restarted, EJB A will fail until server A is restarted too.

The issue is that if server B is restarted, all calls to InitialContext.lookup by EJB A fails with a "java.io.IOException: End-of-stream" exception until server A is restarted. I wasn't able to find information on whether our app server (GlassFish) does any kind of caching for InitialContext.lookup. Are there any other reasons why lookups would fail until a server restart? If InitialContext.lookup does cache connections, how would I get around that?

Our servers run Sun Application Server 9.1. The lookup is actually done via org.springframework.jndi.JndiTemplate, but the stack trace says that the JndiTemplate is calling InitialContext.lookup().

Thanks for any insight.

P.S. I should clarify that I'm trying to figure out whether it's possible to avoid having to restart server A every time server B is restarted.

Definition of JndiTemplate (with some text blacked out with 'x's and '#'s)

<bean id="xxxxxxxxxx"  class="org.springframework.jndi.JndiTemplate">
  <property name="environment">
   <props>
    <prop key="java.naming.factory.url.pkgs">com.sun.enterprise.naming</prop>
    <prop key="java.naming.factory.initial">com.sun.enterprise.naming.SerialInitContextFactory</prop>
    <prop key="java.naming.provider.url">iiop://xxxxxxxxxx:####</prop>
    <prop key="java.naming.factory.state">com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl</prop>
    <prop key="org.omg.CORBA.ORBInitialHost">xxxxxxxxxx</prop>
    <prop key="org.omg.CORBA.ORBInitialPort">####</prop>
   </props>
  </property>
 </bean> 

And the stack trace (with one part replaced with '[application methods]'):

NAM0004: Exception during name lookup : {0}
java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is: 
 org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 208 completed: Maybe
 at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
 at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
 at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
 at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
 at com.sun.enterprise.naming._SerialContextProvider_DynamicStub.lookup(com/sun/enterprise/naming/_SerialContextProvider_DynamicStub.java)
 at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:398)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:155)
 at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:88)
 at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:153)
 at [application methods]
 at org.springframework.web.servlet.mvc.SimpleFormController.processFormSubmission(SimpleFormController.java:267)
 at org.springframework.web.servlet.mvc.AbstractFormController.handleRequestInternal(AbstractFormController.java:265)
 at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
 at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
 at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:299)
 at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
 at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
 at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
 at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.process(SSLReadTask.java:440)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.doTask(SSLReadTask.java:228)
 at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
 at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 208 completed: Maybe
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2862)
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:2880)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1788)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1263)
 at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
Caused by: org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 211  completed: No
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2946)
 at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:2965)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:2000)
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1713)
 ... 2 more
Caused by: java.io.IOException: End-of-stream
 at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1989)
 ... 3 more

Answer 1

By the stack trace, I can conclude that Spring Framework itself doesn't cache - it actually invokes InitialContext.lookup(). Still, the stack trace and the error code imply that there's a connection breakage.

WebSphere used to have a similar bug back in the days, which was fixed since. Turned out to be a caching problem done by the application server itself. I suspect SunAS has a similar bug in 9.1...

Isaac

Answer 2

Method syncChanges() is running periodically for synchonization purpose. The main idea is to close context if error occurs(after restarting server for exsample) and reinitialize it. After closing context with method context.close() you should null it's value.

public class SyncOperationsService {
    private InitialContext context;
    private SyncOperationsRemote syncOperationsRemote = null; //RemoteBean

    public String syncChanges() {
        try {
            if (context == null) {
                context = new InitialContext(properties);
            }

            if (context != null) {
                syncOperationsRemote = (SyncOperationsRemote) context.lookup("PathToYourRemoteBean");
            }

            result = syncOperationsRemote.syncChanges(dbImportList);  //Try execute remote bean

        } catch (Exception e) {
            log.error(e.getMessage());

            try {
                context.close();
            } catch (Exception e2) {
                log.error(e2.getMessage());
            }
            context = null;
        }
    }
}