WSO2 ESB unknown error code 102511

Question

I have recently upgraded WSO2 ESB to version 4.7 on the Windows Server 2008 R2 and encountered the next error while simply proxying SOAP request to an endpoint:

Receiving response while the handler is in an inconsistent state REQUEST_HEAD

ERROR_CODE : 102511  
ERROR_MESSAGE : Error in Sender  
ERROR_DETAIL : Error in Sender  
ERROR_EXCEPTION : null 

Thing is, that this error code is not described in the documentation and without an exception it is not obvious what to make of it. The closest code I could find was SND_INVALID_STATE = 102510 and judging from the source code it seems that the request comes with invalid headers. But not all requests are getting failed. The same request can pass or fail randomly. I've recorder all requests with fiddler and replayed them. The one that has failed can eventually pass and vice versa. Before that I've deployed and tested new version of ESB on my local machine (Windows 7) and encountered such error only on the cold start.

The simplest configuration to reproduce it consists of Path Through Proxy service and an address endpoint.

Proxy service config:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="TestEP" transports="http" statistics="disable" trace="enable" startOnLoad="true">
   <target endpoint="TestEP">
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>

Address Endpoint description

<endpoint xmlns="http://ws.apache.org/ns/synapse" name="TestEP">
   <address uri="http://mydomain.test/SystemServices.asmx">
     <syn:suspendOnFailure>
       <syn:initialDuration>0</syn:initialDuration>
       <syn:progressionFactor>1.0</syn:progressionFactor>
       <syn:maximumDuration>0</syn:maximumDuration>
     </syn:suspendOnFailure>
   </address>
</endpoint>

Has anyone else experienced this error or knows how to deal with it? I will be grateful for any insight on the situation.

An update:
It seems the reason why requests are failing is

Expect: 100-continue

option in the request HTTP header. When I created a rule to remove it in fiddler, all queries went successfully. It is still not clear whether there is a way to deal with such behavior on WSO2 ESB side or should this part of the header be removed.

Answer 1

I encountered this problem today when upgrading from WSO2 ESB 4.5.1 to 4.7.0. I had another problem with 4.5.1 that I had to upgrade because of, so upon encountering this problem with 4.7.0, I had no choice but to solve it.

After thinking for some time, I remembered that the default transport was switched from NHTTP to Passthrough in 4.6.0 to increase performance. 4.7.0 ships with configurations for both, but with PT enabled by default. The config files are in the axis2 directory:

${carbon.home}/repository/conf/axis2/

The PT config file is axis2_pt.xml. The NHTTP one is axis2_nhttp.xml. You can diff them to get an idea of what changes; the diff is luckily pretty clean.

You can easily switch from PT to NHTTP by modifying the main config file:

${carbon.home}/repository/conf/carbon.xml

There you have the <ConfigurationFile> element under <Axis2Config>. The default file, axis2.xml seems to be more or less a copy of axis2_pt.xml. To switch to NHTTP, just change the <ConfigurationFile> to ${carbon.home}/repository/conf/axis2/axis2_nhttp.xml.

Switching to NHTTP solved my problem of ESB 4.7.0 not dealing with 100 CONTINUE correctly. Specifically, I was trying to upload a PDF using curl through the ESB to another service. Using PT, it failed; using NHTTP, it worked fine. My obvious conclusion is that PT is simply buggy in this scenario.

From the reading that I have done about PT vs NHTTP, the only "official" side-effect of switching from one to the other is that PT should be faster in certain scenarios. However, NHTTP has been around for longer and is therefore likely to be a bit more solid simply by virtue of having gone through more bugfixing rounds. I don't know this for sure as I'm not involved with the development of WSO2 ESB, but it's an educated guess. :)

Also I'd love to comment on Isuru Perera's answer, but I don't have the necessary reputation so I'm afraid I have to put my two cents regarding https://wso2.org/jira/browse/APIMANAGER-1007 here. This issue does indeed seem related - especially based on my own experience with curl today - but unfortunately the kind folks at WSO2 have resolved the issue as "Not a bug" after a number of comments that ultimately recommended avoiding the use of curl because other clients "work as expected," thus rendering this a "curl issue." IMHO having broken behaviour with HTTP-spec-compliant requests is not a client issue and should be resolved. This effectively renders the PT transport unusable in certain scenarios - i.e. ones where the client is a little bit more intelligent about how it POSTs large bodies. This is really a shame, because scenarios where the request body does not need to be parsed are precisely what the PT transport was designed for and where it should excel!

Oh and this is my first post on stackoverflow so I'm sorry if I've broken any house rules; I've been a passive participant for some time so I'm hoping I didn't do anything too wrong!

Answer 2

It seems a similar error has occurred with WSO2 API Manager also. See https://wso2.org/jira/browse/APIMANAGER-1007

ESB runs within API Manager and therefore you could be experiencing a similar issue as mentioned in the JIRA comment.

Following doc has listed most of the error codes.

http://docs.wso2.org/wiki/display/ESB470/Error+Handling

But the error code mentioned in this question is not listed unfortunately.