GWAN terminating under load

Question

I have a web app that needs to be a restful interface. So I have a connection handler that tries to change the inbound request into something that gwan can use.

Every connection to this service is the same, so I am doing a replace on every connection:

  #include "gwan.h" // G-WAN exported functions

  #include <stdio.h>
  int init(int argc, char *argv[]){
     u32 *states = (u32*)get_env(argv, US_HANDLER_STATES);
     *states = 1 << HDL_AFTER_READ; // we assume "GET /hello" sent in one shot
     return 0;
  }

  void clean(int argc, char *argv[]){}

  int main(int argc, char *argv[])
  {
     const long state = (long)argv[0];
     if(state == HDL_AFTER_READ) {
        xbuf_t *read_xbuf = (xbuf_t*)get_env(argv, READ_XBUF);
        xbuf_replfrto(read_xbuf, read_xbuf->ptr, read_xbuf->ptr + 16, "/classify.htm?", "/?boost.cpp&");
     }

     return 255;
  }

The problem is, under load, after a little while, G-WAN crashes and give me an error:

G-WAN 4.3.14 (pid:20477)

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct NULL not valid


Signal        : 6:Abort
Signal src    : -6:tkill
errno         : 0
Thread        : 2
Code   Pointer: 7fc5dcd748a5 (module:libc.so.6, function:raise, line:0)
Access Address: 000000004ffd

Registers     : EAX=000000000000 CS=00000033 EIP=7fc5dcd748a5 EFLGS=000000000202
                EBX=0000006693e8 SS=0000000a ESP=7fc5d5c7bf38 EBP=7fc5880008d8
                ECX=ffffffffffffffff DS=0000000a ESI=00000000504e FS=00000033
                EDX=000000000006 ES=0000000a EDI=000000004ffd CS=00000033

Module         :Function        :Line # PgrmCntr(EIP)  RetAddress  FramePtr(EBP)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Aborted (core dumped)

The problem is only when I put the server under load. With one machine I'm hitting around 8000 requests/sec, and it lasts about 5 seconds before crashing.

If I DON'T do the rewrite (move main.c to main.c_) and call the cpp script directly, no crash...

Help! Any ideas?

Thanks

Answer 1

Add proper return value checks, do not take anything as granted.

This looks like get_env fails and returns NULL, but there is a lot todo for you (gdb, strace, ..) so somebody can help you.

Answer 2

IT was in a different part of the code... the biggest clue was It looked Like I was getting a C++ unhandled exception.

According to docs, G-Wan is written in C.

Dug into my code,and added some exception handling around where I suspected the code was crashing, and I saw the exception, but the server kept going, which is what I wanted!

Snippet where I fixed it:

try {
  get_arg("key=",&name,argc,argv);
  string url = name;

  boost::thread cls(classify_url,url,boost::ref(rp));
  if(!cls.timed_join(boost::posix_time::milliseconds(8))) {
   cls.interrupt();
   rp="{\"c\": [998]}";
  }
}
catch(...) {
   rp="{\"c\": [997]}";
   cout << "EXCEPTION" << endl;
}

Added the try/catch, and all is good!