multiple buffer management in C

Question

What would you recommend I do to buffer multiple streams (each stream would require it's own buffer) in C?

I sort of put together a solution in my head:

This is theoretical "pseudo code"

[ptr, ptr, ptr, ptr ... ]

Each ptr is a pointer to a struct:

{
    stream_id
    buff_ptr
}

stream_id is the ID of the stream (I have that information) and the buff_ptr is a pointer to some malloced data. Also, the buffers would probably have to be flexible and it'd have to call malloc and free a lot to make it flexible (at least I think it would?

I would basically find the buffer I need by iterating through the array of ptrs.

---

So, is that "architecture" good, or is there a better one?

If anyone is wondering "Why the nyan does she need that?!" I've made a server in Ruby/EventMachine, the protocol is really rotten (to say the least) and I don't know how else to handle it... Unless there's a way to force EventMachine to buffer it for me, that would be the best solution, I suppose.

Thanks.

P.S: Sorry for the rave! ~

Answer 1

Well, it seems the only option but you can change a few things:

• Don't malloc and free. Use realloc instead

• Iterating through the list could be OK but if there are a lot of buffers

  • Can you make the caller pass you a pointer to the buffer ?
  • Can you use a hash instead of a list and search by stream_id ? For example glib2 provides such a hash

Many APIs (for example epoll) allow you to associate a pointer with an id (a descriptor in the case of epoll). When something interesting happens to that id, the API passes you your pointer -> zero search at the expense of 4-8 bytes of memory.

Answer 2

Use libev to make a simple event loop and let it handle polling file descriptors. There are lots of examples out there and it's manual page (man 3 libev) is one of the best I've read.

Just define a few callbacks and create some buffers for your read and write pools. The size of those pools can remain static so long as you block or skip your reads and writes when the buffer fills up or exhausts, respectively.