After many hours of searching I found the solution. I had to use read function from unistd.h It fills an array of given size, with characters from the input. When a key is pressed the buffer is filled with all read characters (works also on multiple keys). So an ESC has simply {27,0,0,...,0} and arrow {27,'[','A',0,0,...,0}. I've rewritten my function using termios.h and put in library, so anyone can benefit.
Here is the code: readkey on github