I was able to get rid of the latency. Let me answer it myself.
There seems to be some problem with Twython, I did not look up the internals but I am pretty sure it is with Twython because when I started using Tweepy everything works fine. I am guessing Twython is opening multiple connection caused a bottleneck, once again I haven't looked up the internal so I am not really sure what is going on with Twython.