Generally a SQL request (Given that the DB is local) should be quicker than that. *granted. Try looking here for a great list of performance Tweaking. http://www.askapache.com/mysql/performance-tuning-mysql.html
That aside, I am presuming its the structure of your requests. Try keep your call into very few requests (1-10) and handle the data as a lump sum. Doing Many requests can hinder the system and drastically reduce that performance. Your server has more than enough ram and processing power.
Thats the best i can offer for now.