Reasons May be:
Maybe your PHP is running per CGI or FastCGI in a separate process that shuts down after a while of inactivity. Does it change if you register at an uptime monitoring service? A cache plugin like W3 Total Cache may help too.
One of the typical reasons for such behavior is if you are loading RSS feeds in front-end. Their cache expires and you get very long pause next time, because they need to be fetched over network.
Try to replicate it on local server, it will give you more options for troubleshooting - ultimate one being making full profiler dump (with xdebug for example) and analyzing it.
Page Speed Optimization
This page makes 30 parallelizable requests to kerala.letmeshare.org (Reduce them)
Cachea resources have a short freshness lifetime.
http://kerala.letmeshare.org/wp-content/themes/TheTravelTheme/includes/timthumb.php?src=http://kerala.letmeshare.org/.... [Remove these loops - Request again requesting looping same hostname]
Size of Homepage 1.2 MB [Reduce Size - Optimize Images]