Can you count the number of lines returned from HoughLines - if there are 1000's then it will likely take that sort of time to generate.
I'd recommend changing your Canny settings to reduce the number of edges that HoughLines needs to work on if possible.
Also, you can try different parameters for HoughLines. My values of 80, 30 and 10 respectively for the call to HoughLines seems to give manageable results.