rEFIt menu takes around 10 seconds to even show up

Question

I'm using a macbook pro 8,1. I've installed rEFIt so that I can dual boot with Ubuntu. This is all "working" except that each time I turn on my laptop, it takes ten or fifteen seconds to even get to the rEFIt menu for me to select an OS.

At this point, I have done nothing special except to make Ubuntu the default OS and reduce the wait time in the rEFIt menu from 20 to 5 seconds.

How can I reduce this lag between startup and OS selection?