If you can use external libraries, guava has a RateLimiter class which probably does what you ask for:
Rate limiters are often used to restrict the rate at which some physical or logical resource is accessed. This is in contrast to Semaphore which restricts the number of concurrent accesses instead of the rate.