large scale time triggered event handling

Question

Another question in large scale programming:

I have a job queue with time stamps and target file name. (For each timestamp, there might be up to 500 target files to process). The processing algorithms are the same for all the 500 target files. I want to do:

• Write program in Java
• whenever arriving at the timestamp, trigger all the 500 jobs all at once
• do it efficiently, in terms of computation efficiency, cpu usage and scalability

I know stupid way to do it, using while loop, checking if current time is the timestamp in job queue.... But any other alternatives? I also google it online, and there are also some ppl saying using cron command in Unix. (Yes, my target setup is in Unix.)

I am new to this large scale computing field, any recommendation or suggestion is welcomed.

Answer 1

Use a batch scheduler such as Quartz, if you want your job queue to be persistent.

A lighter-weight alternative is ScheduledThreadPoolExecutor from the java.util.concurrent package, which you can create using the Executors factory class. This allows you to register Runnable tasks to be executed at a fixed time.

Answer 2

It sounds like what you want is a priority queue. You basically need to sort your items by the timestamp within the queue.