Quartz.Net problems updating a job, error reporting/monitoring

Question

I have a quartz.net client/server setup to fire some text messages on a schedule using another Third Party library.

Quartz.Server.exe is running as a windows service on my staging and development environments, pointing to jobs on SQL and my website uses Quartz.Simpl.ZeroSizeThreadPool and just schedules the jobs. Everything was working fine until it wasn't.

Apparently something had caused the exe to stop running and even though it was running as a windows service with recovery options set to send me an email when it went down, I did not get the email.

So when I restarted the server 15 days worth of old text messages went out with misleading phrases like "Appointment tomorrow @ 9:00 AM" for appointments for 5 days ago.

So now I have updated my IJob code so that the job is discarded if the fire-time is after the appointment time. Since I was using DateTime instead of plain strings I changed "quartz.jobStore.useProperties = false". When I was ready to deploy I realized that this change along with some other changes could break the scheduler so that it wouldn't fire historical triggers from before the changes.

I spiraled down a 12+ hour black hole trying to wrestle with the settings to get my new jobs to fire alongside my old jobs locally and on my staging environment. Tried a million things, including every combination of Quartz running as a service or console pointing to SQL local or on staging. Said potty-words. Then said some prayers.

Started over. Tested to see if the working code on the live server would fire both new and old jobs without testing. It wouldn't. I Changed "quartz.jobStore.useProperties = false" in quartz.config. It worked! So I did a new deployment Monday evening with the changes and everything seemed to be working fine. I did a "Keeping my job dance", and revisited my error logging/recovery setup and created a nightly job in SQL to count number of triggers yesterday, today, and tomorrow each morning.

Here it is Wednesday morning and I check the SQL job I set up to find 150+ triggers from Yesterday have not been handled (everything past 9:00 AM EST). So I go to my live server and the Quartz service is still running. I stop the process and go to my folder where it lives and Right_Click>Run as Admin. Here is the error I get in my log.txt file. And of course now the Windows service fails to start.

I need major help fast! I need a montage, and at the end of the Montage I need to be a quartz.net ninja. Marko Lahma (or anyone) will you "hold my hand" and tell me I am going to be able to keep my job? I need my setup to be bulletproof (be more robust/fail more noticeably).

EDIT - 201404241938Z

Here is my code to check for the new values that broke the old jobs

 // I think this next line is causing an error
 DateTimeOffset oDateReminder = data.GetDateTimeOffset("reminderTime"); 
 if (oDateReminder.DateTime > DateTime.MinValue)
 {
     // Do stuff with other "new job" datamap keys
 …

 // Changing it to…
 if (data["reminderTime"] != null)
 {
     // Do stuff with other "new job" datamap keys
 …

EDIT - 201404242205Z

This seems to have worked. I'll wait and award some bounty

Answer 1

As you probably know, the data is now corrupted in the DB containing both JobDataMaps (binary) and NameValueCollections (binary).

Would you like to give a whirl to the latest head of repository master? I've ninjacommitted support for recovering from this mixed data situation. Just take the latest and build with custom snk file and drop the DLL for run.

This is version 2.2.x so take note if you are using something like 2.1 or 2.0.

The issue is tracked here https://github.com/quartznet/quartznet/issues/172

What comes to your original issue one easy fix would to ignore misfires, if SMS notification doesn't fire within expected time then it probably should be ignored and not fired immediately when possible. Logging information might help you get info about what caused the original downtime (windows updates, SQL server not started yet etc).