Windows Azure Worker Role not getting past first line of code

Question

I have a worker role that works perfectly in development but doesn't work when deployed. "Doesn't work" is rather vague, but that's really all I have to go on as I'm not seeing any errors or anything (in the event log anyway - maybe there is somewhere else I can look). I have added some trace statements to my code and I'm seeing the first one come out, but none of the others.

WorkerRole Code:

public class WorkerRole : RoleEntryPoint
{
    #region Member variables

    private IWindsorContainer _container;

    private IJob[] _jobs;

    #endregion

    #region Methods

    public override bool OnStart()
    {
        ConfigureDiagnostics();

        Trace.WriteLine("WorkerRole.OnStart()");

        try
        {
            Initialize();

            Trace.WriteLine("Resolving jobs...");
            _jobs = _container.ResolveAll<IJob>();

            StartJobs();

            return base.OnStart();
        }
        catch (Exception ex)
        {
            TraceUtil.TraceException(ex);
            throw;
        }
        finally
        {
            Trace.WriteLine("WorkerRole.OnStart - Complete");
            Trace.Flush();
        }
    }

    /// <summary>
    /// Sets up diagnostics.
    /// </summary>
    private void ConfigureDiagnostics()
    {
        DiagnosticMonitorConfiguration dmc =
            DiagnosticMonitor.GetDefaultInitialConfiguration();

        dmc.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
        dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;

        DiagnosticMonitor.Start(Constants.DiagnosticsConnectionString, dmc);
    }

    /// <summary>
    /// Sets up the IoC container etc.
    /// </summary>
    private void Initialize()
    {
        Trace.WriteLine("WorkerRole.Initialize()");

        try
        {
            Trace.WriteLine("Configuring AutoMapper...");
            AutoMapperConfiguration.Configure();

            Trace.WriteLine("Configuring Windsor...");
            _container = new WindsorContainer();

            Trace.WriteLine(string.Format("Installing assemblies from directory...{0}", 
                Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot)));

            _container.Install(FromAssembly.InDirectory(
                new AssemblyFilter(Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot))));

            Trace.WriteLine(string.Format("Setting the default connection limit..."));
            ServicePointManager.DefaultConnectionLimit = 12;
        }
        finally
        {
            Trace.WriteLine("WorkerRole.Initialize - Complete");
        }
    }

    /// <summary>
    /// Starts all of the jobs.
    /// </summary>
    private void StartJobs()
    {
        Trace.WriteLine("WorkerRole.StartJobs()");

        try
        {
            foreach (IJob job in _jobs)
            {
                job.Start();
            }
        }
        finally
        {
            Trace.WriteLine("WorkerRole.StartJobs - Complete");
        }
    }

    public override void OnStop()
    {
        Trace.WriteLine("WorkerRole.OnStop()");

        try
        {
            foreach (IJob job in _jobs)
            {
                job.Stop();
            }
            _container.Dispose();
        }
        finally
        {
            Trace.WriteLine("WorkerRole.OnStop - Complete");
        }
    }

    #endregion

    #region Private util classes

    public static class AutoMapperConfiguration
    {
        public static void Configure()
        {
            Mapper.Initialize(x => x.AddProfile<ModelProfile>());
        }
    }

    #endregion
}

TraceUtil Code:

public static class TraceUtil
{
    public static void TraceException(Exception ex)
    {
        StringBuilder buffer = new StringBuilder();

        while (ex != null)
        {
            buffer.AppendFormat("{0} : ", ex.GetType());
            buffer.AppendLine(ex.Message);
            buffer.AppendLine(ex.StackTrace);

            ex = ex.InnerException;
        }
        Trace.TraceError(buffer.ToString());
    }
}

Config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  ...
  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
             name="AzureDiagnostics">
          <filter type="" />
        </add>
      </listeners>
    </trace>
  </system.diagnostics>
</configuration>

Once the worker has started, if I look in the WADLogsTable, all I see is "WorkerRole.OnStart()" and nothing else!

Any ideas on what the problem could be or how to troubleshoot this would be appreciated.

Update: If I stop the role, I don't see any of the debug statements from the OnStop() method either.

Update: I must have something configured incorrectly with my diagnostics. I thought I was seeing my debug come out correctly when debugging locally, but it turns out I'm not. I see everything in the output window, but I don't see everything in the storage table. I am seeing the following entries in development:

WorkerRole.OnStart()
WorkerRole.Initialize()
Configuring AutoMapper...

I realize that the trace output is only periodically uploaded, but I've waited 5 minutes or so, so I think this should be long enough, since I have it set to 1 minute.

Update: As suggested by @kwill in the comments section I have tried adding a file trace listener as follows:

  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
             name="AzureDiagnostics">
        </add>
        <add name="File" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:\TextWriterOutput.log" />
      </listeners>
    </trace>
  </system.diagnostics>

This works fine in my development environment and seems more reliable as well as I get all of the debug out that I would expect. When I deploy it to staging however, the TextWriterOutput.log file isn't even created!

I really need a reliable way of getting debug out of my worker role so that I can troubleshoot the ultimate problem, which is that my jobs aren't working - at this point I still have no idea what they are even trying to do as I can't get any debug out!

Update: I am pretty sure that the missing dll idea suggested by most people is not the problem. To hopefully prove this, I have overridden the run method as shown below and I am seeing the "Heartbeat..." debug come out. It seems to me that either the diagnostics functionality or at least the way I have it configured is unreliable, which is preventing me from investigating why on earth my jobs aren't running.

    public override void Run()
    {
        Trace.WriteLine("LearningMiles.JobProcessor.WorkerRole.Run()", "Information");

        try
        {
            while (true)
            {
                Thread.Sleep(10000);
                Trace.WriteLine("Heartbeat...", "Verbose");
            }
        }
        catch (Exception ex)
        {
            TraceUtil.TraceException(ex);
            throw;
        }
        finally
        {
            Trace.WriteLine("LearningMiles.JobProcessor.WorkerRole.Run() - Complete", "Information");
        }
    }

Update: I have now cross posted this problem on the Windows Azure MSDN forum.

Update: As suggested in the comments, I have now tried removing all 'useful' code. In development this resulted in all of the debug being output. I then tried just removing the call to AutomapperConfiguration.Configure() since previously I was seeing nothing come out after that call. This resulted in some of the trace statements not coming out again. Importantly however, I was seeing the trace statements that I have put in the "jobs". Since it's the jobs not running that I ultimately want to resolve, I deployed that version of the code to staging, but there I just see the OnStart() trace and the "heartbeat" trace. I don't think this really helps, but maybe it will give someone some ideas.