How to call SemaphoreSlim.Release() without risking app failure?

Question

I'm trying to use the new SemaphoreSlim class in .NET 4.0 to rate-limit a fast-paced loop that can run indefinitely. In unit testing this, I've found that if the loop is tight enough and the degree of parallelism is high enough, the SemaphoreSlim can throw an uncatchable exception when you call Release(), even if you check the .Count property first and lock around the semaphore instance itself during the entire check-count/release sequence.

This exception takes down the app, period. There is no catching it as far as I can tell.

Digging deeper, I discovered that the SemaphoreSlim is trying to access it's own .AvailableWaitHandle property internally during the Release() call, it throws the exception there, not from me accessing the SemaphoreSlim instance itself. (I had to debug with Debug->Exceptions->Common Language Runtime Exceptions->Thrown all checked in Visual Studio to discover this; you can't catch it at runtime. See The Uncatchable Exception for more details.)

My question is, does anyone know of a bullet-proof way to use this class without risking immediate app termination in such a case?

Note: The semaphore instance is wrapped in a RateGate instance, the code for which can be found in this article: Better Rate Limiting in .NET.

UPDATE: I'm adding complete console app code to reproduce. Both answers contributed to the solution; see below for explanation.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading.Tasks;
using System.Linq;
using System.Text;
using System.Threading;
using PennedObjects.RateLimiting;

namespace RateGateForceTerminateDemo
{
    class Program
    {
        static int _secondsToRun = 10;
        static void Main(string[] args) {
            AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
            OptimizeMaxThreads();
            Console.WriteLine();
            Console.WriteLine("Press any key to exit.");
            Console.ReadKey(true);
        }

        static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e) {
            Console.WriteLine("Unhandled exception, terminating={0}:{1}", e.IsTerminating, e.ExceptionObject.ToString());
            Console.WriteLine("Press any key to terminate app.");
            Console.ReadKey(true);
        }

        static void OptimizeMaxThreads() {
            int processors = Environment.ProcessorCount;
            int processorsSq = Convert.ToInt32(Math.Pow(processors,2));
            int threads = 1;
            double result;
            Tuple<int, double> maxResult = new Tuple<int, double>(threads, 0);
            while (threads <= processorsSq) {
                Console.WriteLine("Running for {0}s with upper limit of {1} threads... ", _secondsToRun, threads);
                result = TestThrottling(10000000, threads, _secondsToRun);
                Console.WriteLine("Ok. Result is {0:N0} ops/s", result);
                Console.WriteLine();
                if(result > maxResult.Item2)
                    maxResult = new Tuple<int, double>(threads, result);
                threads *= 2;
            }
            Console.WriteLine("{0} threads achieved max throughput of {1:N0}", maxResult.Item1, maxResult.Item2);
        }
        static double TestThrottling(int limitPerSecond, int maxThreads, int maxRunTimeSeconds) {
            int completed = 0;
            RateGate gate = new RateGate(limitPerSecond, TimeSpan.FromSeconds(1));
            ParallelLoopResult res = new ParallelLoopResult();
            ParallelOptions parallelOpts = new ParallelOptions() { MaxDegreeOfParallelism = maxThreads };
            Stopwatch sw = Stopwatch.StartNew();
            try {
                res = Parallel.For<int>(0, 1000000000, parallelOpts, () => 0, (num, state, subtotal) =>
                {
                    bool succeeded = gate.WaitToProceed(10000);
                    if (succeeded) {
                        subtotal++;
                    }
                    else {
                        Console.WriteLine("Gate timed out for thread {0}; {1:N0} iterations, elapsed {2}.", Thread.CurrentThread.ManagedThreadId, subtotal, sw.Elapsed);
                        // return subtotal;
                    }
                    if (sw.Elapsed.TotalSeconds > maxRunTimeSeconds) {
                        Console.WriteLine("MaxRunTime expired for thread {0}, last succeeded={1}, iterations={2:N0}, elapsed={3}.", Thread.CurrentThread.ManagedThreadId, succeeded, subtotal, sw.Elapsed);
                        state.Break();
                    }
                    return subtotal;
                }, (subtotal) => Interlocked.Add(ref completed, subtotal));
            }
            catch (AggregateException aggEx) {
                Console.WriteLine(aggEx.Flatten().ToString());
            }
            catch (Exception ex) {
                Console.WriteLine(ex);
            }
            sw.Stop();
            double throughput = completed / Math.Max(sw.Elapsed.TotalSeconds, 1);
            Console.WriteLine("Done at {0}, finished {1:N0} iterations, IsCompleted={2}, LowestBreakIteration={3:N0}, ",
                sw.Elapsed,
                completed,
                res.IsCompleted,
                (res.LowestBreakIteration.HasValue ? res.LowestBreakIteration.Value : double.NaN));
            Console.WriteLine();
            //// Uncomment the following 3 lines to stop prevent the ObjectDisposedException:
            //Console.WriteLine("We should not hit the dispose statement below without a console pause.");
            //Console.Write("Hit any key to continue... ");
            //Console.ReadKey(false);
            gate.Dispose();
            return throughput;
        }
    }
}

So using @dtb's solution it was still possible for a thread "a" get past the _isDisposed check, and yet have thread "b" dispose the semaphore before thread "a" hit Release(). I found that adding a lock around the _semaphore instance in both the ExitTimerCallback and Dispose methods. @Peter Ritchie's suggestion led me to additionally cancel and dispose the timer, before disposing the semaphore. These two things combination let the program complete and dispose of the RateGate properly with no exceptions.

Since I wouldn't have gotten this without that input I don't want to answer myself. But since StackOverflow is more useful when complete answers are available, I'll accept whichever person first posts a patch or pseudo-patch that successfully survives the console app above.

Answer 1

Based on your comment, it sounds like you're spawned a bunch of threads to work with a RateGate object that gets disposed before those threads are done using it. i.e. the threads are still running after your code exits from the using block. UPDATE: if you do what you described in your comment; but don't use a using block you don't get the problem. the exception I witnessed was actually an ObjectDisposedException; which would make sense if the RateGate was disposed before code was done using it...

Answer 2

The problem is in the RateGate class you're using. It has an internal Timer that runs its code even after the RateGate instance is disposed. This code includes a call to Release on a disposed SemaphoreSlim.

Fix:

@@ -88,7 +88,8 @@
    int exitTime;
    while (_exitTimes.TryPeek(out exitTime)
            && unchecked(exitTime - Environment.TickCount) <= 0)
    {
+       if (_isDisposed) return;
        _semaphore.Release();
        _exitTimes.TryDequeue(out exitTime);
    }