Prevent deadlock issue with WCF duplex callback service

Question

I have a problem with a self-hosted wcf duplex callback service. I get an InvalidOperationException with message:

This operation would deadlock because the reply cannot be received until the current message completes processing. If you want to allow out-of-order message processing, specify ConcurrencyMode of Reentrant or Multiple on CallbackBehaviorAttribute.

Here is my service behavior:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode =  ConcurrencyMode.Reentrant, UseSynchronizationContext = true)]

Here is my service contract:

 [ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IClientCallback))]

[ServiceContract]
public interface IClientToService
{
    [OperationContract(IsOneWay = false)]
    LVSSStatus GetLvssStatus();

    [OperationContract(IsOneWay = true)]
    void PickSpecimen(long trackingNumber, int destCode);

    [OperationContract(IsOneWay = true)]
    void CancelCurrentPickTransaction();
}

Here is my callback interface:

public interface ILvssClientCallback
{
    [OperationContract(IsOneWay = true)]
    void SendClientCallback(LvssCallbackMessage callbackMessage);

    [OperationContract(IsOneWay = false)]
    List<SpecimenTemplateDescriptor> GetTemplateDescriptorList(DrawerLayout drawerLayout);

    [OperationContract(IsOneWay = false)]
    SpecimenTemplate SelectSpecimenTemplate(string templateName, int version);

    [OperationContract]
    void SpecimenStoredInContainer(string containerID, bool isValidRackID, int rackRow, int rackCol, int deckRow, int deckCol,
     int drawerRow, int drawerCol, long trackingNumber, RobotErrors robotError);

    [OperationContract]
    void LvssRobotStatusChange(LVSSStatus status);
}

I understand that the InvalidOperationException is caused when a callback operation is invoked on the client, the service is already locked for the processing of a current operation. So, a deadlock occurs.

I have tried changing my ConcurrencyMode to multiple, and UseSynchronizationContext to false.

I still see two problems with my service:

First: The following service operation freezes my client wpf application when GetLvssStatus() is called rapidly (by clicking the UI button rapidly). This method is not one way, and returns an enumerated type from the service back to the client synchronously.

    [OperationContract(IsOneWay = false)]
    LVSSStatus GetLvssStatus();

* What causes my wpf application to freeze in this case? * What can I do to prevent the application from freezing? If I use backgroundworker thread as an asynchronous call, the application does not freeze. I really need this method to work synchronously.

Second: When I assign the callback method LvssRobotStatusChange to IsOneWay = true, I get an ObjectDisposedException: Cannot access a disposed object. Object name: 'System.ServiceModel.Channels.ServiceChannel'.

    [OperationContract(IsOneWay = true)]
    void LvssRobotStatusChange(LVSSStatus status);

* What causes this ObjectDisposedException? * Is it okay to omit the IsOneWay assignment in this case? Omitting IsOneWay in this case allows the callback to complete without any exceptions.

* Could these issues be a result of a lack of thread safe code?
* If so, what is the best practice to make a ConcurrencyMode.Multiple service behavior thread safe?

Any help with these questions is much appreciated.

* FIRST EDIT A little more information regarding creation of my duplex channel. My wpf view model creates a proxy object that is responsible for handling creation of my channel. Any attempts so far to set my channel on a new thread on the client side result in an ObjectDisposedException when the service attempts to use the callback object.

* SECOND EDIT I believe my service should work if I can get the operation contracts with void method to set IsOneWay = true. With concurrency of reentrant, the main channel thread should let these methods pass through regardless of any locking.
Here is my callback interface:

public interface ILvssClientCallback
{
    [OperationContract(IsOneWay = true)]
    void SendClientCallback(LvssCallbackMessage callbackMessage);

    [OperationContract]
    List<SpecimenTemplateDescriptor> GetTemplateDescriptorList(DrawerLayout drawerLayout);

    [OperationContract]
    SpecimenTemplate SelectSpecimenTemplate(string templateName, int version);

    [OperationContract(IsOneWay = true)]
    void SpecimenStoredInContainer(string containerID, bool isValidRackID, int rackRow, int rackCol, int deckRow, int deckCol,
     int drawerRow, int drawerCol, long trackingNumber, RobotErrors robotError);

    [OperationContract(IsOneWay = true)]
    void LvssRobotStatusChange(LVSSStatus status);
}

When I set method LvssRobotStatuschange operation contract to IsOneWay = true, my cached callback channel throws a CommunicationObjectAbortedException. For some reason my callback property is being aborted.

***What can cause a callback channel to become aborted?

Answer 1

I've run in to this before, this link should help, which discusses creating channels on a thread other than the applications main thread.

Answer 2

Problem I encountered:

CallBackHandlingMethod()
{
    requestToService();    // deadlock message.    
}

way out:

CallBackHandlingMethod()
{
    Task.Factory.StartNew(()=>
    {
        requestToService();
    });
}

Answer 3

I had similar problem which I solved simply by adding

[CallbackBehavior(ConcurrencyMode=ConcurrencyMode.Multiple)]

to my callback implementation.

Answer 4

[CallbackBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
public class ServiceCallbackHandler : IServiceCallback
{
 ...
}

Answer 5

When using UseSynchronizationContext = true with a value for CallbackBehavior.ConcurrencyMode of anything but Multiple, you are creating a deadlock when making a callback from within a call to the service. The call stack looks like:

• Client: btDoSomething_Click → Service.DoSomething();
  • Server: DoSomething → Callback.ReportUpdate();
    • Client: on an IO callback, CallbackSynchronizationContext.Send(delegate { Callback.ReportUpdate(); })

The call to CallbackSynchronizationContext.Send hangs because it refers to the thread executing btDoSomething_Click. There are a number of ways to get out of this loop:

1. Avoid making the service synchronous (by applying [OperationContract(IsOneWay = true)]
   (This causes the client to release the UI thread as soon as the request is sent to the server rather than waiting for a reply from Service.DoSomething).

2. Make the callback not require the UI thread (by applying [CallbackBehavior(UseSynchronizationContext = false)] or [CallbackBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
   (In either case, the call to your callback will be made from a threadpool thread. If you need to marshal back to the UI thread, you can still do so with Synchronizationcontext.Post)

3. Update your client side calls to be async (change your service contract from [OperationContract] void DoSomething(); to [OperationContract] Task DoSomethingAsync();)

---

TL;DR: the combination of making a callback with [CallbackBehavior(UseSynchronizationContext = true)] from within a non OneWay operation will cause a deadlock on your client's synchronization context. Change your operation contract to use async if you need callbacks to be on your synchronization context:

Old:

[ServiceContract(CallbackContract = typeof(IControllerServiceCallback))]
public interface IControllerService
{

    [OperationContract]
    void OpenDrawer();
}

New:

[ServiceContract(CallbackContract = typeof(IControllerServiceCallback))]
public interface IControllerService
{
    [OperationContract]
    Task OpenDrawerAsync();
}