I think I can answer my question by myself now. The issue was that the LoadAsync method doesn't work so well together with the await/async construct. The method was called by ThreadPool Thread A and then resumed (after await) by ThreadPool Thread B. This constellation threw the Exception. But I can't exactly say why...
With this answer (How to integrate WinRT asynchronous tasks into existing synchronous libraries?) I wrote the LoadAsync method into a synchronous method and now it works because the same thread is calling the method and uses the results of it.
Here is the modified code fragment:
IAsyncOperation<uint> taskLoad = reader.LoadAsync(receiveBufferSize);
taskload.AsTask().Wait();
bytesRead = taskLoad.GetResults();
Thanks Geoff for bringing me on to the right path with the threads :) I hope I can help someone who also (will) have this issue.