The answer depends on whether the objects you want to dispose adhere to the guidelines or not.
If they do some things get simpler, from IDisposable.Dispose:
If an object's Dispose method is called more than once, the object must ignore all calls after the first one. The object must not throw an exception if its Dispose method is called multiple times. Instance methods other than Dispose can throw an ObjectDisposedException when resources are already disposed.
So if calling Dispose multiple times has no effect other than the first time it's called you only have to ensure that you don't call them at the same time. And that should absolutely be the responsibility of the caller.
Stephen Cleary wrote an article on CodeProject about how to implement IDisposable and the possible problems which I find extremely helpful.