If you have the proper exception handling in place, the cast to your type would fail - and no harm would be done.
If you want more control over how your data is serialized, and you want to keep an eye on what is happening in the de-serialization process you will ultimately need to implement your own object classifier of some sort.
Maybe even a simple XML format, so that you can just iterate over the nodes, and if anything doesnt look like it should, just throw it out.
Although, if you want to figure this out yourself:
Here is the official specification for the BinaryFormatter: http://msdn.microsoft.com/en-us/library/cc236844(prot.20).aspx
And also, here is smaller, third party specification for it: http://primates.ximian.com/~lluis/dist/binary_serialization_format.htm
Using those resources, you would be able to peek into the binary stream and see the type of serialized content. However, seems very unnecessary.
Just do extensive checking on the data after it has been deserialized.