Is there a way to lock the whole thing
There is no way to prevent mutation of an object (or object graph) that is mutable. Put differently: If the object can be modified (such as if it has a public field that isn't readonly or if it has a property that has a setter), there is no way to prevent it from being modified.
I tried looking at readonly modifier, but I guess it won't suit me because it constraints only the reference of the object and not everything that comes under it.
Correct. When a field declaration includes a readonly modifier, assignments to the fields introduced by the declaration can only occur as part of the declaration or in a constructor in the same class. (Source: msdn)
However, you can design the type so it is immutable or at least so it can't be modified through public fields, properties or methods.
You might also consider to return clones of the object graph instead of the original object graph. If the clone gets modified, the original object graph is still unmodified.