I think I figured out the problem. I have many different types of GUIDs, and I thought they all used the same IConnectionPoint, so I saved it to the same object in the RegisterEvent function.
When I tried creating a new IConnectionPoint object for each GUID, and saving each IConnectionPoint separately, the DeregisterEvent function also worked properly.