What is the assertion that is failing exactly? That's the clue to what the problem may be.
assert(!XMVector3Equal(EyeDirection, XMVectorZero()));
assert(!XMVector3IsInfinite(EyeDirection));
assert(!XMVector3Equal(UpDirection, XMVectorZero()));
assert(!XMVector3IsInfinite(UpDirection));
Since you are using XMVECTOR and XMMATRIX as class members, have you made sure that your class Entity is always allocated with 16-byte alignment? This is only guaranteed in x64 (64-bit) native programs by default. If you are writing a 32-bit app, you need to use XMLoad*/XMStore* functions and avoid using XMVECTOR and XMMATRIX directly in your class. Make sure you know the difference between when you get a C assertion breakpoint to fire and when you are actually getting an exception thrown like 'access violation' due to alignment.
You may find that using SimpleMath (in the DirectX Tool Kit) more natural to use than directly using DirectXMath, but it is generally designed around RH coordinates.