If your machine is not under domain then where does WORKGROUP\mtsuser
come from? Who is managing its credentials? I think you should use local machine name\mtsuser
and manage credentials locally. Besides you are not supposed to be able to access CME with MTSUser. This user is system user and is only to be used by the system. Also, as you are on 5.3 version, you should check documentation and make sure you've granted all the necessary rights and permissions to this user.
And you should really consider one of the supported versions of Tridion