Solution I implemented was adding local Administrators group on my server to security policy Deny access to this computer from the network
.
That prevents users with rights to do the modification from doing it remotely, while pertaining possibility to remote desktop, this forcing administrators to login. Just keep in mind that this will affect also your service account etc.. (anything logging remotely to computer)