You should not remove everything under the domain\servers\Admin
folder. You can probably remove the tmp/stage/cache folder under the Admin folder, but you should not delete others such as data folder. For example, the embedded ldap data are stored under data\ldap\ldapfiles. That stores the data for all the internal users/groups. If they are not their, weblogic tries to load the default settings. Taht's why your users/groups get lost after every restart.
Lan