I assume the RegisterGUI is a class name, hence the method is static. Obviously in Java it is impossible to override static methods, so you cannot do it with mockito. I would recommend refactor code and don't use static methods.
However, if you are dealing with legacy code you could use Powermock which manipulates bytecode and overrides statics.
Another approach to work with legacy code is to extract the static method call into a new method. So your controller will be:
public class RegisterController {
public void regHandle(UserDataObject user1){
ValidateRegisterInputController validate = new ValidateRegisterInputController();
validate.validateInputHandle(user1);
OracleUserManagement usermanagement = OracleUserManagement.getInstance();
if (usermanagement.storeUser(user1) == false) {
errorUsernameDuplicate(); }
}
void errorUsernameDuplicate() {
RegisterGUI.errorUsernameDuplicate();
}
}
Now you could use mockito spy to verify your controller:
verify(regcontroller).errorUsernameDuplicate()
Obviously the method itself becomes untested, however it is trivial and it is not a problem.