Assuming that you aren't too picky about what parameters are passed to the OpenDialog
method , you can setup the mock service method like so:
mockUIService
.Setup(_ => _.OpenDialog(It.IsAny<String>(),
It.IsAny<String>(),
It.IsAny<String>())
.Returns("SomeFakeFileName.ext");
It will return the hardcoded filename "SomeFakeFileName.ext"
to your SUT.
Similarly, if your service also offers a stateful "last FileName" property:
mockUIService
.SetupGet(_ => _.FileName)
.Returns("SomeFakeFileName.ext");