The prompt did disappear after some time so I thought it to be a glitch of Windows 8 like Ricky assumed. But when I had to update my certificate I had the same problem again, this time with Windows 8.1. It took some days but finally it doesn't occur any more.
I assume the reputation of the application has increased to a level where it no longer needs to show the prompt.