After further investigation it seems that it's not possible to have two endpoints with different SSL settings.
So if one endpoint of a service has SSL enabled, the other one needs it as well.
Unless someone else can proof that this is not correct, I have solved the issue by duplicating the service and thus creating two services. One has SSL enabled, the other one has it disabled.