Found a solution on my own. I used the patterns & practices Improving Web Services Security: Scenarios and Implementation Guidance for WCF to get a better understanding of security in WCF. This book also includes HowTos for certain scenarios. Only thing I had to do additional.
Because my Web Service is separated in three smaller services, one of them is using the WsDualHttpBinding I needed to tell iis that this one doesn't requires the ssl certificate for transport.
For now on my service talks, but I need to check if it really uses the security I want to have.
UPDATE: Found this article that also contains a step by step tutorial.